In this article we discuss why a test automation strategy is essential for truly successful test automation. Several aspects of creating a test automation strategy are covered, including an example of what happens when there is no strategy for test automation, a process and methodology for creating a test automation strategy, and how an effective test automation strategy can lead to better test automation plans and test tool integration.
Introduction
In just about any endeavor, it is very helpful to have a strategy or approach at the very outset – even before defining a plan. Strategies are very general, high-level descriptions and are very “big picture” in nature. Actually, a strategy can be seen as a frame around the big picture, with a major high-level plan to be the big picture.
Strategies are where you define your major objectives and constraints, and in essence why you want to embark on an effort. The details which flow from the strategy are tactics that describe how the strategy will be achieved.
For example, a career strategy would be helpful for defining your big career goals, such as becoming an expert in your field and commanding a large salary. The details of getting the right training, finding your first job and moving forward would all be details in the plan.
Likewise, a test automation strategy defines your big objectives for test automation, along with the major benefits, risks, and challenges. This leaves the groundwork for a detailed test automation plan which can specify the technologies, tools, skills, people, and other resources needed to make the strategy a reality.
Table Of Contents
- The Distinction of Test Automation
- Why You Need a Test Automation Strategy
- What Might Be in a Test Automation Strategy?
- How to Define a Test Automation Strategy
- Automation Test Strategy Sample
- Automation Testing Methodologies
- The Role of a Test Automation Plan
- Test Automation Organization
- Automated Test Cases
- Automated Test Case Design and Implementation
- Test Automation Risks
- Test Automation Best Practices
- Conclusion
The Distinction of Test Automation
At this point, let’s draw a point of distinction and clarification for this article. The focus of this article is test automation, which includes tools and other means (such as pure scripting in a language such as Python, Ruby, etc.) with the objective of repeatable and unattended test execution and evaluation.
The broader field of test tools includes test automation and also includes the capabilities fortest management (such as provided in PractiTest), defect reporting, and so forth.
Test automation must be able to integrate and interoperate well with other tools in use in an organization. In that context, an understanding of the other tools in use in an organization is very important to include in a test automation strategy.
Another important point is that your test automation strategy may include solutions that are not tool-based, but instead implement automation as coded in a particular language such as Ruby, Python, Kotlin, and so forth. However, identifying exactly which tools, languages, and other details will emerge after evaluations, proofs of concept, and pilot projects
At the strategic level, the main concern is to identify and document the major direction of the test automation effort considering the desired objectives and real-world constraints. For example, if you have little or no budget but strong technical skills, you may be more likely to have a test automation strategy that is centered around open source solutions or pure coding.
On the other hand, if you have sufficient budget but lack technical skills, you may lean toward tools that are user friendly and also include acquiring people with the needed technical skills to evaluate and implement the tools. In addition, people with technical skills are needed to maintain the test automation regardless of who creates it.
Why You Need a Test Automation Strategy
-
It allows time to get clarity about the problem domain
It is a very common tendency for humans to identify a need or problem, then immediately dive into the effort of solving it. This is especially true for those of us in the field of Information Technology. We see the problem or objective, and start to see solutions right away. However, the solutions we see at first may not be the best solutions.
How many times have you attempted to solve a problem, only to realize that you started with invalid assumptions or a flawed approach?
All too often, we are so much in a hurry to get started solving a problem that we don’t take the time to ask clarifying questions or gain further understanding about the context around the problem. So, we may create an elegant solution for the wrong problem. Therefore, that kind of solution would be ineffective to solve the original need.
I have seen this happen many times in organizations where I conduct testing assessments and find a wide variety of tools in the inventory of tools, but some are not even being used. Of those tools that are being used, some are being used redundantly.
For example, an organization may own multiple tools that can capture and playback tests. Upon examination, it may be that different teams use different tools to automate the exact same functionality because they may find one tool easier to use than another. The result is a mixture of tests created and maintained in a variety of tools no other reason than convenience.
At this point, trying to consolidate the tools is not only labor-intensive, but also culturally difficult because people may be accustomed to using a certain tool.
Had the organization started with a basic test automation strategy, the multiplicity of tools and their associated costs could have been avoided. Perhaps a more unified tool framework could be achieved which would deliver better value for the money spent on creating and maintaining test automation.
-
It helps you focus on the effort
A test automation strategy also helps you focus on the effort at hand. Many people tend to underestimate how long it takes to get a successful test automation initiative in place and working. It is certainly more than just acquiring some tools and starting to create basic automated tests!
By defining the initial scope of the test automation effort, it becomes more evident when things start to deviate from the original scope.
-
It helps to communicate approach and build realistic expectations
One big risk in test automation efforts is that people, especially those in management tend to have misplaced expectations about the effort involved, the timeline of realized value and benefits, and the cost of test automation.
When assumptions and expectations are clearly communicated in a strategy, people have a better basis for expecting realistic outcomes.
-
It helps to see test automation as a project
People have a tendency to see test automation as a side activity to work on as time permits. In reality, getting test automation in place and working well, you need to treat it as a project with a life cycle.
For example, you need to identify the requirements for the solution, the design of the framework, how the tools or other means will be implemented, and how the solution will be verified and validated.
-
Identifying the unknowns
A very common thing in any endeavor is identifying what you don't know. Even more troubling are the infamous “unknown unknowns.” In other words, sometimes we don't know what we don't know.
To deal with the unknown unknowns, we must always be aware that we may encounter new information unexpectedly that must be dealt with. Contingency plans are one way to deal with new and emerging information.
In developing strategies, I have found that one of the great benefits is when people tried to define a point in the strategy but realize they lack the information to define something on paper. At first this may seem like a bad thing, but actually it's very beneficial because at least you know something that needs further investigation.
What Might Be in a Test Automation Strategy?
Basically, a test automation strategy can contain whatever you need it to contain as long as it describes the major objectives and considerations of test automation in your organization.
Possible elements of a test automation strategy are:
- Goals and objectives
- Major timelines
- Test automation methodology
- Technical and human resources needed (general)
- The technical environment and applications to be tested
- Other tools in use which require integration and interoperability
- Risks, contingencies and mitigation
- The level of test automation coverage desired (perhaps at stages of effort)
- Challenges
- Assumptions
- Measurements and metrics
- Proof of concept
- Pilot projects
- Future plans
How to Define a Test Automation Strategy
Collaboration is often the best way to define a test automation strategy. When created in a vacuum, a test automation strategy may fail to capture the needs and concerns of all the stakeholders. In this case, stakeholders may include test engineers, test analysts, test management, project management, developers, architects, executive management and anyone else who may interact with the automation tools or may benefit from the use of test automation.
The general nature of test automation strategies is a benefit for creating a simple, high level, and concise document. An effective way to get thoughts on paper is to brainstorm the points in your test strategy. This leads to a rough strategy which can be further refined and one or two cycles of review and revision.
Once defined and accepted, the test automation strategy can be distributed or posted and used as the basis for other test automation activities such as test automation planning, test automation creation, tool selection and many others.
If the test automation requirements change, then a revision of the test automation strategy and plan may be needed.
Automation Test Strategy Sample
To give an idea of what a test automation strategy might look like you can find an example at the link below as well as a template that you can adapt and use for your own purposes. In one format, a concise one-page template is used to capture the information. In the other format, a more traditional structured format is used. An effective technique is to use the one-page template to capture information, then elaborate in the structured format.
While the structured format may seem a little long, there are times when more words are needed to convey the strategy. Conciseness is still an objective and it is fine to tailor these samples to fit your context.
Automation Testing Methodologies
An important thing to document in a test strategy is which automated testing methodologies will be used. The methodology as indicated in the automation strategy should not be defined in detail, just noted. A more detailed description of the test automation methodology can be documented in a test automation plan.
Automated Testing methodologies may include:
- Capture playback test creation
- Creating automated test by direct scripting (such as creating tests in Selenium WebDriver)
- Modular and reusable test scripts
- Data driven testing
- Keyword driven testing
- Artificial intelligence
It is beyond the scope of this article to describe each of the above methodologies, but they are easily found in articles, books, and certification programs such as the ISTQB Advanced Test Automation Engineer Certification syllabus.
The Role of a Test Automation Plan
A test automation plan provides the tactical view of a test automation effort. The plan describes how the test automation strategy will be achieved in very tangible terms.
For example, test automation methodologies will be defined in more detail, in the scope of test automation will be defined more exactly. Schedules and timelines will have specific dates defined. Roles and responsibilities will be expressed in terms of specific individuals.
Test Automation Organization
A vitally important topic to be covered in a test automation strategy is how test cases and test suites will be envisioned.
The reason this is so important is because at the strategic level, basic parameters are set forth so that people can know and understand the desired structure of automated test cases, automated test scripts and procedures, and how those items will be organized into automated test suites.
Automated Test Cases
Automated test cases tend to be different from manual test cases in that they are expressed in tool language as opposed to natural language such as English. One notable exception to this is if a behavior driven development approach is taken, such as using the Gherkin language which expresses test cases in the structure of “given, when, then“.
In the following examples, we see the difference between a purely scripted test case (Figure 1) and one that uses the Gherkin type of expression (Figure 2).
Figure 1 – Selenium Secripted Test Case
Figure 2 – Gherkin Test Case
Automated Test Case Design and Implementation
Even though automated test cases will appear different from manual test cases, having an understanding of how a test is performed manually is a prerequisite to creating automated test cases, test scripts and test procedures.
Just to lay a quick baseline of understanding:
Automated Test Case
An automated test case can be seen as an atomic, detailed test of functionality. A common example is testing a login function with valid and invalid input. We see this in Step 2 of the test script shown below in Figure 3.
Automated Test Script or Procedure
An automated test script or procedure would be a step-wise scripting of a user process that includes login to accomplish a larger process such as logging into a mobile banking app. (Figure 3) Upon closer examination, each step in this test script or procedure could be a test case in its own context. It is an effective practice to build test procedures from detailed test cases.
The key to understand here is that to perform Step 2 (Test case 2), you have to perform Step 1 (Test Case 1) first.
Figure 3 – Test Script Definition in PractiTest
Automated Test Suite
An automated test suite is a way to organize related test cases or test scripts by functionality or other criteria. For example, a test suite could be defined for all account-related mobile banking test scripts or test cases.
Test Automation Risks
Risks abound in test automation. Some of the greatest risks are:
-
Unrealistic expectations
This can be seen in multiple ways. Some people think automation will solve many of the problems in testing such as meeting schedules and finding more defects. Others may think that test automation success can be achieved in the short term.
-
Picking the wrong tool
This is easy to do. That’s why good tool evaluations are so important. Sometimes, however, you can’t really know about the full capabilities and pitfalls of a tool until you actually use it in a project context.
-
Tool abandonment
If people find the tool hard to use, they may just give up on the tool and do manual testing instead. Three things can help: training, mentoring, and making tools use an integral part of your testing process.
Test Automation Best Practices
There is no magic wand solution for making test automation a successful reality. That’s why test automation is such a challenge for so many organizations.
There are, however, some basic principles that seem to have worked for many people. These include:
- Have a strategy to define and communicate major objectives and approaches.
- Get a commitment from senior management to fund and support the effort. It helps to have a champion on your side in senior management to help make your case.
- Measure the benefits of test automation to build your support.
- Build expectations that test automation takes time and money.
- Define your tool needs in advance and do an objective evaluation to compare potential tools.
- Perform a proof-of-concept, followed by a pilot project, before committing to any particular tool or vendor.
- Start small and grow. Instead of trying to automate complex functions, start with the mundane and simple functions. If you are feeling like a robot when you manually test something, that’s a good indication of something to automate.
- Roll out the tool incrementally to the organization. This gives a chance to capture lessons learned and to build support.
- Use a checklist to help remind yourself of things to consider in test automation and in creating a test automation strategy. A sample test automation checklist can be found here:
Test Automation Strategy Checklist
Conclusion
If a test automation plan paints the big picture of test automation, then a test strategy is the frame around that big picture. Trying to define a test automation plan without first defining a test strategy places you at risk of having the wrong plan.
For the time and effort needed to define a test automation strategy, the payback is large in terms of increasing your chances of test automation success.
By Randall W. Rice
Published on:
Randall W. Rice
Randall W. Rice is a leading author, speaker, consultant and practitioner in the field of software testing and software quality, with over 40 years of experience in building and testing software projects in a variety of domains, including defense, medical, financial and insurance.
You can read more at his website.
