Test Automation Frameworks - The Complete Guide

Why do we use frameworks in test automation?

The first word to define is automation, it means that there is minimal human intervention during software testing. This is done to check how the application responds to different actions.

It is a fact that each and every kind of software must be tested before being launched on the market; and one way to do this is by using automation testing. It is an efficient method that allows to automate most of the testing efforts and there are many tools available to perform this type of test.

It usually happens that automation testing is still disconnected from manual testing, and results are reported separately. This makes holistic decision making, based on the overall testing, close to impossible. PractiTest is the solution to this problem. It allows combining all testing types in an integral way - from test creation to smart visualization of any test element

Nowadays, software testing is a must, and test automation frameworks are at the top of the software testing list.

Using a framework in test automation is essential because it specifically improves the automation testing team’s efficiency and test speed. The framework works as a support for automation that serves as a technical implementation guideline.

In other words, a test automation framework can be understood as a platform that is a combination of programs, compilers, features, tools, etc. that provides an environment where automated test scripts can be run. It makes it easy to perform tests and comprehensive reporting of test results.

It also helps to improve test accuracy, offers the benefit of reusing the code in various scenarios and helps the team to write down the test script in a standard format. Additionally, there are many other benefits of using automation framework testing as listed below:

Benefits of a test automation framework:

• Helps reduce time to market.
• Reduces maintenance costs.
• Ensures better quality products.
• Ensures higher test performance than manual testing.
• It is reusable.
• Delivers comprehensive bug reports.
• Identifies defects early.
• Ensures maximum test coverage.
• Reduces manual efforts.
• Saves a lot of QA time.
• Ensures quality continues to be present.
• It can be run anytime and anywhere.
• Allows access to data and resources anytime, anywhere.
• Supports parallel testing. Multiple tests at the same time are possible.
• Allows simultaneous testing of different types of devices.
• It is very reliable.
• Reduces the chances of making mistakes.
• Reduces challenges in synchronization, error handling and reporting.
• Reduces the need for more staff.

Testing framework tools

There are many framework tools available, here are some of them:

Robot Framework

Robot Framework is based on Python. It uses a keyword-driven approach to make tests easy to create.
It is a set of open source tools that can be used to test and automate software processes. Robot Framework is widely used as a test tool for acceptance test- driven development (ATDD) in a variety of applications.

WebDriverIO

WebdriverIO is a popular JavaScript test automation framework that runs on Node.js. It's extensible, compatible, feature-rich, and easy to install. Created to automate modern web and mobile applications and capable of supporting both desktop browsers and mobile applications.

Citrus

Citrus is an open-source framework written in Java that enables automated integration testing of message-based EAI applications. Incoming and outgoing messages are predefined by testers.

Selenium

This tool makes code maintenance easy and efficient. It is based on the JavaScript framework and it is able to run tests directly on the target browser, drive the interactions on the required web page and rerun them without any manual intervention.

There are mainly three type of frameworks created by Selenium to automate manual test cases:

• Data Driven Framework in Selenium
• Keyword Driven Framework in Selenium
• Hybrid Framework in Selenium

Cucumber

It is an open-source software testing tool written in Ruby. This tool provides a way to write tests that are easy to understand for everyone, regardless of their technical knowledge. It works through two important files: the feature file which contains code written in plain English. And the step definition file which contains the actual code written by the developer.

Cypress

Cypress is an end-to-end JavaScript testing framework that provides a visual interface to indicate which tests and which commands are taking place, passed, or failed. It is fast, and easy to set up and run

PractiTest

With PractiTest, it is possible to run, control, view and manage both automation and manual testing. It provides a complete visibility for the entire process in one place. PractiTest's xBot test automation framework is an application which allows a user to run (or initiate a run by scheduling) automated test scripts on a remote machine from the PractiTest UI.

Test framework development

When creating a test automation framework, the following main points should be considered:

Building an automated testing framework can be a complex process and time-consuming. It must contain the language that will be used for the codes and scripts, the actions that are taking place and the tools that will be used.

Some of the recommended steps to build an automated testing framework are as follows:

• Start by structuring the test assets in a folder structure. It will be useful to keep the files separated, such as name mapping criteria, tests, scripts, and other files that are considered necessary. This will allow team members to refer to them quickly and will ensure that tests are stable when updates are made.
• In this step it is recommended to explore the application, study it and know it in depth. Perform exploratory tests to get an idea of how internal system workflows are configured. This exercise helps to know how the application works. Then it will be necessary to create a system or perform an update. Depending on the tool that has been chosen, it may be necessary to create basic name mapping properties or write scripts for the actions required to identify objects. It will be essential to take notes on the requirements that will be necessary and how the automation assets will test them.
• Next, collect the data that is planned to be used for the tests and configure the environments. The key is to create configurations that can be run in more than one environment. It’s time to use event handlers. It is a function composed of code that triggers the scripts when an event is taking place. They are the instructions for how a system should work. As for the test data, the framework should save the data separately from the tests. Store data in repositories and keep the properties and references generic.
• At this step, set up a smoke test project to verify utilities. It validates that the most important functions of an application are working well and determines if additional testing is needed.
• By this stage, create shared utilities for common user interface (UI) actions. These are the basic building blocks of tests, which can be put together to form the test logic.
• This is the step in which it is necessary to build and manage verifications, applying the same logic that was used when structuring data. Verifications should be optional and also shared. The different input utilities should be able to accept data objects created in previous steps so the elements can be linked.
• Finally, it will be necessary to create a logging and reporting mechanism. Notes should be taken throughout all the steps mentioned above. Messages should not be in technical language, everyone should be able to understand them and the errors do not have to be guessed. Reporting should also be automated.

Different types of framework used in automation testing

In the following image the most popular types of framework can be appreciated:

Figure 1 – Different types of framework in automation testing

Let's take a look at these types of framework used in software automation testing in more detail:

Behavior Driven Development Testing Framework

Features:
The main characteristic of this framework is that it facilitates the participation of everyone, since it does not require a programming language. A normal language is used to create test specifications. Some of the tools available for BDD testing framework are: JBehave, Cucumber, etc.
Due to the simple language, everyone can understand what is happening in the software project. This helps and improves communication between the entire team. Users can work with multiple test data without much intervention in the software. This increases code reuse and saves a lot of time.

pros:

• Programming language is not required. This allows every person to participate actively.
• It increases collaboration between the tester and the developers during the project. This increases code reuse and saves a lot of time.

Cons:

• Technical skills are needed.
• Test driven development experience is required.

Data-driven Testing Framework

Features:
This type of framework separates test script logic from test data, it stores both the input and the expected output results. Then stores the test data set in external files or resources like MS Excel sheets, XML files,SQL database, etc. It contains navigation that runs through the program and covers both reading of data files and logging of test status information.
Reusable test scripts are created to test different data sets.

Figure 2 – Data driven testing framework

pros:

• Changes made to the test scripts do not affect the test data.
• It supports multiple data sets.
• Saves time by running more tests.
• Since the data is kept separate from the test script, a variety of test scenarios can be performed simply by changing the test data in the external data file.
• No need to encode test data.
• Data input / output and expected results are stored as easy-to-maintain text records in the database.
• Greatly reduces the total number of scripts required to cover all possible combinations of test scenarios. Therefore, a lesser amount of code is required to test a full set of scenarios.

Cons:

• High setup time for both Test Scripts and Test Data.
• Bug-fixing may be difficult due to the lack of exposure the tester has on a programming language.
• Testers must have excellent technical knowledge.
• It is necessary to have experienced automation testers to design the framework.

Modular Testing Framework

Features:
In this type of framework, testers can divide the application, which is under test, into different modules and create smaller independent tests. Specifically, an abstraction layer is created so that the components are hidden from the application under test.
The next step is an incremental and non-incremental approach. There, independently, the modules are tested and then the entire application is tested as a whole. This type of abstraction concept ensures that the master module is protected from changes made to individual tests.

Figure 3 – Modular testing framework

pros:

• It ensures the division of scripts that leads to easier maintenance and scalability.
• Reusable.
• Separate test scripts can be written.
• Changes made to one module have little or no impact on the other modules.

Cons:

• Requires technical knowledge to set up the framework.
• It takes more time to analyze and identify reusable flows.
• When updating the code or script, the fact that the test script has embedded test data can be a problem.
• Since data is still hard coded in the script, the same test case cannot be run for multiple data values without changing data after each run.

Hybrid Testing Framework

Features:
The Hybrid Testing Framework is a combination of two or more frameworks. This combination is created to take advantage of the benefits of the different frameworks and mitigate their weaknesses.

Figure 4 – Hybrid testing framework

pros:

• Leverages the advantages of the different related frameworks.
• It provides flexibility: framework independent of system or environment under test.
• It allows reusing test cases and library functions.
• It improves testing efficiency due to higher code reusability.

Cons:

• It requires technical knowledge and experience to build a hybrid framework from scratch.
• Increased Complexity.

Keyword Driven Testing Framework

Features:
It is also known as table-driven testing and it is suitable only for small projects or applications. Data tables and keywords are used to explain the actions to be performed on the application under test. Keywords are specified in the project's Excel sheet.
In other words, multiple keywords are created and a function is associated with each of these keywords. It allows less prepared testers to work on the test scripts.

Figure 5 – Keyword-driven testing framework

pros:

• It is not necessary to be an expert to write test scripts. Once the libraries are ready, it becomes very easy to write the actual test script steps in excel sheets.
• Keywords are reused across multiple test cases. High reusability can be achieved by re-using across multiple test cases.
• Offers greater flexibility when it comes to maintenance and fixing of bugs.

Cons:

• High development time.Takes time to build and stabilize the framework and to create the function libraries.
• Due to the flexibility the test cases become longer and more complex.
• High automation knowledge required.
• The initial cost of setting up the framework is high. The benefits of this can only be realized if the application is considerably big and the test scripts are to be maintained for quite a few years.

Library Architecture Testing Framework

Features:
This testing framework identifies tasks that are similar within the test script. Then, testers put together these similar tasks into different groups based on their functions. Once the functions are grouped, these groups will be stored in a library. The test scripts reuse these libraries to create new test cases.
The Library Architecture Testing Framework is useful when the application has similar functionalities in different parts of the application.

Figure 6 – Library architecture testing framework

pros:

• It introduces the high level of modularization which leads to easier and cost-efficient maintenance and scalability too.
• It has a high degree of reusability, as common functions are developed that can be used efficiently by the various test scripts in the Framework.
• The automation scripts are less costly to develop due to higher code reuse.

Cons:

• Because the test data is lodged into the test scripts, changes should be made to the test script whenever changes are made to the test data.
• Coding skills are required.
• More time is needed to plan and prepare test scripts.
• Due to the introduction of libraries, the framework becomes complicated.
• Test Data is hard coded within the scripts.

Linear Scripting Framework

Features:
Commonly named as 'Record and Playback', this testing framework has a basic level. It is based on the concept of record and playback mode in a linear fashion.The creation and implementation of test scripts are done individually. This type of framework is used to test small applications.
Manually, the tester captures each step and inserts checkpoints in the first round . Then the recorded script is played back in the subsequent rounds to carry out the tests.

pros:

• Coding knowledge is not required.
• A quick way to create test scripts.
• Expertise in test automation is not necessary.

Cons:

• Lack of reusability when the user interface is changed.
• It doesn't allow running with multiple data sets.
• Maintenance is high and very tough when the user interface and feature enhancements happen.