The complete guide to test automation: best strategies, frameworks and tools

Test automation is here to stay. That’s because good test automation reduces manual effort through repeatable, stable processes, increases ROI on testing, accelerates testing and feedback, and in the long run reduces business expenses. 

If your company is undergoing a digital transformation or you’re simply ready to take the plunge and make the initial effort involved, count this as your starting point. 

Let’s dive into the basics of test automation and the strategies, frameworks and tools to get you started.

What is test automation?

In short, test automation is a software testing technique to test and compare the actual outcome with the expected outcome. To achieve this, testers write test scripts and automate them using a test automation tool. You can use test automation to automate repetitive tests or ones that are difficult to perform manually. Good automation makes testing faster, more systematic, and reduces human error.

The goal of test automation is to increase the effectiveness and efficiency of testing. This includes: to reduce the number of test cases that testers have to perform manually as well as those that are challenging to perform manually, therefore saving time and effort.

How can you benefit from test automation?

The main objective of test automation is to simplify efforts as much as possible with a minimum set of scripts. Automated tests provide frequent results and data points. 

The most important and interesting benefits of test automation include: 

1. Increased coverage

The more tests you perform, the more code, functionality, and non-functionality you can cover. This is where testing automation significantly increases your code coverage and overall leads to an increase in software quality, since you are able to significantly increase the amount of code you are covering. More coverage leads to faster time to market and increase in business competitiveness.

2. Faster results and feedback

Testing manually takes time and effort. When you implement test automation, you can receive results within seconds and have almost immediate feedback into your results. When your test automation results are visible inside of your testing and development toolset, like Xray and Jira, your entire team, including testers and developers can act on the results from the automation and quickly make the changes. This also enables CI/CD pipelines and provides short feedback loops.

3. Better ROI on testing

Testing automation takes the redundant and repetitive tasks that testers do manually and automates them. To manually test a system takes an enormous amount of resources. When automation is in place, testers have time to focus on other types of testing like exploratory testing which uncovers new test ideas and risks that you would otherwise never have thought of. It allows testers to use their skills, knowledge and experience to explore the system and find defects and bugs that otherwise would have gone unnoticed.

4. Business return after initial investment

Finding defects and bugs sooner, is proven to save money, as the cost of catching a bug later in the development cycle is 40x the cost of finding it during coding. Finding skilled testers that can set up a testing automation framework can take time as well as the initial time and effort invested in building the framework, collecting good test data, choosing and investing in tools, and building the infrastructure. However, in the long run, test automation can potentially increase the number of bugs you find sooner in the SDLC, since you can run multiple tests simultaneously.

Types automated tests: functional vs. non-functional

One way of grouping the different types of tests that can be performed with test automation, is by considering whether they address functional or non-functional requirements.

Non-functional tests

Functional testing is based on requirements and business scenarios. These include the initial requirements that are drafted during the planning phase of the software development lifecycle.

It’s important for testers to be involved in the software development lifecycle from the start when requirements are being discussed because they understand functionality and how it can go wrong. Functional testing validates these given requirements and essentially tests the functionality, making sure that the system performs as expected in the real world and for real users. 

Types of functional tests primarily include:

• Acceptance Testing;
• System Testing;

Besides these, we can also consider: 

• Integration Testing
• Interface Testing
• Smoke Testing
• Unit Testing
• And others..

Which tests should you automate?

Now that you’re aware of all of the different types of tests that you can automate, you’re probably thinking “Which tests should I automate first?”. With so many possible tests to automate, which tests are going to give us the quickest ROI on our efforts to automate and act as the backbone of our automation strategy?

In his book, Succeeding with Agile, Mike Cohn presents his test automation pyramid, which is a great starting point for discussing a strategy that can help you decide which tests to automate first and where to focus your test automation efforts on.

Use the test automation pyramid as the starting point for your test automation strategy

As you can see in the Test Automation Pyramid, the higher up you go on the pyramid, the more it costs to maintain, execute and cover the tests. Therefore, the smartest way to go about it is to push tests down to the lowest level possible for a higher ROI.

Instead of focusing on UI tests which can be brittle, time-consuming and expensive, focus on automating Unit and API tests. The majority of your testing automation should be done in the Unit and API layers in order to have better test coverage with less overhead to create, execute and maintain the tests.

However, the test automation pyramid is not a strategy by itself and should not be used blindly. You should discuss with your team and agree on the most important risks and how you can use automation to tackle them.

What are UI, Unit and API tests?

UI tests

UI tests are highest on the pyramid and they are the hardest to automate.  UI test automation usually spills-over to subsequent sprints because you wait to finalize development and design. Automating UI test cases within the sprint can be challenging but you can do it.

Unit tests

Your Unit tests form the foundation of a solid test automation strategy. The unit tests are written in the same programming language as the application. You can write unit tests relatively fast and give fast feedback to developers on the quality of the code so they can make changes.

API tests

API automation lets you validate the core logic without being dependent upon the UI. API tests are easy to automate within the same sprint. With API automation you can start testing your application early — even without a UI. This helps to identify and fix issues early in the development lifecycle which would otherwise be expensive to fix when identified during UI testing.

How to build a test automation framework

Continuous Integration is a practice of DevOps, which merges all of the developers’ code to a central location and validates their merges with an automated build. In order for Continuous Integration to be successful, it must have automated testing. 

Automation will provide you with constant feedback, so you can have a smooth process and keep the flow of “build > test > deploy.”

Test automation is such an important element in the CI/CD pipeline because it ensures that the testing you’re doing is integrated with the rest of the cycle, and that it is reliable and stable. Without automated testing, a DevOps workflow won’t be successful.

Use this free 10 step checklist to build a mature Test Automation Framework:

Stay agile and adopt in-sprint test automation with BDD and TDD

To ensure that your testing and development teams are working in an optimized and agile workflow, it’s important to make sure that they are working in the same sprint. 

You can use a methodology called Behaviour-Driven Development (BDD) and Test-Driven Development (TDD) to implement useful test automation within the same sprint and keep testing and development working together. 

Behaviour-driven development (BDD)

Behaviour-driven development, also called BDD, is a methodology that involves developing software through examples and provides collaboration between different stakeholders on the team.

Behaviour-driven development also advocates to write tests first, in this case, acceptance tests followed by unit tests. BDD is an extension of TDD which focuses on improving shared understanding and communication between the business and technical teams, ensuring business value is delivered by working through examples. These examples are then formalized and automated using BDD tools like Cucumber.

To achieve in-sprint automation, the acceptance criteria for the new features are automated within the sprint. Once the sprint ends, the automated acceptance tests become a part of the regression test suite.

Test-driven development (TDD)

Test-driven-development (TDD) is an engineering practice that adopts a “Test-First” approach in which developers write unit tests first before functional code. In the TDD approach, the developer will not write functional code until the test exists and it fails (as the function is not developed). 

Tools to use to automate BDD and TDD

There are many automation tools/frameworks that enable BDD in test automation, and Cucumber is one of the most popular. In Xray, you can map user stories using BDD, and then automate testing in BDD using Cucumber or Robot Framework. 

Robot Framework is another open-source framework that you can use to automate BDD. Robot Framework can also be used for teams that adopt BDD and test automation; it supports Gherkin syntax for specifying test scenarios.

How do you choose the right tool for test automation?

There are many open-source tools that are used to facilitate testing, test automation and automation in general. We all know Selenium WebDriver, Jenkins, Maven and Docker, others include JUnit, RSpec, and Robot Framework.

New testing frameworks and tools appear every single day, so how do you know what works best for you? Are commercial tools better than open-source alternatives or the other way around? There is no clear answer and “it depends” highly on your needs.

• What will you use the tool for?;
• Will it solve problems you really have?;
• Can it optimize your work?;
• Does it match your team’s skill set?;
• Do you need SLAs?.

Teams are unique and should use whatever tools they want in order to be more efficient, productive, and happy. The secret is to take advantage of what both commercial and open-source tools have to offer. 

Integrate your test automation tools into your CI/CD pipeline and use Xray and Jira as the single source of truth

If your team is using Jira, you can easily integrate testing results coming from the CI/CD pipeline. By using Jira and Xray as the central and single source for all testing efforts (manual test cases, exploratory testing, automated scripts), your team can assess the actual readiness of features before deploying them.

Teams using Jira can easily track their Robot Framework test automation results in Jira and use Xray as the single-source of truth. It’s as simple as submitting the results to Xray and letting it do the rest. Submit test automation results using a CI tool like Jenkins, TeamCity, Azure DevOps, and Bitbucket.

Make test automation work for you

Like with every adoption of a new software, method or tool, it’s going to take some time for you and your team to get used to. However, test automation is worth it in the long run since it will dramatically improve your process, save you time and resources, and ultimately lead to better quality software.  

Looking for more resources to get started?

• Download our free eBook and use test automation to achieve your goals;
• Discover how TCS scaled their test automation using Xray and Boozang;
• See all of the Integrations for test automation with Xray;
• Learn how to centralize test automation in Xray and Jira.