In modern development practices like Agile and DevOps, testers work in compressed test execution cycles, and automate regression tests to reduce time, efforts and costs involved in testing. But even so, testing doesn't advance at the same speed as development.
The most common form of test automation is N-1 Sprint automation in which QA teams test one sprint behind development. The delay can be due to many reasons including application unavailability, changing requirements, and limited time due to short release cycles. This often causes a barrier for testing at the speed of Agile/DevOps because QA continues to play catch up.
To keep test automation from blocking timely releases, testers must adopt in-sprint test automation. In-sprint automation aims to automate user stories at Unit, API and UI levels - But how do you do that?
In this article, we share an effective in-sprint automation strategy using a BDD and TDD approach at the Unit, API and UI level to keep testing and development working in the same sprint.
In an Agile/DevOps setting, quality is a team responsibility. Practices like Test-Driven Development and Behavior-Driven Development allow everyone on the team to contribute to quality.
Test-Driven-Development (TDD) is an engineering practice that adopts a “Test-First” approach in which developers write unit tests first before functional code. In the TDD approach, the developer will not write functional code until the test exists and it fails (as the function is not developed).
Once the test is created, the corresponding functional code is written such that the tests pass. Some developers think that TDD slows down the development process. I think people who say this have a limited understanding of TDD. The TDD discipline may look difficult to beginners and can be time-consuming to start with, but with little practice developers can get into speed.
TDD, in fact saves times, finds errors early, allows safe refactoring, and increases test coverage and quality.
Behaviour Driven Development (BDD) also advocates to write tests first, in this case, acceptance tests followed by unit tests. BDD is an extension of TDD which focuses on improving shared understanding and communication between the business and technical teams, ensuring business value is delivered by working through examples. These examples are then formalized and automated using BDD tools like Cucumber.
Cucumber uses Gherkin language to define test cases. This syntax promotes a shared language that allows developers, testers, and product owners to be on the same page and have a common understanding of the user stories.
When practicing BDD, the acceptance criteria is automated and is a part of Definition of Done. In agile, acceptance criteria constitute the “Definition of Done”(DoD) and defines what must be done to complete a user story. The acceptance criteria ensures that everyone on the team is on the same page in terms of what is expected from a user story. A story is not complete until the acceptance criteria have passed.
To achieve in-sprint automation, the acceptance criteria for the new features are automated within the sprint. Once the sprint ends, the automated acceptance tests become a part of the regression test suite. BDD enables teams to shift left and makes in-sprint automation look effortless.
Understanding Mike Cohn’s test automation pyramid helps us formulate an effective in-sprint test automation strategy. At the base of the pyramid are the “Unit Tests.” Unit tests are the fastest to execute and the least expensive to create. Next are the “API tests” which test services separately from their user interface. “UI tests” sit atop the test automation pyramid. Costs to create and maintain, execution time, fragility and coverage of tests increases as we travel up the pyramid.
The Automation Pyramid clearly shows that you should push tests down to the lowest level possible for a higher ROI. The test automation pyramid suggests automating UI tests as little as possible because they are brittle, time-consuming and expensive. Thus, you must push as much testing as possible down the pyramid (to the unit and API layers) to have better test coverage with less overhead to create, execute and maintain the tests.
Unit tests form the foundation of a solid test automation strategy. The unit tests are written in the same programming language as the application. You can write unit tests relatively quickly and give fast feedback on code quality. As you can see in the pyramid above, unit tests should constitute the majority of your automated tests.
In the traditional development process, unit tests are not written later leading to low coverage and technical debt. The problem with this approach is that you won’t find errors until the functionality is fully complete, and that is already too late. I have seen many teams who only estimate for the time required to develop a feature and do not account for unit tests during the sprint planning.
Higher on the pyramid is API testing which is more suitable for test automation than UI testing. API automation lets you validate the core logic without being dependent upon the UI. API tests are easy to automate within the same sprint.
With API automation you can start testing your application early - even without a UI. This helps to identify and fix issues early in the development lifecycle which would otherwise be expensive to fix when identified during UI testing.
UI tests are highest on the pyramid and they are the hardest to automate. UI test automation usually spills-over to subsequent sprints because you wait to finalize development and design. Automating UI test cases within the sprint can be challenging but you can do it.
By way of example: Assume that you need to test the “login screen” in your first sprint. You haven’t completed development but you do have the wireframes. As long as you have a relatively stable design, you can start to automate UI tests without waiting for the development to complete.
A more proactive approach is to discuss with the developers what locators they plan to use for the username, password, and login button during the sprint planning. In-sprint UI automation can be handled smartly by coming up with a contract for mapping locators at the beginning of the sprint.
This way the automation engineer will know what the locators for the login screen will be without being blocked by the developer to complete the feature. This approach will help you shift-left and develop tests in parallel to development, enabling you to ship releases faster.
Testing automation shouldn’t be a bottleneck. With an effective test automation strategy, you can prioritize your tests so they don’t slow your team’s development velocity. Master in-sprint automation by adopting the right mix of Unit, API and UI test automation (in that order).
Collaborate with developers during sprint planning to take into account not only the time to develop features but also to test them. Implement Test-Driven and Behaviour-Driven-Development strategies to ensure that testing is interwoven throughout development.
Finally, adopt a quality mindset as a team and collaborate with developers to incorporate approaches like BDD and TDD to speed up test automation and release velocity.
With Xray, you can test in BDD with Gherkin based frameworks like Cucumber, SpecFlow and Behave. Xray’s tight integration with Jira enables full collaboration between testing and development under the same toolset.