To follow up on the previous article, Guide to Testing Automation with Robot Framework, I will go into more detail about the overall process of writing automated Robot Framework test cases and how to track automation in Jira using Xray.
Test automation may seem hard, but it gets easier with the right tools and an open and helpful community like the one surrounding Robot Framework.
If your team is using Jira, you can easily integrate testing results coming from the CI/CD pipeline. By using Jira and Xray as the central and single source for all testing efforts (manual test cases, exploratory testing, automated scripts), your team can assess the actual readiness of features before deploying them.
Testing in Jira and Xray
Xray is a test management tool that tracks all of your testing efforts in Jira, including results coming from test automation. Teams use Jira as the platform to manage their development (software/hardware) projects and track related tasks, including user stories and bugs.
If you think of testing as an intrinsic part of development, it makes all sense to also track it in Jira. So, how does it work?
It all starts with a user story or some sort of “requirement” that you wish to validate. This is materialized as a Jira issue and identified by the corresponding issue key (e.g. ROB-11).
You can promptly check that it is “UNCOVERED” (i.e. that it has no tests covering it, no matter their type/approach).
Create a Test Plan to define the scope of the testing that you aim to perform, group, and consolidate the corresponding results.
Besides the user story, you can also add the Test Plan to the Agile Board and assign it explicitly to a sprint. This increases the visibility of testing progress and helps close the gap between developers and testers.
How to implement automated test cases
A tester or software development engineer in test (SDET) could simply focus on implementing the automated test cases:
- The tester writes one or more test suites and corresponding test cases, using their favorite tool/IDE;
- You link each test case to the corresponding requirement/user story in Jira by adding its key as a tag;
- You can run tests locally or from the CI pipeline;
- Unique, non-duplicating test entities would be auto-provisioned in Xray, corresponding to each test case; tester could also, enforce the result to an existing test entity by specifying its issue key as a tag.
Let’s take the following .robot file as an example, which acts as a suite containing one test case.
A test issue will be auto-provisioned the first time you import the results. If you maintain the test case name and the respective test suites, the test will be reused on subsequent result imports. You can always enforce the results to be reported against an existing test - just specify its issue key as a tag.
Tags can also be used to cover an existing requirement/user story (e.g. “ROB-11”.) When a requirement issue key is given, a link between the test and the requirement is created during the results import process.
Otherwise, tags are mapped as labels on the corresponding test issue.
To import test results, simply submit them to the REST API with a POST request (e.g. curl), or use one of the CI plugins available for free (e.g. Xray Jenkins connector).
Using Xray, you can look at the Test Run details which include the overall result and also specifics about each keyword, including duration and status.
Tracking your results
With Xray, you can track your test case results throughout, including on the user story screen, agile boards, and on test plans.
On the user story screen
Right from within the user story issue screen, we now see one test (i.e. automated script) covering it. We can also see its latest result and how it impacts the overall coverage calculation for the user story; if the user story shows as “OK”, you know that all tests covering it passed, accordingly with the latest results obtained for each one of them.
On Agile Boards
With Xray's enhancements on Jira agile boards (e.g. Scrum boards), you can now assess the coverage of your user story taking into account the testing results.
You can also track the overall Test Plan consolidated progress on the Test Plan issue related card. Note that you could include Test Executions in the board if you wish, however, in a CI scenarios that could be counterproductive.
On a Test Plan
At the Test Plan-level, the entity that defines the scope of testing and tracks its progress, we can quickly assess the latest consolidated test results (i.e. the latest result obtained for each Test being tracked).
