We see a lot of teams working with automated testing, and many feel the need to take the next step in order to optimize and excel in their product release cycle. Prior to implementing DevOps, companies can experience a stall in releases and only make new ones every 6 months. This means that development efforts are not bringing value to users quickly enough.
For companies that work in regulated industries, it’s also critical that they set up reporting and traceability into their DevOps pipeline.
In this blog post, we’ll explain how a powerful DevOps setup that includes test automation and reporting can speed up the release cycle.
For many companies, one of their goals is to reduce the release cycle so that they can release every two weeks, or even more often. Automating the bulk of their tests, helps in two ways:
The automated tests will be one of the final steps in their new DevOps pipeline. The idea is that each software commit would trigger a build and a test cycle. If the test cycle is green, an automatic deployment onto their staging environment will follow.
Out of all the investments in DevOps, the automated tests are the biggest ticket item. Investing so much in automated tests is only justified in light of the business objectives.
A company that works in the medical industry and manages an app, for example, can’t ignore their regulatory obligations. To release a new version they also need to provide extensive test and traceability reports. These are the steps to generate these reports using Jira, Xray and Jira Snapshots for Confluence:
Let's take a closer look at how it works. Giving you all the tips and tricks to make this happen for your team too.
One of the reasons a company facing these issues would choose Xray is that its main benefit is native support for automated tests.
Xray has native support for many testing frameworks. Some companies use a mix of technology stacks, with each generating tests reports in another native format.
Xray offers an integration with JSON format that allows testers to import into Jira through the API. So, for each framework the test team can implement a small converter from the native report to that standard format. The effort involved in implementing this extra step is small and worthwhile.
This allows all tests to generate standardized reports that Xray supports. Importing these reports to Jira is the responsibility of the DevOps team. From their side, they end up having to support the import of one single format. So this is a perfect decoupling of responsibilities (between testers and DevOps engineers).
To streamline the tester's work, not only the test results but also the test cases themselves, need to be created in Jira by the pipeline. When a new test is first run and appears in the test result report, the import to Xray/Jira takes care of creating the corresponding test issue in Jira.
Inside Jira, Xray manages manual and automated tests with the same issue types. Both are test issues. The difference is indicated by the Test type: Manual vs. Generic. When the DevOps pipeline creates new Tests in Jira, it sets their Test type to be Generic.
A core topic for companies in the medical industry is their Test Coverage. From a regulatory perspective, they need to show that all requirements are covered by tests.
In Jira, we express coverage by tests as a Jira link from the test issue to a requirement specification issue. This link is manually created in Jira- both for the manual tests and the automated tests. It's a low maintenance high leverage activity. Once added, a link facilitates traceability reports for years to come.
The execution of a manual test in Jira is captured as a Test Run. Each time the pipeline imports a result of an automated test to Jira, this is captured as a Testrun:
Once the last cycle of tests is complete, the next step is to generate the test and traceability reports.
Xray comes out of the box with strong reporting capabilities. But some teams manage all their documentation in Confluence, and want to have a Confluence-based reporting flow. To achieve that they can install Jira Snapshots for Confluence, along with Snapshots traceability extensions for Confluence, which provides the integration with Xray.
A plus is that Jira Snapshots has a “no code” approach to customizing reports. The Product Managers can now generate reports exactly how they need them.
With this setup, Product Managers can prepare the report pages long before the release is ready. It’s then a matter of plugging into them the latest information from Jira. Once the manual test cycle is complete, the tester notifies the Product Owner. With one click, they refresh the snapshot on each page with the latest data, and the report is ready.
Companies can have all their documents approval process set up in Confluence. So when these two last reports are ready, they are immediately routed for review and approval.
This way, it takes a couple of hours from the last test run, to having signed reports and the Software approved for release.
After integrating with Xray, companies can end up with a optimized DevOps pipeline where:
With this pipeline in place, the time from code commit to releasing a new version is shorter and sweat-free.
After a while of successfully achieving an optimal release cadence with Xray, companies will see the ripple effects of this change everywhere. Not only are their users raving about the quick progress, also the team shifts into another state. Each team member feels that they have a much more immediate impact on the product.