Effective Coverage Analysis provides powerful insights into the status of what your team is building, so you can make smart decisions based on real-time analysis.
Coverage Analysis in Xray for Server and Data Center provides you the ability to analyze in real-time how requirements, features, user stories, and tests are, in a particular context.
In this post, we’ll show you how you can take advantage of Coverage Analysis in Xray so you can easily assess your readiness to deploy and release software with confidence.
Benefits of Coverage Analysis
- Quickly gain a high-level overview of the true status of your software;
- Answer critical questions like: “Are you ready to make a release?” “What is the status of certain components?” “Were all risks mitigated?”;
- Gain insights for Risk-Based Testing and help with risk monitoring;
- Assist regression testing to make sure previous features still work and avoid regressions, by having a clear overview of the status of previously implemented features on newer versions;
- Quickly evaluate the status of tests or requirements in different versions or in different environments;
- Quickly assess results against a very specific environment (e.g. browser, mobile device) or the joint results, taking into account the results obtained for all the relevant environments.
What are “status” and “coverage”?
How many times have you been asked for the "status of the release," "status of the sprint," "status of these tests," or the "status of these stories/requirements?" Normally these questions are made in a specific context, such as the "release" (i.e. product version). In this case, "status" is an ambiguous word but it can be simplified as coverage status, or simply coverage.
First of all, coverage tells us whether some item being worked on, such as a requirement, is "covered" by tests. However, this "on/off" information (covered vs uncovered) may not be enough to know its readiness. Why? Because you're lacking the actual feedback from the testing that provides the "actual status" about what you're implementing based on the latest testing results.
Therefore, coverage, as provided by Xray, intrinsically provides the testing feedback for the context that you need to evaluate. It's no longer "on/off " information; it's about dynamically asking the system, insights about how your tests or your requirements are, in a particular context.
Coverage exists to provide information and confidence
Coverage starts at the unit level, with "code coverage" which provides the first level of confidence to ensure that all statements and all branches are exercised with automated unit tests. While it is important to have unit tests to validate how you build your system, from a delivery perspective, you're also concerned with what you deliver.
Features are materialized as functional requirements and non-functional requirements. In order to validate them, we need to move up in the Test Pyramid and have integration, API, and UI or E2E tests, independently of using scripted (manual or automated) or exploratory approaches. Higher-level tests ensure that whoever is going to use your product has a fully working product.
Evaluating the (coverage) status of business-level requirements is crucial to understanding what can be delivered right away and what needs further attention.
While code coverage focuses strictly on the developer's immediate concerns, "requirement" coverage focuses on the quality of the deliverables that address the concerns of project managers, product owners, customers/end-users, and all team members working on delivering it.
How does coverage work?
First, coverage works in a hierarchical way. The coverage status of a requirement for some context is dependent on the status of the related Tests in that context. As "requirements" may be decomposed into "sub-requirements,” a Test may cover a requirement either directly or indirectly.
The status of a Test for some context depends on the latest results obtained for that Test (i.e. the Test Runs), in that context. The overall result of each Test Run depends on the recorded results provided at step level.
Where can you perform coverage in Xray?
Coverage Analysis can be performed in several different places, starting with the screen of the covered issue, so you can immediately assess coverage.
Note: Coverage Analysis in Xray Cloud works differently, take a look at Coverage Analysis in Xray Cloud to understand how you can take advantage of these insights.
Issue screen
Whenever working on a story, epic, or any deliverable which can be targeted by testing, it's important to assess its progress. While "progress" for some of your team members may mean "coding progress," you're actually concerned with overall progress.
Within the issue screen, the coverage status can be evaluated for the specified scope within the Test Coverage section. The calculated coverage status is shown on the right side.
Reports/Gadgets
From a project perspective, the Overall Requirement Coverage report/gadget can be used to have a high-level overview, as shown in this example for version v3.0 of the project.
You can see that some deliverables are flagged in "red," as being NOK. Others are flagged "green," as OK. There are a few that are uncovered and still many Tests that need to be run.
This report provides you the ability to group requirements by component, priority, or other fields such as risk level. Thus, you can analyze them from different dimensions. In the example, you can see that the "core" component has two requirements that have problems (i.e. NOK).
Agile Boards
You can easily track coverage information right within your existing Scrum or Kanban boards. The Agile Boards provide real-time quality insights directly from the place where the team tracks their work.
You can find out the overall results of the related tests for a story (i.e. OK vs NOK) or whether it's not yet covered at all. No need to jump to a separate report/screen and miss this precious information.
Learn how to configure coverage on your existing Agile Boards and how to track testing progress using them.
JQL
Searching information within Jira is done using JQL. JQL allows you to obtain Jira issues based on their properties and also by using specific JQL functions.
Xray provides an extensive set of JQL functions, where some of them are related with coverage and thus provide a mechanism to query the internal information stored by Xray.
One of them is the requirements() JQL function, which may be used to obtain all "requirements" (i.e. covered items) in project Calculator that are OK for v3.0, as shown in the following example.
|
issue in requirements('OK','Calculator','v3.0') |
Possible use cases for Coverage Analysis
- Evaluate if a release can be made or not;
- Analyze the project for regression purposes;
- Understand if a release has been tested in a specific environment and how it behaves in that specific environment;
- Evaluate which requirements are being addressed by a Test Plan and how they are performing in that restricted scope;
- Deprecate tests without deleting them.
Release with confidence with Coverage Analysis in Xray
Coverage can be measured in many different ways and for different reasons. It’s up to you to decide how you want to measure coverage and what that means for your team.
With Coverage Analysis in Xray, you can analyze the status of requirements, features, user stories, and tests and make smart decisions on your readiness to deploy.
Check out these additional resources related to coverage analysis:
- Coverage Analysis in Xray Server/Data Center
- Coverage Analysis in Xray Cloud
- Understanding coverage and the calculation of Test and requirements statuses
- Overall Requirement Coverage Report and Historical Requirement Coverage Report
