In 2019, software testing was a $40 billion dollar industry. By 2026, it’s predicted to grow to $60 billion. This signals a paradigm shift in the importance of software testing and quality assurance. With an increased amount of testing and more advanced test strategies, it’s crucial to have a powerful test management tool.
Xray, a Jira native test management app, provides you with the structure to organize, plan and report on the progress of your testing and ensure your readiness to deploy.
In this guide, you’ll learn about the most important benefits and features of Xray, including full requirements traceability, easy collaboration between developers and testers, integrations with the leading automation frameworks like Jenkins and Robot Framework, and advanced reporting to level up your software test management.
So let’s get started.
What is Xray test management for Jira?
Xray is the fastest growing cloud app for Jira and a Success Story in the Atlassian Marketplace. It’s a full-featured tool that lives inside, and seamlessly integrates with Jira. Xray aims to help companies improve the quality of their products through effective and efficient testing.
More than 4.5 million testers, developers and QA managers trust Xray to manage 100+ million test cases each month. Xray is a mission-critical tool at over 5,000 companies in 65 countries. Take a look at our success cases with Bosch, Swiss Federal Railways and TomTom.
With Xray, managers can enhance Jira agile boards by tracking the requirement status and test execution progress in real-time. You can also customize and generate reports about your Jira data using the Document Generator.
Integrate with your favorite Continuous Integration tools like Cucumber and Jenkins through our built-in REST API.
Why do you need a test management solution in Jira?
For companies that still use traditional methods of test management like Word and Excel, you probably find it challenging to perform all of the necessary functions of proper testing and management.
Other companies that use legacy test management apps which are commonly outside of the project management tool, create silos between testing and development. Developers don't want to be involved in testing and testing is left as a siloed operation separate from the rest of the software development lifecycle.
Without a test management tool, you can not ensure proper quality management, team collaboration and a focus on testing and the quality of your products will inevitably suffer.
Here's how Xray can help you build quality in throughout your entire software development lifecycle:
Ensure you meet all requirements
Xray helps you meet your testing requirements in different environments on any device, in any browser, and at any testing stage. The app also provides traceability between requirements, tests, and defects while assisting with the root cause analysis that identifies the real cause of a problem or a bug.
Improve productivity with automated testing
In Xray, tests can be manual or automated. With automated tests you need to choose your test automation platform like Cucumber, Jenkins, Robot Framework, or Selenium amongst others, and connect it with Xray using the REST API. This helps you reduce manual work by automating test cases and covering your story with the right tests.
You can also review the code of an automated test to check what it’s doing, directly from your issue tracker. This way, testers and developers can easily track their work and all other activities including testing ones in the issue tracker or agile board.
Prioritize your time with Risk-Based Testing
A test management tool like Xray helps you prioritize test cases by giving priority through risk-based testing. You can ensure compliance for auditing, which is especially important for companies in regulated fields like medical device manufacturers.
Before releasing a software product, the test management tool can systematically ease the go/ no-go decision process and provides everyone involved in the software development lifecycle (the testers, developers, analysts, or project managers) with access to the testing progress.
Ensure proper organization and planning
A proper test management tool should define specifications/ authoring of the test cases, including the composition of the test cases and the reusable preconditions, and you can easily do that with Xray. You should be able to a clearly organize your test cases so they can be easy to find and execute.
In Xray, tests can be organized in multiple ordered lists or in a hierarchical structure with folders called the Test Repository. With the Test Repository you can intuitively organize your tests for maximum visibility. Next comes, test planning which you can do using Xray’s Test Plan Boards.
Get advanced reporting to make smart decisions
Once testing gets going, there are many things to track, like the execution status, different versions, and requirements coverage. You can access all of this data using Xray’s advanced reports and gadgets. Share the reports with your team and other stakeholders to give an overall status on testing progress.
Once tests are running, you can evaluate coverage of the requirements/ stories through Xray’s reports and gadgets. Xray’s real-time dashboard will give you all of the information you need to make decisions on releases and your readiness to deploy.
What are the core features of Xray?
1. A Jira native app that integrates testing into the software development lifecycle
Xray is a Jira native application. It is the only test management app in the Atlassian Marketplace to have a native integration that uses Jira issue types for your testing activities. When you install Xray, you’ll see new issues for: Tests, Preconditions, Test Set, Test Repository, Test Plan, Test Execution, and Test Runs.
When you integrate your testing into Jira, the leading agile software project management system, you merge development and testing together under one integrated toolset. This breaks down silos between testing and development teams which are otherwise working in separate tools. When your testing and development teams use the same tool, you experience more collaboration, transparency, and communication.
2. Full requirements traceability between requirements, tests, defects and executions
One of the most powerful features of Xray is its requirements traceability. Meeting your requirements is one of the most important aspects of software testing.
With Xray you get full traceability between requirements, tests, defects, executions. This feature gives you immediate visibility into your testing progress and readiness to deploy.
For companies like MyndTec, a medical device manufacturer, who has to meet strict FDA and ISO audits, they need to ensure they meet all of their requirements and provide proof of the requirements traceability. With Xray, this is as simple as generating the Requirements Traceability Report.
3. Integrates with leading automation frameworks like Jenkins and Selenium
Xray has native integrations with test automation frameworks like Cucumber, Jenkins, Selenium, JUnit, NUnit, and Robot Framework to improve your team’s efficiency.
With the popularity of Continuous Integration and Continuous Delivery practices, automated testing is taking up more testing time. This is no surprise, since automated tests provide time-saving benefits over manual testing. Automation makes the testing process faster, systematic, and reduces the effort associated with the execution and validation.
Automated tests are implemented as code. Usually, they are executed during the Continuous Integration process, triggered by code changes, or on a timely basis.
4. Provides advanced coverage analysis
Understanding coverage for different requirements is a powerful metric that will guide your decision making on readiness to deploy and the status of a release.
Coverage is a key feature of Xray since it gives you the ability to analyze what the status of your issues are. You can analyze coverage by stories, requirements and even bugs, making it a useful metric that you can customize to your needs.
Coverage analysis provides real-time information on the status of the coverable issues like requirements, so tests and coverable issues can be analyzed from different perspectives or scopes.
5. Integrates with your CI/CD pipeline through powerful API
With Xray, you can easily incorporate testing into your CI/CD pipeline. DevOps and its components, Continuous Integration and Continuous Development are a way to speed up the rate of software delivery by integrating all of the software development functions together.
Automated testing is one of the most important parts of any CI/CD pipeline because it ensures that testing is reliable, fast and integrated. Without the proper automated tests that run fast and have good coverage, there can be no successful DevOps workflow.
6. Advanced reporting and gadgets to make smart decisions
Xray provides built-in reports that can be used to analyze your project in different ways, including testing progress, current requirement statuses, and others. With these reports you can look at the big picture, like the Test Execution completeness, and hone in on more detailed reports like the Test Sets Overview and Test Plans overview.
Jira itself has reporting gadgets that can be easily added and configured to use Xray custom fields and generate reports such as lists of Test Executions and Test Sets with their current status, among others.
7. Supports agile methodologies with full visibility and transparency
If your testing and development teams work separately, you know the silos that this causes. Without the proper tools to collaborate, there can be a lack of understanding between testers and developers. A lack of communication between the teams can lead to confusion on the status of everyone’s work, as well as poor requirements coverage.
With Xray, your testing and development teams have full visibility into each other’s work directly on Jira’s agile boards. With this agile component, your team has better collaboration, visibility and transparency. This is an agile mindset that all modern companies should be adopting in order to stay competitive in the ever-evolving market.
How to use Xray: A step by step guide
Xray seamlessly integrates with Jira and has Jira native issue types. When you install that app, you’ll have 7 new Jira issue types that you can use to maximize your testing activities.
Because Xray uses issue types for mapping test concepts, you can organize your projects to best suit your organizational needs. You can keep your current projects separate from Xray, or combine them together.
The typical testing life cycle follows a set of phases that do not necessarily need to be executed sequentially.
For each testing phase, you would use the following issues:
- Plan phase: Test Plan issues;
- Design phase: The specification can be done with Pre-Condition and Test issue; types: The organization of the tests can be done using Test Sets;
- Execute phase: Test Execution issues;
- Report phase: Test Execution issues, along with the built-in requirement coverage reports and some custom ones using Jira tools.
Here’s how to kick-start your testing project in Xray:
Specification
To start, specify the requirements for your project. Normally, these requirements need to be reviewed by stakeholders. Once you have your requirements, you want to create tests to meet these requirements.
Creating tests in Xray is easy. Hit the “create” button in the menu bar and choose the “Test” issue type. In the test details, you’ll want to specify if it’s a manual, generic or automated test. If the test is manual, you’ll write the description, assign a version of your software and set the assignee and the reporter.
If you want to create an automated test, you can select the Test Type as Cucumber. On Cucumber Type, select either Scenario or Scenario Outline as test type. On Cucumber Scenario, enter the Cucumber Scenario clauses in Gherkin.
Keep in mind that automated tests are executed through an external tool which controls the execution of tests and the comparison of actual outcomes to predicted outcomes.
Organization
Organizing your tests is a crucial part of your daily testing activities. Since your project may have hundreds or thousands of tests, it is imperative that you organize them efficiently so you can find relevant tests quickly. We recommend you organize your tests using Xray’s Test Repository, which lets your team organize all their tests into a hierarchy of folders and sub-folders to improve test execution and visibility of results.
Another way to organize your tests is using Test Sets, which is a collection of tests with detailed description or goal. Test Sets are simple, flat lists and are grouped in a logical way. For example, you can group together all of your security tests or high-level business features in a Test Set. There are many combinations and ways to organize your tests, you can figure out what works best for your organization.
Planning
Planning is the activity where you decide your testing strategy, which requirements you want to validate, and how would you validate them. Depending on several factors, you may want to prioritize some of the tests, or give more focus on some requirements.
To create a Test Plan, go to the menu and click on “Test Plan.” From here, you can create a new folder, and select all of the Tests you want to track for this Test Plan, for example, in a given version. Once you begin executing your tests, you’ll be able to see the results of your execution in a consolidated view in the Test Plan Board, giving you a quick overview of your testing progress.
Execution
When you are executing a test, you will fill the following fields: Begin Date, End Date, Revision, and Test Environments. A Test Execution is validated if all the Tests that verify the specific environment target passed. In order to execute, monitor the progress and report the results, each Test has its own Execution page details.
To execute a test, click the “Execute Test” button in your test case. This will take you to the Xray execution screen. If you’re doing a manual test, you will execute your test then come back and set the Execution Status. This will describe the testing progress. You can set the status for a Test Run as TODO, EXECUTING, FAIL, ABORTED and PASS.
You can also attach evidence as screenshots and notes on your test run. You’ll be able to see the status of your executions on your Jira agile board.
Common use case
The Swiss Federal Railways is the backbone of the Swiss public transport system. Testing is a crucial element for SBB, and they have more than 600 applications. As a railway company they test both software and hardware, and they have over 100.000 test cases and 1.000+ trained employees to help ensure that operations stay on track. In 2018 SBB underwent an agile transformation and chose Xray to help them manage over 100,000 test cases and test management activities.
Switch to Xray Test Management
The benefits and features that we’ve listed above have a compound effect on the quality of your software testing. A test management tool like Xray alleviates many of the common challenges that software testing teams face.
Through proper organization you are able to reduce the manual effort and focus on test strategy and risk mitigation. With integrations into many of the existing test automation frameworks, you can reduce manual work while properly organizing your test results.
Finally, with the right reporting and documentation, you are able to extract valuable data, meet requirements and comply with regulations that require traceability.
If you’re interested to know more, take a look at an Xray Quick Demo and sign-up for a free trial.
You can also find The Comprehensive Guide to Xray on the Idalko blog.
