Continuous integration and continuous delivery (CI/CD) have become essential for delivering high-quality software at speed - automating testing processes ensures that new code changes are thoroughly validated before reaching production. However, managing these automated tests across multiple CI/CD tools can be challenging.
This blog post explores how Xray Enterprise can be seamlessly integrated with popular CI/CD tools to enhance your automated testing workflows. The tool offers powerful capabilities to help you automate test management, track results, and ensure the reliability of your software releases. Let’s dive into it!
Why CI/CD
CI/CD is a set of practices and tools that automate and speed up software development, testing, and deployment. Continuous Integration (CI) automatically merges code changes into a shared repository multiple times a day, ensuring early and frequent testing. Continuous Delivery (CD) takes this further and automates the process even more, deploying every change that passes all pipeline stages directly to end users.
Benefits of CI/CD Integrations:
- Faster, more efficient development cycles: automated testing and deployment pipelines accelerate the release of new features and updates, freeing up developer time;
- Quality engineering: immediate detection of issues, ensuring high-quality code and reducing bugs;
- Consistent delivery: automation ensures that deployments are consistent across different environments, minimizing the risk of human error.
Xray Enterprise: Enhance Efficiency of CI/CD Pipelines
It is important to have visibility into your test results as they directly impact the status of your requirements. To achieve this, you must link your tests with requirements and report automated test results from your CI/CD environment, ensuring you can evaluate requirement coverage at any moment.
Xray Enterprise allows you to manage manual and automated tests in the same way, making it an ideal tool for teams that require consistent test management across different types of testing.
To maximize the benefits of automated testing, you can integrate Xray Enterprise with CI/CD tools for:
- Improved synergy: trigger tests from a Jira-native test management tool and/or import the results back via a CI/CD task;
- Result consolidation: collect and track test results directly within Xray Enterprise;
- Robust reporting: generate reports on test results, coverage, and trends, aiding in decision-making and continuous improvement.
Let’s explore the process of integrating Xray Enterprise with popular CI/CD tools in more detail!
Part 1 - Launch execution from Xray Enterprise
This part is optional (i.e. you can still launch the execution from the CI/CD platform itself and only consider Part 2) but crucial if you’re looking to expand test automation triggers to more teams and even non-technical users.
The common workflow involves:
- Test creation: define your tests in Xray Enterprise, linking them to the relevant requirements;
- Test execution: launch the CI/CD execution from Xray Enterprise using Remote Jobs Trigger (RJT).
RJT allows you to configure and launch remote jobs in various CI/CD tools directly from Xray Enterprise, enabling you to execute specific actions such as validating changes, deployments, or running automated tests without leaving the Xray interface.
Benefits of Remote Job Triggering
- Improved efficiency: streamline testing workflows by managing and triggering jobs directly from Xray;
- Centralized control: manage and monitor all testing activities from a single interface;
- Flexible integration: adapt the process to the CI/CD tool of your choice.
You can integrate Xray Enterprise with CI/CD tools like Jenkins, Bamboo, GitHub, GitLab, and Azure DevOps to trigger remote jobs from Xray entities such as Test Plans, Test Executions, and Test issues, including viewing logs, job status, and results.
Taking a closer look at the Jenkins example
Jenkins is supported by Xray through the “Xray for JIRA Jenkins Plugin” plugin and REST API, enhancing the functionality by providing tasks that facilitate actions like importing test results.
Prerequisites include Jenkins server setup, Xray Enterprise for Jira installed and configured, and necessary permissions on both sides.
In this example we will define a job in Jenkins and invoke it from Xray. This job will execute the Playwright tests and generate a JUnit report.
First, you will need to complete the configuration, which would look like this on the Xray Enterprise side:
Next, you will find the corresponding trigger option on one of the Xray entities, for example, a Test Plan.
Lastly, we need to send those results back to Xray for full visibility.
For complete examples, make sure you check the following resources:
- Remote Jobs Triggering (Server + DC)
- Remote Jobs Triggering (Cloud)
Part 2 - Prepare the report and import the results to Xray
For the avoidance of doubt, this part is not limited to Xray Enterprise - regardless of where you launch the execution from, you can import the results to Xray and enjoy the test management benefits we mentioned earlier.
The common workflow involves:
- Test execution: the tests are executed, the report is generated in the format supported by Xray with the appropriate attributes;
- Importing results to Xray: the results report is imported into Xray.
- If the corresponding Test issues do not exist, they will be created automatically by Xray.
- Otherwise, the existing Test issues and, if you choose so, Test Executions will be updated.
In addition to JUnit, TestNG, and NUnit, Xray supports other popular frameworks and report formats, including:
- Robot Framework;
- Behave;
- xUnit;
- Ranorex;
- Xamarin.
XML is the most popular overall format, but JSON is also supported in Cloud.
For the import, you can utilize specialized plugin support (e.g. “Xray - Results Import Task” from our plugin for Jenkins) or command line methods. We have more dedicated integration instructions for tools like CircleCI, Travis CI, TeamCity. Please check the following resources:
- Xray Integrations (Server + DC)
- Xray Integrations (Cloud)
Once the import finishes, we can review the specific execution results and the overall status on the, e.g., Test Plan issue (the screenshot corresponds to the RJT example from part 1, where we also added the “XrayImportBuilder” step to the Jenkins pipeline definition).
Tips for Smoother Implementation
In this section, we have collected useful pieces of advice from across our extensive documentation. If you prefer a more structured learning approach, we encourage you to check the “Test Automation with Xray” course in our Academy.
Generic vs Cucumber tests
We wanted to highlight two Xray issue types commonly associated with automated testing, so that you can make an informed decision about which one works best for you.
First, Generic Tests in Xray serve as an unstructured abstraction layer, allowing to report on the execution results without enforcing specific constraints on how the tests are implemented on the Xray side. It is often used when only Part 2 is implemented as the popular default for Test issues automatically created after the report import.
The Generic Test Definition field is used to provide additional identification details, such as the class and method that implements the test code or the automated script file name.
Second, Gherkin, primarily used in Behavior-Driven Development (BDD), is a language for writing test scenarios in a way that is understandable by all stakeholders, including non-technical ones. The corresponding issue type on the Xray side is “Cucumber Tests”.
Xray's comprehensive support for Cucumber tests includes the following capabilities:
- Creation and editing: supports Scenario, Scenario Outline, and Background entities;
- Automated steps library: Centralized management and refactoring of Gherkin steps;
- Bulk import/update: handles .feature files using the REST API or CI plugins;
- Export: allows exporting of .feature files containing Scenarios and Backgrounds through the UI, REST API, or CI plugins.
There are several nuances when it comes to utilizing the Cucumber tests in your CI/CD process. For more information, please see the following articles:
- Testing in BDD with Gherkin based frameworks (Server + DC)
- Testing in BDD with Gherkin based frameworks (Cloud)
Cloud vs Data Center CI/CD integration differences
Depending on your Xray deployment, you need to pay attention to the corresponding API documentation and the overall import process, which is why we provide the reference links in pairs.
For instance, Cloud requires you to first obtain the token, then proceed with the import call, whereas you can do the import straightaway in the Data Center. Also, make sure to leverage the documentation specifically for your API version of choice (1 or 2). Check the following resources:
Report Customizations
Depending on your automated framework/report format and the import method, you may be able to add custom attributes that further enhance the synergy between the CI/CD tools and Xray. For instance, you can link the results to a requirement and a test plan automatically during the import, rather than manually after the fact. Check the following resources:
- Testing web applications using Selenium and Junit5 in Java (Server + DC)
- Testing web applications using Selenium and Junit5 in Java (Cloud)
Also, please keep in mind that the API endpoints on the Xray side can be different depending on the report format.
Please refer to the tutorials we linked above to learn about the specific options for your tools of choice.
Elevate your game
The efficient approach of automated testing and the speed of CI/CD implementation ensures that any new code integrates smoothly with the existing codebase, maintaining the overall quality and functionality of the software.
Xray Enterprise is a leading enterprise test management tool that seamlessly integrates with Jira, offering a solution for managing both automated and manual tests in a centralized manner. With support for various testing methodologies and integration with multiple CI/CD tools, Xray Enterprise stands out for leveraging these capabilities. It results in comprehensive test coverages, accurate results, and your ability to manage your test cases and test plans with unmatched efficiency.
