Xray for Jira automated tests with a CI Platform - Bamboo
Many customers have been asking us for a simple way to integrate Xray in a Continuous Integration Platform. By doing this, bugs are identified and fixed on a regular basis and quicker time-to-market is achieved.
In this article, we are going to show you how you can automate your tests using Bamboo (CI platform from Atlassian) and our Xray for Jira.
We all know that the process of testing can be, for a software tester, an everyday challenge. Rather than just an activity, software testing has to be considered as one of the most important processes inside the Software Development Life Cycle. It must assure that the product can be implemented with the same characteristics, that all the business and technical requirements in terms of design and development are met and, above all, that it works as expected.
Considering this, Continuous Integrating your testing process gets primordial importance when you want to enable a quicker time-to-market. With that, most integration bugs are identified and fixed on a regular basis, and the continuous cycle of: Code->Build->Test->Deploy raises the probability of getting high quality and bug-free releases.
All this is just possible if you are automating your Tests and Test Sets. Inside Jira ecosystem, Xray offers this possibility.
In this article, we will just focus on Automated Tests, even though Xray allows you also to manage your Manual tests inside Jira.
In Xray, Automated Tests are executed through an external tool, which controls the execution of tests, and the comparison of actual outcomes to predicted outcomes. They can automate some repetitive tasks in a formalized testing process already in place, or add additional testing that would be difficult to perform manually.
Xray for Jira add-on for Bamboo aims to simplify the process of continuous integration. With the setup of some Xray specific tasks, users can now easily export Cucumber features, report automated testing results for many different automated testing frameworks and associate them with new or existing Tests, Test Executions, Test Plans, Test Environments, and more.
Available Xray tasks
- Import Execution Results: This task imports the execution results to Jira. It supports importing results in Xray's own JSON format, Cucumber, Behave, JUnit, Nunit, among others. For some formats, it enables the user to have full control over newly created Test Execution fields, by allowing to send an additional JSON specification, similar to the one Jira uses to create new issues, more info here;
- Export Cucumber Features: This task exports the Cucumber Tests (i.e. Scenario/Scenario Outlines) in feature files. If the features are archived in a zip file, then they are automatically unzipped. This task is based on this rules to export the Tests.
Examples
Let's get to know the add-on by running two examples. For each example we will create two plans, both taking into account the specific testing framework and with the common final goal of importing the execution results to a specific Xray endpoint.
We will assume that you already know how to pull your test project from your repository and how to run the tests in the Continuous Integration environment. As such we will only focus on how to configure the Xray specific tasks.
JUnit (applicable also to Nunit, Robot)
This use case covers the execution of JUnit tests in the CI Environment and then import the results back to Jira.
Having this scenario in mind, we will build a simple plan to execute JUnit tests and import the execution results in JUnit XML format. It will be consisted by the following tasks:
- Pulling the source code of the JUnit project;
- Executing the tests in the CI environment;
- Importing the execution results to Jira.
Importing the execution results to Jira
We will configure this task so it publishes the results to Jira, creating a new Test Execution issue associated with three Test Environments (Android, IOS, Cordova). Note that, the import task should be a final task to ensure that the results are imported to Xray even when there are failures during test execution.
After the task configuration is finished, we just hit the save button.
After finishing the configuration for each task, you can then run the plan. If everything goes well, the build ran successfully and a Test Execution issue will be created in the Jira instance, similar to the one depicted on the following image.
Cucumber
In this use case, we will be based on the Standard Cucumber Workflow. The plan will be the following:
- Pulling the Cucumber project;
- Exporting Cucumber features from Jira to your Cucumber project;
- Executing the tests in the CI environment;
- Importing the execution results back to Jira.
Exporting the Cucumber features
Before executing the tests, we need a task to export the Cucumber features from the Tests issues in the Jira instance. As such, we will configure the task accordingly with the one depicted in the following image
Note that this task will also take care of unzipping the features files.
Importing the execution results to Jira
After running the tests we will need to have the execution results published in the Jira instance. For that, as we did in the JUnit example, we will create an import task. In addition, we will want to have control over the Test Execution fields, so we can add in the summary the ID of the build that ran the tests.
To achieve this we will select the Cucumber JSON Multipart. With this option, in addition to send the JSON file with the execution results, we can send another file with extra information used by Jira to create the issue. Also, note that in this example we will take advantage of Bamboo variables to generate the ID of the build.
Once again, the import task should be final to ensure it is executed regardless of previous task status.
After running the plan, you can see the results published in the Jira instance.
In sum…
Importing automated execution results and exporting Cucumber features is now easier with Xray for Jira add-on for Bamboo. With some simple parametrization, you can take advantage of the Xray’s REST API and every available endpoint for importing results, to “Continuously Integrate” and thus continuously improve your product quality, while having on-time feedback that will help you promptly fix any issue.
Good luck in “Continuous Integrating” your Tests!
Editor’s Note: This post was originally published in December 2015 and has been completely revamped and updated for accuracy and comprehensiveness in 2018.
