Xray for Jira automated tests with Jenkins
Following the segment initiated with Bamboo, in this article we will show you how you can automate your tests using Jenkins and our Xray for Jira.
In Xray, Automated Tests are executed by an external tool, which controls the test executions and the comparison of actual outcomes and the predicted ones. They can automate some repetitive tasks in a formalized testing process already in place, or add additional testing that would be difficult to perform manually.
As the testing tool, we will use Cucumber because it allows you to execute plain-text functional descriptions as automated tests. You can find more information about how automated tests are created in Xray, as well as more information about Cucumber, in Xray documentation and Cucumber’s website, respectively.
It is also important to point out that this is just possible since Xray makes also available it’s Rest API and, with that, we will be able to export Test definitions and import execution results.
Xray for JIRA add-on for Jenkins aims to simplify the process of continuous integration. With the setup of some Xray specific tasks, users can now easily export Cucumber features, report automated testing results for many different automated testing frameworks and associate them with new or existing Tests, Test Executions, Test Plans, Test Environments, and more.
Examples
Let's get to know the add-on by running two examples. For each example we will create two plans, both taking into account the specific testing framework and with the common final goal of importing the execution results to a specific Xray endpoint.
We will assume that you already know how to pull your test project from your repository and how to run the tests in the Continuous Integration environment. As such we will only focus on how to configure the Xray specific tasks.
JUnit
Apart from supporting Cucumber natively, Xray enables you to take advantage of many other testing frameworks like JUnit. In this sense, Xray for Jenkins lets you import results in other formats besides Cucumber JSON.
- Pulling the JUnit project;
- Executing the tests in the CI environment;
- Importing the execution results, including Tests, to Jira.
Importing the execution results to Jira
To start the configuration, add the post-build action Xray: Results Import Task.
After that, configure it.
In this example, we have a configuration where the JUnit XML format is chosen.
After running the plan, the expected result is a new Test Execution issue created in the Jira instance.
Cucumber
In a typical Cucumber workflow, after having created a Cucumber project and the Cucumber tests specified in Jira, you may want to have a project that exports the features from Jira, executes the automated tests on a CI environment and then imports back its results.
- Pulling the Cucumber project;
- Exporting Cucumber features from Jira to your Cucumber project;
- Executing the tests in the CI environment;
- Importing the execution results back to Jira.
Exporting Cucumber features
To start the configuration, add the build step Xray: Cucumber Features Export Task.
After that, configure it.
In this example, we configured the task to extract the features from a set of issues (PROJ-78 and PROJ-79) to the folder that holds the Cucumber project.
Importing the execution results
To start the configuration, add the post-build action Xray: Results Import Task.
After that, configure it.
In this example, we configured the task to import the Cucumber JSON results back to Jira.
Once all configurations are done, click Save at the bottom of the page.
After running the job, the expected result is a new Test Execution issue created in the Jira instance.
Importing the execution results with user-defined field values
For Cucumber, Behave, JUnit, Nunit and Robot, Xray for Jenkins allows you to create new Test Executions and have control over newly-created Test Execution fields. You can send two files, the normal execution result file and a JSON file similar to the one Jira uses to create new issues. More details regarding how Jira creates new issues here.
For this scenario and example, the import task needs to be configured with the Cucumber JSON Multipart format. When selecting this option, you can additionally configure the Test Execution fields in one of two ways:
- Insert the relative path to the JSON file containing the information, or
- Insert the JSON content directly in the field.
In this example, we configured the following object:{
"fields": {
"project": {
"key": "PROJ"
},
"summary": "Test Execution for Cucumber results (Generated by job: ${BUILD_TAG})",
"issuetype": {
"id": "10102"
}
}
}
And configured the task to import the Cucumber JSON Multipart results back to Jira.
Once all configurations are done, click Save at the bottom of the page.
After running the job, the expected result is a new Test Execution issue created in the Jira instance, with the Test Execution fields as specified in the Jenkins build step configuration.
In sum…
Importing automated execution results and exporting Cucumber features is now easier with Xray for Jira add-on for Jenkins. With some simple parametrization, you can take advantage of the Xray’s REST API and every available endpoint for importing results, to “Continuously Integrate” and thus continuously improve your product quality, while having on-time feedback that will help you promptly fix any issue.
Good luck in “Continuous Integrating” your Tests!
Editor’s Note: This post was originally published in August 2016 and has been completely revamped and updated for accuracy and comprehensiveness in 2018.
