If you want to manage a release process that is easy to follow, we find that Jira with Xray can be a great fit for most organizations that want to keep quality at the center and implement agile best practices.”
Gianni Pucciani - QA Specialist - TCS
About TCS
With over 1.5 million members, Touring Club Suisse is the largest mobility club in Switzerland. Founded in 1896, TCS provides assistance, road safety education and accident prevention to all travelers and vehicles along the entire Swiss road infrastructure.
TCS adopted Xray for their test management in 2015, which allowed them to manage all the testing activity in Jira and have a simple and solid QA and release process all in Jira. They saved time in Test Definition (through re-usability), Defect Analysis (through the link Defect-Test), and Reporting (no need to collect input data via email and aggregate test results from several Excel files).
In December 2019, TCS transitioned away from writing test automation code using Selenium. Instead, they would adopt Boozang to automate their tests, a tool that allows non-developers to participate in automation. The integration with Xray was easy thanks to the well-documented REST API’s that allowed them to export test definitions and import test execution results.
In this Success Case, they share why they outgrew their automation process with Selenium and Jenkins, and how their new automation process with Boozang helps them scale their test automation.
The challenge
In 2015, TCS started investing in test automation with a custom made solution based on Selenium, Cucumber, Xray, and Jenkins. The objective was to create a regression tests suite to cover the main business processes on the Salesforce platform, used not only as CRM but as a product catalog, sales, and contract management tool.
Features and Scenarios were documented in Jira via the Xray and executed nightly on 2 test environments via Jenkins. Results were available in Jenkins (via the cucumber-reports plugin) as well as in Xray.
As their maturity evolved and application complexity had grown they wanted to: reduce dependency on developers; do end-to-end testing over several applications; keep existing ways-of-working, including Xray and Jenkins.
Old automation process
- A Business Analyst defines features and scenarios in Gherkin language directly in Jira via Xray;
- A Developer creates page objects and methods to implement scenario test steps;
- Jenkins runs the scenarios on the QA environment;
- When stable, the same scenarios are run on the Pre-Production environment;
- Reports are captured in Jira, Xray, and Jenkins.
What was working?
Simple and smooth process: The definition of automated test cases in Xray via the Test Type Automated Cucumber was well adopted by business analysts. The integration with Jenkins allowed the nightly schedule of automated tests and the possibility to analyze quickly the test results in the morning, with a drill-down view from features to scenarios and steps.
Automated and Manual tests all in the same place: allowed to report the integration of the QA activity in a single dashboard. Tests for new features and bug fixes were mainly manual, and regression tests automated. They use the Overall Coverage Gadget to show the progress of testing.
What were the pain points?
Developer capacity: Developing new scenarios and maintaining the current ones was taking a considerable fraction of the time allocated to develop new features. Only 2 out of 6-7 developers were able to maintain and extend the framework which was solid but not the easiest to master.
Limited to a single web application: Selenium worked ok as long as the system under test was a single one, in this case, Salesforce. Automating and end to end process going over three web applications (E-Commerce site, Salesforce, and backend applications) was too complex with Selenium.
Outside of coding and maintenance of Selenium tests, the existing process at TCS was working well. The challenge was therefore to replace Selenium while maintaining their current way-of-working.
They wanted to continue to use Xray and Jira for Cucumber test management, which was working very well. They also would like to keep Jenkins as their CI servers and maintain the Cucumber reporting format.
The transition
The team decided to look for a codeless solution that fulfilled these main requirements:
- Focus on web application;
- Cover E2E business processes over several applications;
- Usable by people without developer or test automation background. These would allow more flexibility and the capacity to implement automated tests;
- Integration with Xray and Jenkins;
- BDD (Cucumber) support;
- Low scripts implementation and maintenance effort;
- Respect budget constraints.
In the search for the right tool, they encountered Boozang: AI-powered test-automation that runs in any web browser, and saves the data in the Cloud.
A proof of concept went smoothly, the tool was easy to learn and they did a pilot to further explore it. The pilot phase lasted 2 months, with approximately 20 man-days. The pilot was meant to challenge the solution on real-life scenarios, covering 3 web applications, and integrating with Jenkins and Xray.
Cucumber/Gherkin test in Xray after it’s been imported in Boozang. Xray is the master system for tests and test results
Cucumber test written in Boozang. Boozang replicates the test via the Xray API and links it to test automation code
The solution
In December 2019 after a successful pilot, Touring Club Suisse adopted Boozang to automate their tests and allow non-developers to participate in automation.
The integration with Xray was easy thanks to the well documented REST APIs that allowed them to export test definitions and import test execution results.
“We like that Boozang focuses on web test automation and it does it well. By leaving the scheduling to Jenkins, the requirements management to Jira, and the reporting to Jenkins, we can keep our current setup and just replace the challenging part: implementing tests in Selenium.”
Gianni Pucciani - QA Specialist - TCS
The results
“Test automation is an important factor to increase the quality and time to market. For test automation to be efficient and able to cover end to end business processes we find that a BDD like approach coupled with Boozang is a great solution that integrates well with Xray and your CI/CD infrastructure.
Gianni Pucciani - QA Specialist - TCS
The new process was therefore quite similar to the previous one. The most important change was becoming an agile team that works more efficiently on defining and implementing the scenarios.
The main benefits of the shift were:
- Better workload distribution and a faster turnaround;
- Automation work split efficiently among members of an agile team with different skills and backgrounds. This way tests can be designed and implemented during a sprint, with an approach that is closer to BDD (Behaviour Driven Development). They still recommend one skilled member to act as an automation architect to design robust modules;
- Scenarios are now in a human-readable language not just for the Gherkin part, but as well at the step implementation level;
- No more dependency from web driver-browser compatibility.
The biggest results
- Reduced test maintenance time by 60%;
- Freed up development resources to focus on building features, not automation code;
- Covered a larger set of web-based applications with automation, and increased test coverage;
- Better distributed work throughout the project life-cycle.
What they love about Xray
Clean design: For QA practitioners Xray feels like home. The design is clean and the available objects allow for full quality control during the release process, supporting from simple single applications to complex multi-application and multi-environment enterprise context.
Manual and Automated tests both in the same place: With Xray, you can manage both manual and automated tests using the same process and tool. If during a Sprint part of the Test only part of the test was automated, you can still have a unified report that shows the test coverage of both automated and manual executions.
Documentation: Keeping the documentation up to date is a known challenge for software editors, and the Xray team does this job very well. Be it an API description or a new feature, all you need to know about Xray is always available on their Confluence page.
Great support and clear roadmap: Xray support is reactive and very efficient. Their roadmap is clear and really tailored to users’ needs.
Flexibility to adapt custom workflows with Xray
As TCS’s test automation matured and efforts increased, Xray’s flexibility helped them scale their test automation. Xray allowed them to easily adopt a new workflow, reduce dependency on developers, do end-to-end testing over several applications, and keep existing ways-of-working with Xray and Jenkins.
Now, everyone involved in the software development process can write automated tests, making them faster and even more agile. Now that’s the kind of flexibility you want out of your SDLC toolset.
Additional Resources
- Sign-up for a free Xray trial;
- Follow our 10 Step Checklist to build a mature test automation framework;
- Learn how to accelerate testing with in-sprint automation;
- Align your team with BDD and automate testing;
- Learn more about Boozang.
