The Company
Ajmera Infotech is a software and product development company specializing in building large-scale systems. They provide end-to-end software development and consulting for web, cloud-native, and mobile applications. They blend technology expertise and process discipline to build large-scale enterprise and consumer systems for their clients and also their in-house built tools.
The Challenge
Ajmera Infotech is based in India, one of the most competitive markets worldwide in the software development Industry. To maintain their edge over their competitors, they had to be able to create better, more functional products, accelerate their speed to market, and work more efficiently to keep high quality while keeping their production costs low.
Before Xray, Ajmera Infotech was using Microsoft Word and Excel documents to manage all of their testing. Although this was a low-cost solution, it was not ideal, and, with the growing number of projects and increasing teams, severe maintainability issues started to emerge.
Their biggest problem was the sheer amount of test cases and their exponential growth rate, leading to inefficiencies in writing and maintaining test cases.
Ajmera Infotech had more than 3,000 test cases spread across multiple Word and Excel documents, some of which were duplicated to fit different scopes. This was due to their need to test various combinations of elements that were spread across different areas of their application. For instance, a simple test of 5 elements in 4 different areas would amount to 20 combinations, meaning 20 different test cases.
Their traditional approach to writing test cases was too time-consuming and also spent a lot of resources, requiring more time and more personnel than what they had available. The single process of writing a test case would take between 2-3 hours.
“First we would define all scenarios manually written on a handbook, and only then we would separate through different combinations and then write them again.”
Dhaval Vala, Senior QA Lead
Maintaining test cases was a big challenge for the Ajmera Infotech team. For example, whenever a requirement changed or the Test Case needed to be updated, they had to manually make these changes to all documents containing the Test Cases.
Besides the manual effort required, it was difficult to ensure that all documents were updated and that, in the future, the team would be aligned and use the most recent version of the Test Cases.
Over time, Ajmera Infotech realized that their software development process was taking too long due to slow testing. They needed a solution to improve the QA processes, and that would meet these requirements:
- Provide complete visibility across existing test cases in one place;
- Speed up the creation of test cases with multiple combinations of factors;
- Ensure that all requirements are covered by tests and that these tests are being run.
- Improve test case maintenance processes
- Most importantly, integration with Jira. Besides being the software the development team used to manage their tasks, they wanted to centralize their processes in a single platform. This would allow them to save time switching between platforms.
The Solution
Once the development process bottleneck was identified, Ajmera Infotech started looking into a new application that would help them improve the speed and quality of their application development process by enhancing their testing process.
Ajmera Infotech opted for Xray as the application to help them improve their Test Management processes, as it fulfilled all of their requirements.
Installing and configuring the new tool was very straightforward using the installation guides provided by Xray’s documentation. Since their test cases were mostly scattered across multiple documents, Ajmera Infotech decided not to use the Excel import feature that would allow them to import all their legacy test cases into Xray. Instead, they created all the test cases manually.
Ajmera Infotech used Xray's Test Parameterization feature to increase the speed of its test case creation.
First, they needed to start by defining test-level, project-level, and global-level parameters - Parameters are variables that can change with each execution.
Here’s how Ajmera Infotech broke them down:
- For global-level parameters, variables that were common across multiple projects.
- For project-level parameters, variables that were common to multiple tests in that specific project.
- For test-level parameters, a set of variables that would only be used for that test.
With Xray’s Test Parameterization, Ajmera Infotech’s team only needed to write the manual test case once with multiple steps. Then, Instead of replicating the test case for all different combinations of variables, placeholders are used for categories such as Diagnostic Data (operatingSystemName, browserName, titleOfThePage) and locations (JSON File, Web Panel, and Issue Description).
Xray automatically creates all the combinations of the parameters and all possible iterations between them in a single test execution. By doing this, the number of test cases is exponentially lower than when creating a different test case for each iteration, as Ajmera Infotech was doing before.
Test parameterization enables a complete view of all iterations created, giving a better sense of all the existing scenarios and ensuring that all combinations are accounted for.
“The test parameterization feature is more complex than other Xray features but having documentation about the feature helps.”
Dhaval Vala, Senior QA Lead
The Results
Ajmera Infotech trusted Xray to help them improve their processes, and 4 years later, they are still confident that this was the best choice they could have made.
“Xray is the better option for me and for my company. It reduced the time we needed to create tests, and using Xray made the whole Test Management process much easier”
Dhaval Vala, Senior QA Lead
From a business perspective, the benefits brought by Xray were visible across the board with numbers that show the real impact on their performance:
- Increased test case creation speed by 4x: Before Xray, creating a test case took 2 to 3 hours. Today, creating Test Cases seldom takes more than 30 minutes;
- Reduced the number of bugs in production by 50% by guaranteeing complete requirements coverage and being able to detect bugs sooner;
- Reduced the number of active test cases from 3,000 to 1,700, and continuous optimization continues to drive this number down;
- Increased human resources availability for different projects in 2.5x: Before Xray, the standard teams’ ratio between developers to testers was 4 developers to 1 tester. Today, with the efficiency brought by Xray, the teams’ ratio is 10 developers to 1 tester.
Using Xray, Ajmera Infotech also increased the overall quality and speed of development. All of the information is within Jira, the developers have access to the information about the tests, they can quickly understand what scenarios worked and which ones didn’t, and more quickly address the scenarios that failed. The number of bugs uncovered on each release also dropped by about 50%, meaning that the developers are also benefiting indirectly from having Xray.
“Xray has improved the relationship between Developers and Testers. The developers don’t need to ask for test scenarios, they can click on the ticket, which directs them to the Xray issue, without leaving Jira, where they can see all test scenarios.”
Dhaval Vala, Senior QA Lead
Admitting or not, Dhaval believes the developers are also starting to test independently. Even if they don’t create test cases or perform test executions, they can access the Test Cases created by the QA team and test their code according to the defined testing scenarios. By taking some of the testing into their own hands, the developers ensure that the code passed on to the QA team is already free of bugs and working as intended, saving time and money for Ajmera Infotech and accelerating their software release process.
Xray’s test parameterization also improved the resources needed to maintain the test cases. Today if the team needs to add another variable to one of the parameters, let’s say a new integration, it’s as simple as adding the name of the integration to the data set. All the new combinations will be created automatically on the scenarios’ list and there is no need to write down any new test case.
Harnessing the full potential of Test Parameterization with Xray
Ajmera Infotech showed significant improvements by onboarding a test management tool natively integrated with Jira. Not only did they fix their issues with managing their thousands of Test Cases, but they also gained speed and efficiency across the software development life cycle.
Bringing Xray into its engineering stack improved Ajmera Infotech's performance. They accelerated their test creation process by more than 4x while using fewer resources. Test Parameterization was the biggest leverage for this performance improvement because it exponentially decreased the number of test cases that needed to be created and maintained, from dozens to one.
Moreover, Ajmera Infotech now has a Test Repository, making it easier for everyone to access all active Test Cases as well as all other ones created in the past. And it brought benefits to the team. Firstly, the QA team is now more efficient in easily finding previously created test cases. Secondly, the development team now knows what tests are being run and what functionality is expected of them. Lastly, new team members can now better understand how testing is done at Ajmera Infotech during their onboarding.
