About TomTom
TomTom is the leading independent location technology specialist. Their non-stop innovation keeps the world moving forward – freely, efficiently, safely.
Their maps for automated and autonomous driving, navigation software and real-time traffic and travel information provide a better way forward. That’s why 100’s of millions of people, leading car manufacturers, cities, governments, technology companies and developers rely on TomTom’s products and services every day.
Software development is at the heart of their business. Innovation and proper software testing and management is an essential element for the development and release of every project.
The Challenge
10 years ago, TomTom had no test management system or tool and was entirely relying on Excel to manage their software testing. This was tedious and unreliable and it lacked traceability.
“Managing tests became more and more important, and up until that time, we managed tests, and Excel just didn’t cut it,” says Ronald Hogenboom, Manager of Software Engineering at TomTom.
They decided to switch to a test management tool and HP ALM was introduced company-wide to manage requirements and schedule tests. However, over a few year’s time, it became evident that the tool was causing them more pains than gains.
"Support from HP ALM was always quite poor. They promised a lot of new features and bug fixes that never came. It only worked in internet explorer – causing limitations for people working with virtual machines on Linux. Also licensing costs were quite high.”
Ronald Hogenboom – Manager of Software Engineering TomTom
Having HP ALM for several years, they experienced these main pain points:
- Very poor support
- High licensing costs
- Features that were promised but not delivered
- Bug fixes that never came
- Limitations on browsers (only worked on Internet Explorer)
- Testers thought HP ALM was not user friendly
- HP ALM increasingly became slower
- Limitation in traceability for the test sets structure and test results
- Lack of support from ALM played a role in not implementing test automation
Mieke van Eijck, Lead Test Engineer at TomTom shared her experience with HP ALM. “Everybody was complaining about ALM. It took me 4 hours per week to log the test results. It was completely unusable.”
During that time, they used Jira for backlog management, and issue tracking for their software development projects. Ronald says that “It was a logical step to move to Xray, because then we could use Jira, the tool that everybody likes and understands, and add another thing which was test management.”
“Having all the test management in Jira makes life for us a lot easier than it was when we had HP ALM as a separate entity.”
Ronald Hogenboom – Manager of Software Engineering TomTom
With so many clear pain points across the whole organization from high costs, to bad usability and a block for productivity, it was evident that they needed a new, powerful solution that would solve their pain points and give them a newfound flexibility.
The Solution
They searched for a solution in the Atlassian Marketplace with a requirements checklist for the main features they wanted to see in a tool, Xray met all of their needs and they settled on an unlimited Xray license.
In 2018, they started the migration with a project of 600 test cases which they imported using the Xray HP ALM importer tool.
“One of the reasons to transfer to Xray was because, anyone who understands Jira, which in our company is everyone, can use Xray. There is hardly any training necessary which to us was a big advantage.”
Ronald Hogenboom – Manager of Software Engineering TomTom
The migration was a bit of a challenge but with the right support, they were able to migrate. There was hardly any training because everyone on the team was already familiar with Jira, and if you know Jira then Xray will come intuitively to you.
“During the migration, we received an error from the importer tool. We created a support ticket to get support, someone from Xray analyzed it, and provided a custom development build of the tool, it worked, the support was good and quick.”
Maikel Verbeek – Sr. Test Engineer at TomTom
Once the tool was working, it took only 30 minutes to transfer 600 test cases using the HP ALM to Xray importer tool. Now in 2020, every project in TomTom uses Xray and it is their main, centralized solution for all of their test management.
The Results
TomTom saw significant improvements and advantages with Xray as their main test management tool. The main benefits include: smooth performance and scalability with a growing number of test cases, faster test execution time, less manual work, more visibility and transparency for developers.
Here are the top benefits and features that they saw with Xray within 2 years.
1. Easy to use with extensive documentation
The whole team agrees that Xray is easy and intuitive to use. If you understand Jira, you can understand Xray.
“Multiple people within the team can easily transfer tests from one person to another, someone creates it and someone reviews it. There is easy linking, and no extra tooling required to do it. The documentation is also extensive and that also helped in the beginning.”
Maikel Verbeek – Sr. Test Engineer at TomTom
When they used HP ALM, this was a tool that was used exclusively by testers. Now they integrated everything that is testing and it is much more collaborative and easy to use.
2. Transparency and collaboration
Xray is much more transparent to developers, they actually see the amount of tests available and what components they cover, the results are also visible for them as well. Transparency is a big added bonus.
Testers found HP ALM scary to use, and 1 person would be the only one to manage all the test cases. People are really happy now that they can work autonomously and independently, and don’t want to depend on others. Now everyone knows what they’re working on and have full visibility to jump in and solve challenges faster.
3. Requirement coverage and traceability
In HP ALM, they did manual requirement coverage. When there was a new requirement they would add it manually. Now that everyone is using the same toolset with Xray and Jira. If a requirement changes, it is updated everywhere.
This saves them manual work in logging changes to the requirements which are updated automatically and visible to everyone on the software development lifecycle.
4. Scalability and performance
Xray has been doing extensive enhancements on performance and scalability.
TomTom knows they can rely on Xray, even as the number of their test cases and test executions grow. They have over 4,000 test execution tickets. For larger projects, they create Test Plans to organize their tests.
Project use case #1: Embedded navigation system for a mainstream car manufacturer with 2,200 manual tests, and 450 automated tests that are pushed automatically.
Project use case #2: A major Japanese car manufacturer’s entertainment system navigation. They have 1,400 test cases – with the majority manual, 1/5 of the tests will be automated.
5. Reporting and analytics
Although TomTom still wants to improve how they do reporting, they know that Xray’s reporting features have improved a lot in the past 3 years since they first got the app. For current projects, they send the reports directly to the customers and what they send now is quite extensive. They’re still using Excel and they would like to improve and start using more Xray reports.
“Jira itself already supports quite a lot of dashboards and we use JQL queries to pull out data that we want. It’s pretty flexible in that way and everybody uses it as they see fit.” tells Ronald Hogenboom.
“We use the test execution report for one of the projects. For the rest, for my reporting, I just make queries myself and export to excel and make whatever I want out of it,” says Mieke van Eijck.
6. More test automation
TomTom were using the standard Jenkins plug-in from Xray, but recently they changed to a custom developed library to push test results from Jenkins to Xray. They preferred to have a solution that would be independent of any CI server as they were also exploring CI/CD on Azure. Therefore we created a custom library that satisfied all of our needs. Currently, they’re publishing the test results in Xray as just an empty shell because the actual tests live in their repository and make use of the Google test framework.
Previously with HP ALM, they just had placeholders for automated tests and had to log the results by hand which was far from ideal. The lack of support from ALM, played a role in not doing more automation because they knew they couldn’t rely on them for support with their automation efforts.
7. Faster execution and performance
Compared to HP ALM, now they’re able to quickly make their test executions for their manual tests. HP ALM was a bit unresponsive with more test cases in one place, and now they have an increase of 10-20% faster execution for the test cases.
8. Quick and reliable support
During their initial migration, they reached out to Xray support who gave them a custom solution to their challenges.
“A benefit of Xray is that you actually get support – if something doesn’t work, you can raise a ticket and it can be solved. This was very different with HP ALM.”
Ronald Hogenboom – Manager of Software Engineering TomTom
Leaving a legacy tool behind made TomTom faster and more efficient
TomTom experienced first-hand the slow and burdensome use of a legacy tool like HP ALM. Having their previous test management tool outside of Jira put the testing team in a silo, and QA was the sole responsibility of the testers.
With Xray integrated into Jira, now testers and developers work together to efficiently manage testing and together mitigate risk in their software. Knowing that Xray has fast and reliable support, they can take on more integrations and start doing more automation, because they know their efforts will be supported.
With Xray, all team members on the software development lifecycle have visibility into the progress of testing and everyone works together to meet requirements, execute tests, and increase the productivity and efficiency of testing.
