How do you report on quality, what do you look for, and what are the most important metrics to measure?
"Quality is "value to some person" (quoting Jerry Weinberg) and as such is highly subjective. Because of that, there is rarely a set of metrics that can be used as a recipe for most cases; which in turn makes indicators of readiness to deploy, data to be collected, etc. highly contextual."
João Proença, Quality Owner at Outsystems
Quality can be an elusive concept that is hard to measure, but it’s important to define it within your organization if you want to accurately measure your testing progress, access risk and make improvements.
Measuring quality can be challenging due to the many different dependencies and factors and the best way to approach it is to define what quality means for your organization and how you can get there.
To assess quality, you have to gather evidence about: how you tested, what you found and what that tells you about business risk.
In this blog post, we’ll show you what to look for when defining quality and how that relates to business risk. Then jump into our ebook: How to measure and report quality metrics to dive deeper.
How to define quality for you
João Proença, Quality Owner at Outsystems tells us his approach to defining and measuring quality. “Given that quality is subjective to each context, you should first try to understand what quality means to your team, organization and the customers you serve.
"Then, teams should ask themselves where they want to be in the foreseeable future when it comes to quality, and what are the best small experiments they can conduct to take a step in that direction. Identify the metrics that allow them to evaluate the success of such experiments and that will give confidence to the team that they're taking the right steps (or not)".
Since testing is a means to reach quality, you want to first define quality, and then set your testing metrics to measure your progress. The right metrics measure your current status of quality and testing, and guide you with valuable insights as to where and how to improve.
Breaking down quality helps you extrapolate the right metrics to measure your progress. If you’re not tracking your progress, how will you know what’s working in your testing and QA strategy and what isn’t?
Here’s an approach to better understand how to measure and report quality:
Assess test coverage
Questions you can ask yourself are “what are we testing and what are we ignoring?”. There are many different sides to coverage and it’s important to first understand what you’re doing and even better to be able to explain why.
Understand what you're doing
Once you’ve taken inventory of what you’re doing and the kind of tests you’re running, you need to understand why you’re doing it. Dig deep and try to answer why certain problems are coming up. Then try to answer, what are your findings telling you.
How does what you do relate to business risk
Finally, take it one step further and explore how what you’re doing relates to the big picture goals. What is the purpose of the project, what clear objectives do you need to meet? What are customers and stakeholders expecting? Answering these questions will help you think about what kind of metrics are important to pay attention to in order to reach those goals.
Leverage Xray reporting features to make informed decisions
QA Managers need metrics to measure the effectiveness of their testing strategy and methods. The right metrics guide your decisions and help you make pivots and change direction appropriately.
Knowing what information you’re looking for is a good first step to being able to measure and assess how you’re doing.
If you want to dig deeper and understand how to use reporting to reach organizational goals, and how to leverage Xray built-in reports, dashboards and agile boards to make informed decisions, we’ve created a resource that packs it all in.
