Success. We all seek it.
However, what defines success?
It all depends on the goals we have established.
We often hear that testing is about ensuring there are no bugs. That's wrong in many ways. First, testing doesn't assure; testing gives information about quality so the team can decide what to do next, hopefully using it to improve its current status. Second, having no bugs is platonic; it assumes we all have the same understanding of quality, which we don't, by the way, and it also implies that we could test the product in infinite ways, which, of course, we can't.
What is a more correct goal for testing that ultimately defines its success or not? Are there particulars for exploratory testing?
We can say that, in general, exploratory testing is successful if we can:
Let’s drill down into each one of these.
Finding issues without any criteria is easy. Finding problems that matter is harder.
Quality is multidimensional, so we can look at different quality criteria. We cannot look at all of them because we have to manage the available effort/time.
Therefore, we are successful when we can:
Looking at the last point, we can see that it has a bunch of subjective aspects: "timely enough"; "deeply enough"; "good enough". It means that, in the end, we will have a subjective quality assessment.
The tricky part is that we connect success in testing with finding bugs. Finding bugs is one of the outputs - not the only output. We can succeed in our testing goals but still have not found any problems. That's fine because our goal is to find information about quality (i.e., perform a quality assessment).
Therefore, even though the result of our quality assessment is relevant to the team and other stakeholders, the success of exploratory testing is not only dependent on what we find.
Returning to what was mentioned earlier, we are successful if we target relevant quality attributes and can use the best of our testing skills to expose important risks to these attributes.
What is value? What can impact it?
These core questions will help us tailor our testing.
This is not specific to exploratory testing; this applies to testing.
In testing, no matter the approach, we look for quality information that is relevant to stakeholders that matter. We are not on the right track if we find information that is useless.
Let's see some typical examples of stakeholders and a very, very limited set of what they might care about:
Are we considering exploring these or other concerns identified by your stakeholders during our testing?
Our testing is only successful when we have the stakeholders' requirements in mind.
Whenever thinking about exploratory testing, we can implement the whole “process” in different ways.
We can look at success from a process perspective, at each " component”. As exploratory testing can be adopted in many different ways, you must look at your context:
If your team adopts the concept of sessions (or Session-Based Test Management) with a limited, focused time for testing:
During exploratory testing, testers can produce a set of outputs other than a simple pass/fail:
Typical outputs:
We can have an idea of success around the existence of these outputs. However, these outputs only need some to exist simultaneously. Additionally, and as a mere example, the quality of a bug report is subjective.
Beware of false or incomplete metrics preventing you from performing good testing.
A few examples include:
Note: We don't control the quality of the product; we can get a good enough idea about its current quality.
Measuring success in testing is hard. We should look at what comes from production and use it as feedback to improve our overall testing approach. If major issues escape, that's a clear sign that we must improve.
As testing supports the product team and other stakeholders, we must establish an ongoing conversation with them to clarify what we're testing, if that makes sense, and what we could not test if we communicate our findings most effectively.
Additionally, we must maintain exploratory testing of how the team works. Were we able to incorporate it effectively and efficiently into the existing processes?
Achieving success requires continuously improving and incorporating the findings in our testing process.