Regulated industries and exploratory testing - Xray Blog

In regulated industries, compliance, consistency, and safety are paramount. Structure and predictability are the hallmarks of a regulated test strategy. As a result, software testing is often limited to step-by-step, scripted procedures.

Yet, unscripted exploratory testing is emerging as a modern best practice, even in highly regulated domains.

At first glance, exploratory testing may appear to be at odds with the rules of a regulated industry. It relies on experience and spontaneity rather than scripted instructions. But delve deeper, and you’ll find that it complements and enhances the rigid testing we've come to rely upon.

In fact, regulatory bodies, such as the US FDA, now recommend exploratory testing as an effective tool for building a more risk-based approach to software testing.

As we review exploratory testing, this article will show its usefulness and alignment with industry regulations. We'll discuss real-world use cases and provide a roadmap for adding exploratory testing into a regulated test strategy. 

What is exploratory testing?

When we think about software testing in regulated industries, we almost always think about scripted testing. These are test scripts with step-by-step instructions for the tester:  

Exploratory testing, however, is a form of unscripted testing. 

ISO/IEC/IEEE 29119-1:2022 defines it as:

In simpler terms, exploratory testing is when the tester creates a test script as they go. They rely on their intuition and experience to decide what to do next based on what happens in the software.

Scripted testing is like following a GPS with exact turns and stops. Exploratory testing is like wandering through a city, using intuition and experience as your guide.  

Does exploratory testing align with industry regulations?

Yes. Definitely. 

First, exploratory testing doesn’t replace the scripted testing we’re already doing. It’s added on top. 

Furthermore, it provides several benefits that align with regulated software testing goals:

Higher Software Quality

Users are unpredictable. They try things we never anticipated, and they sometimes expose bugs in the process. Exploratory testing is better equipped to find those bugs than scripted testing. You can only write a script for the errors you anticipate in advance.

Improved risk management

Exploratory testing lets testers rely on their experience and intuition rather than a script.  This will help you identify and mitigate unanticipated software hazards early.

Time efficiency

In the life sciences industry, we spend a lot of time on testing overhead:

Maintaining accurate test scripts requires a lot of overhead.
In practice, it is common for teams to find more defects in their scripts than in their software.

It’s a lot of extra work that is important for high-risk, safety-relevant testing. However, that additional work is unnecessary overhead for lower-risk testing. Exploratory testing is a great way to cover low-risk scenarios quickly and without the overhead. 

This is what the US FDA says about using unscripted testing to support a risk-based strategy:

Note: The above reference is from a draft (non-binding) guidance document. Yet, it clearly shows the regulator’s current thinking on exploratory testing.

What are some real-world use cases for exploratory testing?

Here are a few use cases where exploratory testing can provide tremendous value:

• User validation testing: Ask real users to try your software without a script. Just give them an objective and document their actions and observations. You'll know with confidence whether the software meets user needs and intended uses;
  
  
• Low-risk regression testing: Many teams rely on manual test scripts for regression testing. This creates a process bottleneck at release time. Exploratory testing provides an efficient alternative to manual test scripts for low-risk scenarios (especially scenarios that are not easily automated);
  
  
• Bug hunts: A great software tester is exceptional at finding bugs. They have an intuition for breaking software. Adding exploratory bug hunts into your process creates more opportunities for your testers to be great testers. You’ll find software defects much sooner than before.

How do I build exploratory testing into a regulated test strategy?

These are the basic steps for adding exploratory testing into a formal test strategy:

Step 1: Create an exploratory testing workflow. An exploratory testing workflow looks a lot like a scripted testing workflow. The only difference is that you will replace your test script with a test charter.

Otherwise, you’ll follow the familiar regulated testing workflow, including pre-execution and post-execution approval:

When defining your test goals and the test scope, it’s important to know the difference. While test goals define the objectives of testing, such as identifying bugs or validating functionality, test scope delineates the boundaries of testing, outlining the specific features or aspects to be tested.

As specified under item 1.4, when it comes to regulated industries, it's important to add requirements coverage as applicable, ensuring comprehensive testing that adapts to evolving project needs¹.

Step 2: Update your verification and validation (V&V) plan. You’ll need to add a few declarations:

• Where in your process will you use exploratory testing? There are a few ideas in the prior section;
• How will you manage exploratory testing? Describe the process you designed in Step 1;
• If you use risk-based regression testing, what is your risk assessment process? Provide a description or a reference.

Note: Of course, you may need to make more updates. That depends on the content of your current V&V plan and the advice of your quality stakeholders.

Step 3: Choose a tool set. We’ll look at that next;

Step 4: Start testing!

Which tools should I use?

Exploratory testing doesn’t need expensive technology. Some common toolsets are:

• Documentation Apps: You can manage basic exploratory testing in any traditional office suite (Office 365, Google Workspace, etc.). Create your test charter in a document and route it for approval like any other. Spreadsheets work great for journaling test steps and recording results;
  
  
• Test Management Software: If you use modern test management software, you can also use it for exploratory testing. Create your test charter as a test case. (Xray provides the Generic test type for this.) Record your test steps directly into a test execution or attach a spreadsheet of steps as evidence;
  
  
• Exploratory testing software: This is the recommended option. A specialized exploratory testing tool will make journaling test steps and capturing evidence a breeze. Many, like the Xray Exploratory App, can be downloaded for free. Of course, you don’t have to use a specialized tool. It’ll just make everything easier.
  
  

Ready to get started?

Exploratory testing can transform your test strategy. You will find more bugs much earlier in your process than ever before. Most importantly, you can use it in a way that is compliant with industry regulations. 

I encourage you to get started right away:

1. Download an exploratory testing tool. (Like the Xray Exploratory App);
2. Do a demo for your team and stakeholders;
3. Create a workflow and update your verification and validation plan;
4. Start testing!

For expert guidance tailored to the life science industry, consider collaborating with Agile-Innovations.Tech, a seasoned expert specializing in the life science industry.