Sometimes, you find yourself searching for bugs without any specific purpose or direction. This occurs when your QA team lacks the proper mindset and strategy for exploratory testing. Without clear guidance or goals, your test coverage will be inadequate, and you'll have less productive use of your time.
In his book Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design, James Whittaker provides a practical solution to help you level up your exploratory testing. He advises using a tourism metaphor as a guide to keep you from wandering around and help you make decisions while testing.
In this article, you will find out:
Metaphors are useful in exploratory testing as they help you navigate the software’s complexity by providing a framework for organizing your testing and achieving your objectives.
In the tourist metaphor, you see yourself as a tourist (tester) exploring (testing) a new city (software). Your exploratory activities will change depending on the goals of your travels.
Imagine you travel to Lisbon, and you wander the streets with no goal or idea of where you are going. Yes, you would probably see some cool things, but you would also miss the most iconic landmarks that would help you truly understand the city. In this case, you feel you saw a lot, but you barely scratched the surface. You get a false sense of coverage.
Now, imagine you travel to Lisbon but instead with a well-defined goal and some structure. Having a goal will change how you spend your time and add some structure to your exploration. Now, you get to really understand the city and its culture.
As a tourist, you have both freedom and structure, exploratory testing is no different. You get the freedom to do whatever you want, but you have the structure of deciding what method of transportation to use, what sites you will see, and how to cover as many activities as possible within the available time. Using the metaphor in your exploration journey adds structure and strategy to reach your testing goals.
The tourist metaphor uses intent to divide the tested areas. The goal is to test features and functions combined, not isolated, as they share the same resources and operate on the same internal data. As a tourist (tester), you will mix and match places to visit (functions), so you do the most with the available time.
In the tourist metaphor, James Whittaker segments destinations into six districts, each with specific tours.
Every city has an area where the work gets done and the main business buildings are located. Software is no different. There are some features and functions that make money. These features are advertised by marketing and deeply influence the user in the software's buying process.
In these districts, a tourist can take several tours:
Some tourists visit a city by following the guidebook to the letter. In this tour, a tester would follow the user manual to discover if the advertised features are working well or if the manual is accurate. The tester can discover important information and bugs like unclear instructions, unfunctional shortcuts, and more.
In this tour, you should identify the features that made customers pay for the software, also known as “the money features”. Those features are showcased by the sales team in demos. A good way of executing the money tour is by sitting and watching demo calls. Identify issues with the demos, use cases, and money features' behavior. You will not only save your colleagues from crashing demos, but you will also help them sell.
A variation of the Money Tour is called “the Skeptical Customer tour”. In this tour, you imagine what a difficult customer would ask in a demo call. Usually, he would go off script and always ask, “What if?”.
Landmarks are essential features identified in the Money Tour or the Guidebook Tour. In this tour, you create a list of landmarks, decide on a specific order and then explore one after another until you complete them. By varying the number of features and their order, you will get great coverage and be able to identify issues with the system’s operability.
This tour aims to challenge the software’s limits by asking for answers to complex questions. This is applied differently depending on the software you are testing. You should ask yourself what input or data would require more processing and would stress the system. You can challenge the system by inputting invalid data, ordering many items, and creating complicated documents to be processed.
This tour treats data as a FedEx package. Just like a package goes through multiple checks and distribution centers before it’s delivered, data is created, modified, stored, and used in different ways. In this tour, you should follow the data’s lifecycle and find every feature that interacts with the data.
The After-Hour tour focuses on money features and the maintenance tasks, data archiving, and file backup tasks that happen after hours.
Garbage collectors travel from house to house in a planned manner. Like them, in this tour, you should perform a methodical module check in which you go module by module without stopping to test in detail. Start by deciding a goal of what you will examine (menu items, dialog boxes, error messages, etc.) and then visit each one of them by following the shortest path to reach them.
Inside a city, this district is an area with buildings of historical value. Similarly, in software, the historical district is represented by the legacy code, older versions of the app, and the history of buggy functions.
Within the historical district, James Whittaker proposes the following three tours:
The Bad-Neighborhood of your software is represented by the features that had the highest number of bugs in the past and present occurring issues. Once you identify such problem areas, this tour will keep you on top of the issues. You can also pair it with the Garbage Collector’s tour to make sure that once the bug is fixed, it doesn't affect other features.
The artifacts showcased in a software museum are legacy codes. The goal of this tour is to offer some testing attention to those artifacts (legacy codes).
As the name suggests, this tour focuses on testing a new version of the application by running all the scenarios and test cases used for the previous version.
While the test tours are helping you to find hidden risks and product issues, documenting your process and findings is crucial. If you don't have a record of your session, you risk losing important details, great ideas, and relevant information about quality. Let’s see how you can easily document your exploratory testing by using the Xray Exploratory App.
This district represents the places in a city where tourists love to spend their time, and that the locals avoid going. In software, such places are usually features and functionalities that new users use while experienced users stop using them.
This district had the following tours:
Imagine you are a businessman who booked a hotel far away from the office you visit. This way, during your commute, you get to see the city. This tour suggests that you should test features that require you to travel long through the software before you arrive at the final destination. These features require long paths and many clicks to navigate.
This tour is all about looks. Ignore functionality and only focus on the interface. The goal is to identify issues with the user interface. After this tour, the software might still have bugs, but they won’t affect the looks and the first impressions.
The Test One Get One Free (TOGOF) tour aims to find bugs in copies of applications that run simultaneously. Start by running multiple copies of the application, then use features that require memory and disk usage. If you find a bug in one copy, it is in all of them.
The goal is to tour areas of your system that are hard to find. This tour works very well with large applications with areas where it is challenging to arrive at, but that are still used.
This tour focuses on collecting outputs from your system. The goal is to document all outputs of your exploratory testing session. Like collectors, you would want to have the complete collection, to visit all the places and see everything.
The main goal of this district is to complement other tours. This district represents areas where entertainment tourists would go after they visit the most important tourist attractions. In your software, this district represents supportive features. The goal is to test how they work together with critical features.
In this district, you can take the following tours:
This tour puts into light the features that are the closest to the money features. Due to their proximity to the main features, users would certainly pay attention to them. These secondary features should be bug-free, just like the main features.
The Back Alley tour proposes to test the features used the least. You can combine this tour with the Landmark tour to test combinations of the most popular features with the least popular ones. Mixing them can help you find unexpected issues as developers didn’t anticipate those features to interact together in a scenario.
Like tourists who go clubbing all night, testers challenge the software by running it without closing it. The All-Nighter tour can be used in parallel with other tours and aims to find how long the software can last before it crashes. This is especially important for testing mobile devices.
In this district, there are designated areas where tourists go to rest. Likewise, in software, this district targets functionalities that get busy when the software is at rest.
The tours available in this district are the following:
Many times, it happens that you start something and then cancel immediately. This tour has this exact purpose: starting and stopping operations. Look for time-consuming operations and test all cancel buttons and options (ESC key, shortcuts, or the X button). You will want to see if the application works properly and if the operation can be continued successfully after you cancel it.
The Couch Potato tester will force the software to work harder by not taking any initiative and doing fewer actions (accepting all default values, leaving fields blank, going through screens without clicking anything). In this tour, your non-activity forces the software into “else” clauses that are challenged by blank data fields.
Every city has a place where dangerous people are. These people are doing illegal acts and threatening the city, its locals, and tourists. In software, ill-intended users will try to crash the app and exploit its vulnerabilities.
Inside this district, you can follow several tours to simulate the actions of the seedy district users:
You play the role of a saboteur by forcing the software to take an action and restricting access to the resources needed to complete the same action.
In this tour, you have to resemble the behavior of an antisocial person who does precisely the opposite of what a community would do (opposite to the system’s logic). To do this, you can input bad or unexpected data (out of context, stupid, and without sense), input “illegal” data that breaks the rules (too long, wrong type, wrong format), or do things in the wrong order (submit an order before writing your delivery address).
This tour monitors the system’s behavior while repeating the same actions. Frequently, users take different paths that programmers didn’t anticipate. With this tour, you can discover serious issues that were not anticipated.
Test tours are helpful because they combine features and functionalities in various ways. Due to their nature, they are also easily repeatable and shareable among colleagues. Moreover, they can help you understand what effective exploratory testing looks like.
With time, you will notice that each tour discovers different issues and has different coverage ranges. Remember to record and track testing sessions and their time to execute. With these observations, you can point out when to apply each tour to maximize your testing results. You can also combine test tours to get the best coverage and results.
James Whittaker’s book shows many examples of how the tours are applied and many exercises to practice exploratory testing. One exploratory testing exercise you can easily do is to create your own tour. Doing so allows you to use your experience and creativity to develop exploratory testing frameworks.