Ever felt overwhelmed by the complexity of the system that you’re testing? Navigating intricate software systems can be daunting, leaving us feeling lost and unsure where to begin. While exploratory testing gives us some degree of freedom to test, we still need some guidance on how to approach those complex systems.
This is where heuristics might help by guiding us to efficiently navigate the complexities of software systems and uncover hidden issues.
In this blog post, we’ll share a perspective on these “rules of thumb”, understanding their purpose and significance in the context of exploratory testing.
Heuristics are cognitive shortcuts that aid us in problem-solving and probability judgments. They are mental strategies that operate as generalizations or rules of thumb, lessening cognitive burden.
In exploratory testing, the role of heuristics is paramount, as they provide us with cognitive tools to navigate through the software systems efficiently. Heuristics can, therefore, help us make informed decisions about what to test, how deeply to test, and where to focus our efforts during exploratory testing sessions.
These include experience-based heuristics, which rely on past knowledge to uncover common issues, functionality-based heuristics, which focus on exploring software features, time-based heuristics, testing under different time conditions, and risk-based heuristics, prioritizing testing efforts based on risk factors.
Compared with alternative methods and techniques, a heuristic evaluation method proves simpler and swifter. It enables us to identify issues within a tested product, consuming minimal time and material resources.
The primary advantages of testing heuristics include:
To sum it up, heuristics are linked to a specialized test design technique based on previous experiences. They assist us in making informed decisions while making sure the outcomes come as close to “optimal” as possible.
We can often confuse heuristics and mnemonics. Mnemonics are memory aids that help us remember information more effectively. They often involve creating associations, patterns, or acronyms to make information more memorable and easier to recall. In the context of testing heuristics, mnemonic-based strategies help us categorize and remember different types of heuristics and their applications.
Derived from testing experts like Cem Kaner, the heuristics below offer invaluable guidance for uncovering hidden issues and evaluating software quality. Let’s dive in and explore some good examples we can use at work:
In exploratory testing, it's important to identify the "Never" and "Always" behaviors of the software we’re testing. For example, software should always maintain accurate account balances and never compromise user data.
This heuristic involves considering scenarios with zero, one, or many instances of a particular element or attribute within the system. This can apply to various aspects, such as search results, characters in a string, or files in a directory. When applying this heuristic, we should watch for:
- Issues related to plurals, like displaying "0 record found" or "1 records found.";
- Problems with count or percentage calculations, such as divide-by-zero errors or off-by-one errors;
- Indications of performance degradation as the count of elements increases, which can highlight scalability issues.
The "Beginning, Middle, End" heuristic suggests varying the position of an element in a sequence or taking action on an element at different positions.
Examples include: deleting an item from the beginning, middle, or end of a list; pasting text at different positions in a line, or placing special characters at different positions in a value.
It's important to watch for indications that position matters and to be alert for undesirable behavior, especially when talking about software testing.
In exploratory testing, the CRUD heuristic - Create, Read, Update, Delete - provides a fundamental framework for actions on data elements within a system. Integrating CRUD with other testing approaches enhances exploration:
The "Follow the Data" heuristic guides us to conduct a sequence of actions on data, ensuring its integrity throughout. This approach is relevant in systems where users interact with data, such as entering, searching, updating, and reporting. When combined with other data-related strategies, it enhances testing effectiveness.
For example, we can import data containing special characters to assess system vulnerabilities, generate data violating domain-specific rules to identify logic gaps, and perform data-centric actions to understand system behaviors associated with different data quantities.
The "Some, None, All" heuristic involves considering different states of a defined set of items or attributes, such as permissions, configuration options, tags, multi-select fields, and checkboxes. We should watch for:
- Instances where "None" is treated equivalently to "All," such as the system treating a user with no permissions the same as a superuser with all permissions;
- Problems with count or percentage calculations, including errors like divide-by-zero or off-by-one;
- Display issues, particularly regarding the representation of "None" or "All.";
- Potential data truncation issues when dealing with "All" items within the set.
The "Starve" heuristic involves deliberately depriving a program of essential resources such as CPU, memory, network bandwidth, or disk space during testing. We should closely observe for potential outcomes, including unrecoverable data loss or corruption due to resource depletion, unhelpful error messages hindering diagnostic efforts, and unexpected program or session termination often resulting from resource exhaustion.
The "Interrupt" heuristic involves intentionally interrupting key processes within the system, such as logging off, shutting down the machine, rebooting, killing processes, disconnecting from the network, or forcing hibernation, among others. We, as testers, should monitor for misbehavior, such as unhelpful error messages hindering troubleshooting, an inability to restart the process, and potential unrecoverable data loss or corruption resulting from interruptions.
The "Change the Model" heuristic suggests transforming the representation of the system being tested into a different format. For example, we might convert a state diagram, typically composed of circles and arrows representing states and transitions, into a table format. This table could organize states as columns, events as rows, and resulting states as data within the cells. Similarly, we could translate a linear outline into a mind map, or vice versa, to offer alternative viewpoints on the system's structure and connections.
It is important to note that, despite these advantages, heuristics have some limitations.
Following the utilization of heuristic techniques, which aid in identifying errors in the testing environment, it becomes important to prepare checklists beforehand.
Subjectivity represents another hurdle. Heuristics are inherently based on subjective judgments and experiences, leading to variations in interpretation and application among individuals. This subjectivity can introduce inconsistencies, thereby impacting the reliability of testing outcomes. Which may lead us to biases.
Biases are yet another challenge. Our testing biases can influence the selection and application of heuristics, potentially skewing testing results. This may cause us to focus on affirming existing beliefs rather than objectively evaluating the system under test.
Moreover, false positives and negatives represent significant risks. Heuristics can sometimes yield false positives (identifying non-existent issues) or false negatives (failing to identify genuine issues), potentially leading to inaccurate conclusions. We must exercise caution and validate heuristic-based findings meticulously to mitigate these risks.
As highlighted in this post, heuristics empower us to navigate complex systems with clarity and purpose. They are extremely useful when creating test charters, as they help us decide what areas to focus on and what tests to perform. We can start by creating a list of charters inspired by the heuristics that apply best to your context.
If a heuristic doesn’t suit your specific context, you can choose not to use it entirely. You can also customize it to fit your needs or enhance it by adding your own elements. In certain situations, you can even combine heuristics.
With time, you’ll be able to write your own heuristics based on your testing expertise. Start by examining your testing experiences and identifying recurring patterns or strategies that have proven effective. Then, find the why behind your actions to identify your heuristic’s core.