Generative AI is bringing a new era of “convenience” with ChatGPT, from OpenAI, taking center stage in our daily lives. From aiding in solving complex problems to generating content, this large language model has become a helpful companion for various testing-related tasks. As generative AI is becoming increasingly present in our daily lives, we should understand how to use it and account for its limitations.
In this blog, we will analyze how to use ChatGPT in the context of exploratory testing and how to improve its outputs.
The essence of exploratory testing lies in its dynamic and adaptive approach to uncovering the true state of a software product through experimentation. Like a skilled explorer diving into uncharted territory, an exploratory tester navigates the software landscape, seeking clues from various sources such as UI layout, wording, logs, and infrastructure.
While traditional testing may rely on predefined test scripts and expectations set during development, exploratory testing emphasizes the importance of understanding the context in which the product operates. It's about assessing relevant quality information for the right people at the right time, recognizing that quality is multi-dimensional and subject to individual interpretation.
Now, when it comes to using AI with exploratory testing, we need to be careful. ChatGPT, while possessing impressive capabilities, has difficulties in grasping the entire context to the same extent as us (human testers). It lacks a deep understanding of the product's purpose, relevance, and its target audience, making the interaction time-consuming (contrary to its intended ease). However, this doesn't negate its potential to assist in various testing-related tasks. That is why it is important to be aware of its limitations while using it.
Generative AI (ChatGPT) relies on an extensive training dataset as its primary source and is governed by many parameters, which serve as the repository for its accumulated "knowledge." As a result, it excels at tasks like text classification, question answering, document summarization, and text generation. However, the AI and ML community has raised valid concerns regarding several limitations:
Hallucination and fabrication of facts and references;
Reliance on non-verifiable sources with limited accessibility;
Bias in generated content;
Limited context window (restriction in the number of words imputed leading to content that lacks context).
It's important to clarify that we cannot rely solely on generative AI to perform exploratory testing. Exploratory testing is a process that requires a human tester's expertise, evaluation, learning, and continuous iteration to extract valuable insights.
However, using generative AI in conjunction with us (human testers) can prove beneficial for testing-related tasks and act as a source of ideas and inspiration.
How can generative AI actually be helpful in exploratory testing?
Exploratory testing thrives on creativity, and generative AI can be a valuable tool to fuel this creativity. Due to its pre-training on large amounts of data, tools like ChatGPT are a great way to brainstorm fresh test ideas. Here are some prompts to get your creative juices flowing:
Given the following user story, can you enumerate some exploratory test ideas?
As a user, I can import attachments in a CSV file with Xray Test Case Importer. It should include Test and Test Step attachments.
Test charters are like roadmaps for your exploratory testing sessions, and AI can help you brainstorm ideas to create them. As mentioned above, ChatGPT is highly dependent on your input, so experiment with different prompts until you get the answer you need.
Given the following user story, can you enumerate some ideas for test charters related to security?
As a user, I can import attachments in a CSV file with Xray Test Case Importer. It should include Test and Test Step attachments.
After a productive exploratory testing session, summarizing your findings is essential. AI can help you do this by automatically extracting key observations, identified defects, and areas needing further exploration. Let’s see an example:
Given the following testing notes, can you make a brief sentence with the overall result of the testing session and the overall confidence about the quality aspect?
Session Objective: Explore the banking app to identify usability issues, functionality glitches, security concerns, and performance bottlenecks.
Identifying, addressing, and prioritizing risks is key to quality success. You can use generative AI to assist you in brainstorming and analyzing risks. Remember that your input is critical to generating quality answers (garbage in, garbage out).
How you can use ChatGPT to brainstorm risks related to a requirement:
Given the following user story, can you enumerate some risks related to it?
As a user, I can import attachments in a CSV file with Xray Test Case Importer. It should include Test and Test Step attachments.
Another way to use generative AI for your exploratory testing is by asking for variations of test scenarios to explore different paths and behaviors in the application. ChatGPT can act as a helpful assistant to ignite your creativity and give you a good starting point for your exploratory sessions. Remember, the answers will vary and may be incomplete. Take everything with a grain of salt.
Create performance testing scenarios for a mobile banking app to assess its responsiveness under heavy user load.
Generating sample data is often a time-consuming task. Why not use generative AI to help populate user profiles with realistic data or create datasets? Let’s see a prompt example:
Generate sample financial transactions for a banking application, including deposit, withdrawal, and transfer transactions.
As the output of generative AI can vary significantly, our goal is to improve the quality of its answers. To get the best possible answers, we should be mindful of our intent. When aiming for an answer, we want it to:
To improve the quality of ChatGPT answers, we can take the following steps:
AI can undoubtedly assist in some aspects of testing, but relying solely on it for exploratory testing might leave important stones unturned. It's like having a high-tech metal detector, but you need a seasoned explorer to decide where to start scanning.
So, the key is finding the right balance between AI's capabilities and human expertise to ensure context-aware exploratory testing. When used with careful consideration, ChatGPT can help you brainstorm ideas and kickstart your sessions. But always remember to be mindful of your inputs and to consider the answers attentively.