In a world where microservices have become very popular, the connectivity aspects between systems take center stage when it comes to prioritizing quality engineering. Postman's State of API report highlights that technical and non-technical stakeholders recognize the growing importance of APIs and their testing as part of the enterprise strategy.
While scripted testing for APIs can help debug specific problems, it has the same shortcomings as its counterpart for UI - primarily, the lack of ability to uncover unknown problems beyond the strictly defined, “more traveled” paths. That’s why exploratory testing is gaining ground on the API side, unlocking faster feedback, more comprehensive risk coverage, and more creative problem-solving.
This article will review some of the ideas to keep in mind as you kick off your exploratory testing for APIs to help you get started.
Useful Tips for API Exploratory Testing
Overall Approach
One of the misconceptions is the prerequisite of high coding knowledge. Still, the level of similarity with GUI testing is high enough that you can get started without coding skills.
Gaining product knowledge is one of the cornerstones of the process. You can start by checking some of these sources:
- API Specification
- Data Dictionary
- Source code (if you have access and technical knowledge)
- If you are using an IDE, the auto-complete suggestions can contain valuable hints as well.
The first step is to understand the list of common use cases and (at a high level) corresponding API calls.
In most cases, there should be consistency between API and UI, so you can use the product interface in parallel to help with your exploration. Any discrepancy could signal that something is wrong, or maybe you have discovered behavior not adequately reflected in the official documentation.
Next, you can dive a bit deeper and look for the more detailed information about the architecture - protocols, available endpoints, parameters, and methods. Since every API endpoint is typically built for a limited purpose, it would contain a mix of universal and custom rules, make sure to get the information on both types.
Going through those aspects and asking questions will help you establish the feedback loop with the developers and other stakeholders, better understand the product, and refine your testing strategies for future exploration.
User Roles and Environment
The key consideration here is the mix of human and non-human actors interacting with your API.
On the human side, you should impersonate internal and external developers (both senior and junior), business stakeholders, “regular” consumers who mostly use the front end, as well as hackers.
On the non-human side, try to have a simplified diagram of other services or platform dependencies (including test runners and delivery channels) and the nature of each communication. Some of the questions to consider:
- Is it one-time calls or more of a back-and-forth dialog? If the dialog, how is data retained and what are the conditions for state transitions?
- Does the formal contract exist? How is it updated?
- Are there back-up options for those services?
- What are the bandwidth, performance, timeout, etc. expectations for each party?
In the longer term, as AI becomes more integrated in different application areas, you should consider it as one of the possible suppliers or consumers and have dedicated techniques/heuristics based on the scope of your AI implementation.
Based on the user roles, environment, and your skills, decide on the toolkit - a tool like Postman or Cypress is more intuitive and accurate in simulating less tech-savvy users but doesn’t necessarily represent the “true” experience of e.g. “experienced hacker” or “senior developer” roles. So, you may need to access the APIs in a more direct manner.
Exploration Techniques
Many common heuristics and techniques would be applicable in this use case, for instance:
- CRUD (Create, Read, Update, Delete) actions and associated risks
- Fuzzing (e.g. missing, extra, or malformed data; values beyond boundaries; special character handling)
- You can take some of the inspiration from the “Big List of Naughty Strings”
- Format variety and mismatches
- Troublesome behavior patterns (interruptions, skipped calls in a sequence, out-of-order actions, asynchronous events, etc.)
- Authentication-related methods (interaction with different levels of authorization, without authorization/with expired token, etc.)
Keep in mind that certain buggy patterns show themselves only with repetition, so be sure to run the critical flows more than once.
Remember, the non-functional side - security, performance, and usability are just as important for APIs (we mentioned some of the essential aspects of NFR testing in our testing for non-functional requirements guide).
Be sure to spend some time on edge cases around impatience/flooding of calls and characters associated with security risks (cross-site scripting, SQL injection, etc.). Also, with all the knowledge acquired through exploration, the frequently overlooked angle re-evaluates the clarity and completeness of documentation and examples/snippets for both new and experienced users.
Last but not least, throughout your testing, pay attention to API reporting and error handling - is it detailed enough to troubleshoot? Is it clear, can the artifacts be easily shared?
Advantages of Xray Exploratory App for API testing
- Regardless of the target for your exploration, there is a need for collaboration and detailed reporting, especially for larger enterprises.
Xray Exploratory App allows you to easily consolidate different types of evidence (video, screenshots, network logs, etc.) along with your notes in a consistent way. With version 3, in addition to shareable reports, you also have access to the cloud session storage, which makes involving your colleagues easier. - Standardization at the organization level is useful to improve efficiency and gain better insights from analytics. With the seamless integration between Xray Exploratory App and Xray, you can have your scripted (manual or automated) and exploratory API tests in the same management system. Such a systematic approach ensures consistent coverage and intuitive traceability.
As a side note, you can quickly import the results of your automated API testing into Xray (e.g. from Postman, on Cloud or DC). - We mentioned an overall understanding of the product and its use cases as a key element of your exploration. One other source for that could be the session storage of Xray Exploratory App v3. Imagine that your colleagues did a few sessions via UI several releases back, and now you are tasked with the API side. By opening the previously saved sessions and retracing their steps, you can not only learn about some of the API workflows, but also borrow some of the insights about the tricky things to explore.
Improving API quality with Exploratory Testing
As robust API testing becomes more critical in enterprise workflows, we will likely notice a stronger push towards standardized API testing practices within organizations and an increase in automation and exploration.
Applying exploratory testing to APIs enhances early feedback loops, helps you build a more comprehensive understanding of the product, and boosts your chances of finding more tricky and damaging defects. And the entry barrier is low - it can be surprising how much of your QE or business skillset is transferable to exploratory API testing. As a passive bonus, you can build more technical skills as you go.
We encourage you to add this technique to your arsenal and hope that you find our tips helpful when you do.
