In the past, testing for non-functional requirements (NFR) was often neglected, done to the extent possible at the end of the life cycle. For modern development teams, that is no longer an option - to deliver high-quality applications, they have to implement a comprehensive testing strategy with proper allocation of resources to both functional and non-functional components.
Furthermore, exploratory testing (ET) is being used more frequently to ensure compliance and find more convoluted flaws and bottlenecks in intricate software systems. The trend is not surprising as the cost of compliance is increasing, both from the customer and the regulatory perspective.
As we expect an increase in demand for expertise in exploring non-functional areas of software products, we have prepared this guide to help you get started.
Note: here, we assume a familiarity with the basics of exploratory testing, you can check our previous exploratory testing blog posts and tutorials.
Before diving into the exploratory aspects, let’s briefly talk about non-functional requirements and their testing overall. The high-level goal is to validate product properties that matter for internal and external stakeholders but do not fit the functional category.
|
Non-functional Requirements |
Functional Requirements |
|
|
Focus |
Product attributes, the “how” |
Product features, the “what” |
|
Types |
Performance, security, usability, etc. |
Unit, UI, API, SIT |
The ISO/IEC 250XX standards (e.g., 25002, 25010, 25019) cover many foundational aspects. It is common to group NFRs and evaluate categories as a whole, e.g.:
When it comes to testing NFRs, there are plenty of established scripted methods, especially automated ones. They often require a suite of specialized tools or internal telemetry to validate compliance, for instance, security testing tools like SonarQube, Snyk, ImmuniWeb and performance ones like WebLoad, BlazeMeter, K6.
However, as systems and requirements evolve, those scripted checks become insufficient - the creative aspects of exploratory testing could provide significant benefits.
ET is a versatile approach that is especially useful for complex, high-risk areas. It complements automated testing by allowing testers to uncover unexpected issues, usability problems, or edge cases that scripted tests might miss. Of the many NFR testing types, in this beginner’s guide we are going to focus on the ones we consider more suitable for exploratory testing, namely:
You would follow the process described below:
There are two “challenges” you will need to overcome:
Going into exploratory sessions with a completely “blank slate” can sometimes be useful, but in general, we recommend skimming through product requirements and/or documentation in advance. That gives you an opportunity to ask critical questions and suggest improvements early in the process. It is critical to collaborate with different groups of stakeholders at this stage to understand user desires as well as technical perspectives.
Some of the best practices you want to see from the non-functional requirements are:
With that said, sometimes requirements do not quite follow those best practices, which is when exploratory testing and its fluid nature can shine the most. We suggest leveraging common standards for guidance (or at least inspiration), some of the examples are:
One of the recent examples: CSA is a new set of guidelines from the U.S. FDA. It includes recommendations on when to use unscripted testing in regulated environments.
To improve clarity, you may need to graphically arrange this information in the format that works best for you (e.g. workflows/diagrams or mind maps), to comprehend how the product functions end-to-end and to highlight important ideas and scenarios.
As we mentioned, exploratory testing is a complementary approach, meaning you must identify which NFR aspects are already covered with automated or manual scripted testing. Then, you have to decide which of them are critical enough to double-check in your sessions. At the same time, you need to note down the gaps you would focus on.
Possessing both pieces of knowledge will allow you to more clearly understand the role of your exploratory sessions at the project level and to avoid mismatching expectations across stakeholders.
Again, a completely unstructured approach could be useful at times, but would struggle with coverage and traceability, especially in large and complex projects. We recommend applying at least some structure elements to your exploratory testing, for example session-based test management (SBTM).
One important aspect to keep in mind: your non-functional exploratory sessions will often be multifaceted - focused on a certain NFR type, but not restricted by it. e.g., performance and usability will always be “in the background” whenever you are exploring for security or compatibility.
We suggest using test charters (with or without mind maps) to guide your exploration. Some of the commonly used sections are:
Last but not least, based on those charter sections, identify whether there is any tool preparation you need - real devices, virtual environments, or accessibility helpers like screen reader programs.
Objective: Assess the personal account section of the updated e-commerce platform to discover any issues.
Primary focus = usability, especially around accessibility like Color Contrast and Alt Text.
Secondary focus = security and performance.
Scope: Registration, Login/Logout, all Account Management features (see Constraints) using Win 11 and Mac desktops with Chrome/Safari browsers at 1920 * 1080 resolution.
Constraints:
Notes:
Your priority will often be “unhappy” paths that automated tests may not cover (sufficiently or at all) - “why would a user do that” kind of scenarios that heavily rely on creativity and domain knowledge.
As you explore:
Feedback loop speed is one of the key advantages of exploratory testing, so reporting is not really a separate step - document and report any issues, insights, and observations immediately. Having a tool like Xray Exploratory App that easily allows creating evidence and defects without disrupting the session helps tremendously.
For better analysis, you can apply the combination of labels, components, and custom fields in Jira to assist with defect categorization. Consistent report formats along with the production monitoring will allow you to track your NFR testing trends.
Next, we will talk about specific scenarios by NFR type sticking with our e-commerce charter (but they would be applicable to most software applications).
We start with usability because it is arguably the least automatable area. It involves many aspects defining user experience with the product - ergonomics, intuitiveness, accessibility, etc. One popular approach by Nielsen Norman Group breaks it down into five dimensions:
Sample scenarios - “How does an application handle…”:
Sample metrics to keep an eye on:
For accessibility specifically, you can leverage WCAG checklists based on the level for your product to achieve. For learnability, an often overlooked aspect is the documentation testing (including any onboarding tours or videos, if applicable).
The degree of rigor will depend on the volume and nature of sensitive information in your product, if any. Scripted testing typically covers an “average user”, which means the 2 personas you would want to pay the most attention to are:
Study the latest vulnerabilities and exploits, then try to recreate the most critical and creative ones. In addition to universal standards we mentioned earlier, you can also consider CIS and OWASP benchmarks.
Sample scenarios:
This testing evaluates how well your product functions with other systems and how easy it is to interact with your product across different environments. Basic tests - like installation on popular platforms - should be handled by the scripted side, so the multitude of external/integration factors is your focus.
Sample scenarios:
This testing is rather demanding when it comes to device access, you can consider a virtual solution, e.g. BrowserStack.
As we mentioned, this NFR type will accompany all your exploratory activities as at least a background thought. But there are also a few dedicated scenarios to keep in mind, especially around speed and resiliency components. While scripted tests will often take care of the volume aspects, you would focus on performance in “awkward” and really stressful conditions.
Sample scenarios:
Non-functional testing keeps growing in importance, helping teams achieve lower production risk and expenses related to the product’s security, performance, usability, etc. Exploratory techniques add crucial human unpredictability, substantially increasing the chances of delivering high-quality products that users will enjoy.
With the comprehensive testing strategy involving functional and non-functional tests, scripted and exploratory techniques, it is more important than ever to have visibility across all of your testing efforts. A dedicated test management tool like Xray, in conjunction with Xray Exploratory App and Jira, enables you to consolidate test entities and reports and to boost end-to-end traceability along with insight generation.
See for yourself how easy it is to:
You can also check out the on-demand Xray walkthrough and the Xray Exploratory App intro videos.