The complete guide to Risk-Based Testing

If your testing or QA team has been struggling with deciding where to allocate your time and effort, a Risk-Based Testing Strategy could be transformative. Simply put, Risk-Based Testing is a strategy that uses defined risk as a means for adequate testing in order to maximize your testing goals.

This article will introduce you to Risk-Based Testing - by the end, you’ll be able to understand the purpose and process of RBT, as well as how to evaluate risk at different levels. Keep in mind that RBT is built on top of Risk Management and a lot of similar concepts and principles apply.

If you’re already familiar with the concept and want to learn how to perform RBT using Xray, please take a look at How to Use Xray for Risk-Based Testing. 

What is Risk-Based Testing

RBT (Risk-Based Testing) is a testing strategy that follows the principles of Risk Management but in a testing context.

In software development projects, there is a set of common objectives, including:

• "Quality";
• Cost control;
• Time-to-market (i.e. target release/delivery dates).

These goals can be affected by risks coming from different sources (both internal or external) due to many different factors. In case of negative risks, the risk level will depend on things such as the complexity of the change, the number of impacted users, frequency of usage, likelihood of failure, etc.

Risks can affect the overall quality, including the customer perceived quality.

Quality is a broad word embracing "fit" of the product to customer needs and the many quality attributes which risks may affect. An intrinsic part of quality is ensuring that a product has few bugs (or at least a reduced probability of bugs on important/risky areas).

If testing is performed as a means to find/avoid bugs, RBT can be used as a "mitigation," or even as an "avoidance" strategy. On the other hand, testing is also about understanding, discovering and providing valuable feedback that can help build "better" features and better products. Therefore, RBT assists in making testing related decisions based on the assessment of risks.

Purpose of Risk-Based Testing

The main purpose of RBT is to use risk management principles for adequate testing.

First of all, RBT provides a framework for clear communication and discussion about risks between testers, developers, clients and stakeholders. RBT will help you define terms and agree on a common language, and through this it will make risk visible and actionable for you.

RBT takes into consideration the big picture, as it covers customer needs and also the needs of the development team. It specifically uses risks as the input to support testing activities. 

Typically, customers are most concerned about business features, timing, visible quality and costs. On the other hand, the development team has similar concerns; however, it sees quality in a broader scope as it has to maintain and possibly evolve with the product being developed. 

Timing and costs need to be managed effectively to be on a budget and avoid delays; however quality must not be hurt and if decisions need to be taken to meet timing/cost criteria, RBT can provide the means to ensure that focus will be given to the features/issues that matter most to customers.

Both customers and the development team want to avoid important defects, so, RBT focuses the testing efforts on what matters most - where we can get the most value from.

RBT's benefits can be summed up as:

• More focus on the customer
  • RBT will help you test more thoroughly what customers are most concerned with;
  • Delivers what is most important for the business.
    
    
• Reduced probability and impact of negative risks
  By focusing the testing on higher, negative risks, probability of missing important defects lowers and therefore the probability assigned to the risk lowers as its corresponding risk level. 
  
  On the other hand, as testing also provides feedback to the team including its developers, the impact of a certain risk may also decrease as the feature being worked out may be done differently or better.

• Increased confidence
  RBT can help find more important defects first, by focusing testing on higher risks.
  
  By focusing testing on higher risks (or risky areas/features), RBT ensures that important items are exhaustively tested and important defects are found sooner; thus, product and it's most important items (e.g. features) can be released with a high degree of confidence, while ensuring they meet expected quality goals.
  
  

• Optimized QA efforts and cost
  RBT can answer questions such as...
  
  What should we test?
  
  • • Where should we start?
    • When can we stop testing? 
    • Can we reduce the testing effort somehow? How and at what "cost"?
    • If we have more time for testing, what is the best way to take advantage of it?
    
    RBT provides the means to define the testing scope, focusing on what is relevant, by identifying what tests to execute and their execution priority, given time and resource constraints; the overall amount of test cases may be reduced as not everything needs to be tested or be tested in the same depth;
    
    RBT provides a way (based on risks) to choose tests for regression testing;
    
    RBT provides some clues for selecting candidates for automation.
    
    

• Increased risk level of positive risks
  If an opportunity is identified, RBT can be used to provide thorough testing and take advantage of it. Using testing as a constructive feedback loop, knowing the opportunities and their relative relevance, can increase the likelihood, impact and the overall risk level for opportunities
  
  

• Make better decisions based on risk
  Sometimes you may need to ship the product sooner, for time or budget constraints:
  • • RBT can give you visibility of risks, so you can take a go/no go decision "knowing the risk";
    RBT can be used to overcome risks that block "acceptance" (i.e. customer acceptance of the release);
    
    RBT implicitly explains why certain tests were executed over other ones, and thus ease communication with other stakeholders and avoid discussion on why some tests were not executed at all RBT at Different Levels.

How to evaluate risk at different levels

Many teams perform RBT identifying and evaluating risks at "requirement"/Story level: thorough testing is performed on risky "requirements," while less risky "requirements" are tested in a more pragmatic way.

However, sometimes "requirements" may not exist at all. This may be common in scenarios where teams inherit legacy projects, that have few or no "requirements" but that have Tests that validate the intended behaviour.

RBT is much more than selecting what Tests to execute and their order; RBT affects test planning, authoring and execution.

You may implement RBT in several different ways, formalizing the process a bit more or not.

The overall RBT process

• What and how can "things" affect our project?
  
  
  • Risk Identification
    • Identification of risks to product (e.g. software/hardware) features. This is normally performed by the team and discussed together;
    • Risks are identified before implementation starts and are reviewed throughout the development life cycle; new risks can be identified during implementation.
  • Risk Analysis
    • Risks are discussed together in the team (may involve customer);
    • Risks impact and probability are calculated, and thus the related risk level.
• How will we handle it in the most effective way (i.e. what can return the most value)?
  
  
  • Risk Evaluation and Treatment
    • Test Planning - using the input from Risk Analysis, the test manager can:
      • Define test strategy (e.g. level of testing to perform, techniques to use, environments to choose);
      • Estimate testing effort;
      • Define/estimate schedule (target dates, number of testing iterations).
    • Test Design
      • Specification of the Tests taking the identified risks as input: tests are designed to mitigate the risk (i.e. diminish their probability);
      • Make use of more extensive data with data-driven testing and automated testing/checking, if needed.
    • Test Execution
      • Perform testing by descending order of risk level (i.e. execute Tests related to higher risks first): experienced testers, with a high degree of domain knowledge, may provide more valuable feedback;
      • Thoroughly test risky items, using scripted and exploratory approaches.
  • Risk Monitoring and Reviewing
    • Look at items where you assessed the risk and evaluate if you need to take additional measures/treatments ("Are those items having failed tests? Were bugs found? Are those bugs relevant? Did we find any new risk while testing?").

Risk-Based Testing is defined by you

As you can see, Risk-Based Testing can be as complex or as simple as you want it to be. RBT is adaptable to different workflows and scenarios including Waterfall, Agile or Hybrid. Whichever route you decide to take, Risk-Based Testing will give you the framework, common language and indicators to make wise testing decisions and optimize your efforts and costs.