In Part 1 of this series about Test Case Design, you learned that by using the Test Case Designer tool in Xray Enterprise, you take a model-first approach to designing more thorough and efficient tests. But how does the tool generate optimized scenarios once we define those models? And what do we mean by “optimized”?
Introducing pairwise/combinatorial testing and test optimization
Let’s first take a step back. In any system under test, especially in complex ones, you have most likely encountered what we call “combinatorial explosions.” A combinatorial explosion occurs when there are thousands, millions, billions, trillions (maybe even quadrillions?) of possible tests you could execute.
Technically, when someone asks you to “test everything,” this is almost always impossible - and it’s certainly not worth anyone’s time. So, you’re not going to try to test everything, and you’re also going to avoid the previous, traditional methods to test case creation.
Instead, with test case design, you will meet in the middle on the above spectrum to ensure you maximize critical coverage in as few tests as possible.
At any point, while designing your parameter models, you can click the “Scenarios” button inside the tool to generate optimized tests within seconds (you’re already saving time!). When generating scenarios, the algorithm takes a pairwise analysis approach.
Through academic research and practical application at thousands of firms worldwide, it has been statistically proven that most defects can be found by at most two factors (parameter values) being tested against one another. In other words, most defects are caused by testing every single factor and, furthermore, testing every single factor with every other factor at least once - hence the terms “covering all pairs” or “pairwise” testing.
So, suppose that is how the majority of defects are uncovered. In that case, the Test Case Designer algorithm says, “Well, at a very minimum, then let’s start by covering 100% of those pairs between the defined parameter values.” Not only will you cover 100% of those pairwise (or “2-way”) interactions, but the algorithm will do so in as mathematically few tests as possible. It is impossible for any human to do this, at least at the speed at which an algorithm can.
With the Test Case Designer, now you will objectively create better tests than traditional test case creation methods - you will be covering more and doing so in fewer tests to execute and maintain inside Xray over time.
The tests that will be generated are as systematically varied as possible - packing in as many new critical interactions as possible per test and eliminating any gaps and/or wasteful repetition that can be found in manually-selected tests.
You even have the option to add Constraints inside the Test Case Designer to mark any interactions that are impossible to test for and define system logic to ensure that all of your pairwise/combinatorial scenarios are valid when generated. You also have the option to increase or decrease the level of coverage that is being achieved in your optimized scenario set.
Selecting “3-way interactions” will increase the thoroughness of your set - instead of covering all pairs, your scenarios will now cover 100% of the possible combinations of 3 parameter values. You increase your tests' thoroughness by getting more specific with your combinations.
Increasing the thoroughness will almost always increase your test case count. Remember, based on Academic Research, the majority of defects are found by testing every 1-way or 2-way interaction, and 3-way / 4-way / etc. defects are significantly more rare, so be careful about increasing the thoroughness and have a reason to do so - you don’t want to increase your test case count for no reason.
You also have the option to account for your precise, risk-based efforts. The “mixed-strength interactions” option allows you to adjust the individual level of thoroughness for individual parameters.
Maximize coverage with fewer tests
With test optimization inside the Test Case Designer, you will save significant time and effort - it is faster to create tests, easier to maintain tests, and easier to address defects earlier in the process due to the increased thoroughness.
Let’s see how you can optimize your test execution efforts.
As seen in the below Test Case Designer Coverage Graph, no matter which “coverage strength” you ultimately choose, the algorithm “front-loads” the optimized tests with coverage. Your first tests to execute achieve significantly more new, critical coverage than your final tests.
At the end of your optimized test set, there are not as many new interactions to still cover since so many were covered upfront in your early tests. Using TCD’s detailed coverage reports, you can make smart, data-driven decisions about how much testing is enough.
If you are under a tight deadline, using the Coverage Graph or Coverage Matrix, you may find that executing the final tests to reach 100% coverage may not be worth your time, due to the front-loading of coverage that already happened in your earlier tests. TCD provides you with flexible options and tools to ensure you make any decision with 100% confidence.
Above, you see the Coverage Graph, one of the detailed coverage reports provided inside TCD. Here, you can view the exact % coverage achieved after running any number of your optimized test cases (e.g., “After executing the first 37 tests, we are achieving 80.2% coverage.). Using these tools; we can make significantly smarter decisions about our execution efforts, answering questions such as, “how much testing is enough for this project?”
In addition, the slope of this line is significant and depicts the “optimization” of your test cases that are occurring. Regardless of which “coverage strength” you choose, TCD’s algorithm will “front-load” the optimized tests with coverage. Notice how the slope starts off very steep because the algorithm packs in as much net new coverage as possible per test. So, your first tests to execute achieve significantly more new, critical coverage than your final tests.
At the end of your optimized test set, there are not as many new interactions to still cover since so many were covered upfront in your early tests. So, you may be under a tight deadline and, by looking at the coverage reports, have decided that it would be optimal to stop testing after you reach 80% coverage - it is not worth your time to execute the final tests to reach 100% coverage. There is a decreasing marginal return of executing those additional tests.
Introducing Gherkin, BDD-style scripting inside Test Case Designer
Within Test Case Designer’s scripting capabilities, the Automate screen allows you to write template Gherkin. These Behavior-Driven Development style scripts will be automatically applied to every applicable optimized scenario upon Synchronizing/Exporting. Doing so allows you to transition your optimized tests outside of the Test Case Designer for Xray Enterprise test management and execution purposes.
In general, model-based testing allows for easier maintenance and reusability, while writing Give-When-Then scripts inside the tool helps less technical users contribute more to creating automated tests.
A few output methods are used to transition your optimized scripts outside TCD. Synchronizing syncs your optimized Gherkin (or manual) scripts with their related issues inside Xray Enterprise. Exporting generates Feature Files, Excel files, CSV documents, and skeletal code for automation in various languages/frameworks available - Java, Javascript, C#, Python, Ruby, and more. The main difference between these two methods is that Synchronizing places scripts inside Xray Enterprise. In contrast, any Exporting options are used for reporting/execution purposes outside of Xray Enterprise.
Optimized tests and complete coverage with Test Case Design
You can now optimize your test planning and scripting efforts with Test Case Design. Test Case Designer generates tests that automatically maximize coverage in as few tests as possible, saving you time and effort in writing and executing tests.
Going further, through the Synchronization capabilities, you will seamlessly align those optimized tests with your related issues inside your test management tool, Xray Enterprise. Test Case Designer is built to align with Xray Enterprise’s mission - unlocking quality at scale while testing less, covering more, and releasing faster.
