A balanced combination of manual and automated testing is the way to go if you want to achieve higher test coverage and efficiency. They cover for each other.
Manual testing is easy to learn and do, but it is highly likely to create inconsistent test results, and sometimes it takes a lot of time and effort. On the other hand, automation testing requires a lot of technical expertise to create and maintain scripts, but it checks bugs much faster with higher consistency.
According to the State of Quality Report 2024, automation for web testing, API services, and mobile applications has only increased over the years. This is not a trend to be missed.
Read on and learn how you can beautifully balance automation testing with manual testing with the right skills, tools, and practices.
The 3 Pillars Of Testing
Testing is complex. But at its core, we can break it down into 3 major pillars:
- Automation testing
- Manual testing
- Test management
Automation testing is the process of automating the execution of test steps, either by writing test scripts or leveraging automation testing tools.
Automation testing is the norm these days. Software can do a lot of things, including testing other software. If done correctly, automation testing saves time, speeds up development, increases test coverage significantly, and makes testing more effective overall.
Manual testing is the opposite. When doing manual testing, testers perform the test steps manually, without any assistance from tools or scripts. After completing the steps and seeing the results, they have to manually document it in a test management system.
Manual testing is time-consuming, but there are instances where this type of testing really shines through. Exploratory testing, for example, allows testers to freely explore the system to check for the more obscure bugs, which are so rare and unexpected that it is impossible to automate them.
And in the middle, we have test management.
Test management is a broad term. We can think of test management as “managing everything test-related”, and that could include:
- Test case management
- Test data management
- Test artifact management
- Test environment management
Each of these practices needs an article of their own. Think of automation testing and manual testing as two opposites on a spectrum. You need test management to connect your testing activities together.
From Manual Testing
If all of your testing activities are still done manually, it is best to include some automation in the mix. Of course, the more difficult cases should be reserved for manual testing. Since it is ideal to manually test the “difficult-to-automate” test cases, you will need some tips in your pocket.
Learn More: How Exploratory Testing Fits Into The Software Testing Life Cycle
Tip 1: To truly test, you must do exploratory testing.
Have a look at this Venn diagram. There are 4 scenarios:
- Assumptions: we think that there might be a bug there, either by experience or intuition. We conduct an exploratory session to check this "Unknown Known" zone.
- Requirements: we know that there is likely a bug there since it is a core/new/critical feature. For this zone, we do manual or automated testing depending on the nature of the feature.
- Questions: we have no idea if there are bugs there or not, and we can also explore this area to find out.
- Absolute Truth: we don’t even know that it exists in the first place (for example, obscure bugs that were unintentionally created due to interactions between modules). Only exploratory testing can reach this area.
The scope automation testing can reach (Requirements + Assumption) is much smaller than the scope of exploratory testing (Questions + Absolute Truth). It is a blind spot of any system. You must venture out and explore those areas, so that the tests you conduct won’t remain superficial.
So, how do we do exploratory testing properly?
Tip 2: Do exploratory testing with a purpose
In other words, wander with purpose. You have a mission: uncover obscure bugs manually, but that route is unscripted. Approach it with a learning and experimenting spirit, but accompany it with heuristics.
Learn More: How To Plan and Prepare For Exploratory Testing
Heuristics are mental shortcuts for quickly solving problems. In exploratory testing, ask yourselves:
- What are the boundaries of this system? What happens when we break those boundaries? For example: what happens when you input a username that is 10,000 characters long?
- What are the error messages? A system can work in one way but go wrong in hundreds of ways. Focusing on those scenarios can produce a lot of unexpected bugs.
- What are the confusions you encountered? If a certain area of the product appears “confusing”, is there something wrong with it?
To ask those questions means establishing a route to follow. Think of yourself as a rebel who goes against all the rules the developers put out, but the way you rebel must also conform to a certain rule you set for yourself.
Tip 3: Involve teammates
Manual testing, and especially exploratory testing, can be a mentally-stimulating (and exhausting) process. Having many fresh pairs of eyes to look at a product/feature brings more perspectives to the exploratory session. Not just testers, but even developers can also chime in to share their knowledge. Perform exploratory testing as a team.
Recording and sharing your testing sessions with your team members is a good idea. Leverage all forms of media: video, audio, screenshots, and even technical notes. Having test evidence provides ample information to the dev team.
To Automation Testing
Of course, we also need to incorporate automation testing into the test strategy. Here are some tips for you:
Tip 1: Use automation testing to expand on manual testing
Automation testing is the tool to scale. Let’s imagine a test scenario where you are testing an E-commerce website. The number of webpages to include in the test plan can reach the hundreds and thousands. Manually testing them is obviously counter-productive. The moment your product scales, it is time to think about automation testing.
For example, you can leverage the Record-and-Playback of automation testing tools to record your manual testing actions on screen. That sequence of actions can be turned into an automated script that is reusable across environments.
Balancing automation testing with manual testing is crucial. Automation testing is fast and productive, but at the end of the day, a test script can only do what you explicitly tell it to do. Even if there are problems triggered during test execution, the script still won’t notice it, since it was not programmed to do so.
On the other hand, as a human tester, you would have noticed something strange from the problems you encountered while testing something else. The human mind is ready to capture bugs all the time. This is why you should reserve automation testing for highly repetitive test cases; after that, use exploratory testing to compensate.
Tip 2: Embrace data-driven testing and keyword-driven testing
An issue that comes with scaling tests is that you have hundreds of different inputs/input combinations for one single test scenario. Data-driven test automation is the perfect way to go in such situations.
To do it, first arrange your test inputs and expected outputs in tables, with each row representing a test case. Next, write an automated test process that reads each row, executes the test steps, and verifies the expected results. Spreadsheets are great for storing this information because they simplify data entry and can be easily accessed by most test tools and programming environments, either in their native format or as exported CSV files.
With a data-driven test process in place, you can reuse it for different tests. This method works particularly well for workflows with many data variations. For more complex tests involving different sequences or paths, consider using keyword-driven automation.
In keyword-driven testing, a keyword is essentially a code snippet to automate a very specific action in the system (such as Click). By combining a wide range of keywords for different actions, you get a test case.
Test Case Management Across The Entire Process
Test case management connects these automation testing with manual testing. You need a system to link manual tests with automated tests. The Xray-Katalon integration allows you to:
- Have Xray as the central management tool for their manual tests
- Have Katalon to author, manage, and run their automation tests
- Connect your automation tests to manual tests and get the best of both worlds
- Connect a Jira project and Xray project to Katalon platform
- Push test results and execution artifacts to Jira as well as store them in Katalon TestOps
After creating tests in Katalon, users can easily push those execution results to Xray automatically or manually. Additional details like pass/fail status, fix version, evidence (screenshots), test case table, and test environments are synchronized. This centralization enhances collaboration and visibility.
→ See how you can do it with ease
Improve your Testing Workflow with Katalon and Xray
In conclusion, combining manual and automated testing enhances test coverage and efficiency. Manual testing is simple but can be inconsistent and slow. Automation testing is faster and more consistent but requires technical expertise.
A best practice is Integrating both with proper test management to ensure comprehensive testing. The Xray-Katalon integration centralizes test management, enhancing collaboration and visibility for better quality releases.
