In pursuit of improved software quality and efficiency, many organizations turn to test automation. Let’s briefly explore what is the purpose of test automation in software development.

Test automation refers to the execution of test cases automatically using tools. Its key benefits include faster and continuous feedback on the functionality of software changes, reduced risks, and the ability to accelerate development and releases. This allows people to focus on exploratory testing and quality improvements rather than repetitive tasks.

The common industry promise, particularly from the test automation tool vendors, is that test automation offers faster feedback loops, lower costs, and accelerated development cycles. Ultimately, everyone wants high-quality software, delivered quickly and affordably.

Now take a moment to consider: what is the primary reason you want to implement test automation? Once you understand your ‘why’, the tool presented in this post can guide your decisions on when and what to automate.

This post summarizes key insights from Mika Mäntylä’s When and What to Automate in Software Testing (2016), a multi-vocal literature review that combines formal academic sources with gray literature, such as blogs and white papers, for a comprehensive perspective.

Following the study summary, you can find my insights as a VALA expert, drawing from 15 years of field experience.

Lastly, there’s the questionnaire tool, based on a checklist from the study, to help you assess whether you should invest in test automation. Indeed, there are situations where such an investment may not be worthwhile.

Key factors to consider

The study identifies five major categories of factors that should influence your decision to invest in test automation:

1. Software Under Test (SUT)-related factors: Evaluate the maturity and complexity of your software.
2. Test-related factors: Look at the type of tests needed, especially the need for regression testing.
3. Test-tool-related factors: Evaluate the available tools and their compatibility with your software.
4. Human and organizational factors: Assess the skills and readiness of your team to implement and maintain automated tests.
5. Cross-cutting and other factors: Consider economic factors and overall project goals.

Most frequently mentioned factors

The study highlights several key factors that are most often cited as critical in deciding to automate tests:

• Need for regression testing: The time and effort saved through automated regression tests enables you to explore new features and uncover previously unknown issues and assumptions about software quality.
• Economic factors: Consider the cost-benefit analysis of automation.
• Maturity of the SUT: Mature software may benefit more from automation than software in early development stages.

Investing time upfront to evaluate these factors will help you determine whether test automation is appropriate for your project. Sometimes the best decision is to invest in other areas of testing to achieve your quality goals.

Thoughts from the field – issues that most affect the outcome of a test automation investment

Based on my experience, I can outline four common issues that most affect the outcome of a test automation investment:

1. Testability and stability of the SUT and technical requirements

Like the study by M. Mäntylä shows, the maturity of the SUT is a key factor. If the SUT is not automation-friendly, prepare for higher initial investments in both time and budget. Some examples that hinder the development:

• SUT is either a commercial-off-the-shelf enterprise software or part of such a system where you can only automate parts of the whole SUT.
• Special effort is needed for creating a solid locator strategy, for example.
• There are restrictions in terms of test data and infrastructure, for example lacking a stable test environment.
• Production code / feature development is prioritized over developing features that effective test automation would benefit from.

Test automation demands supportive functions, such as enhanced testability, continuous integration, and effective test data management.

Lack in this area easily leads to flaky and untrustworthy tests and when that happens, people start to overlook the value of test automation. 

2. The goal to replace (some or all) human testing with automated testing 

A common starting point for automation is replicating test cases that humans currently execute. While this approach can help to think about what to actually test, this approach rarely translates well into a 1:1 replacement. Automated scripts don’t have to go through the same paths as a human does. While possible and it’s sometimes useful to have a handful of test automation scenarios like this, there are often more efficient ways to leverage test automation. Moreover, test automation only checks what we specify it to check, it doesn’t provide any new information. Humans are essential for designing test cases, interpreting results, and resolving issues.

3. Test automation is not considered as continuous development

In many cases, the initial approach involves having a consultant lay the groundwork for test automation, with the in-house QA team or developers taking over later. However, the skill level and effort required for effective test automation often surprises development teams. Training and patience are essential, especially if test automation has not yet become routine; it takes time to adopt new approaches. Developers are often focused on implementing new features, which raises a question of priorities: do you want more features quickly, or fewer features with automated tests to support them?

Test automation requires planning and maintenance and should be integrated into the entire development process. It’s not that effective when created in isolation or for one-off projects. 

4. Lack of diversity

It’s common to approach test automation from the user’s perspective. However, GUI test automation remains one of the most challenging levels to automate. For each test case, consider the idea “how low can you go?”, that is, thinking can we test the idea with an unit test or with an API/integration test? For maximum speed and efficiency all the levels from the classic test pyramid should be used.

Here’s the practical questionnaire tool to support your decision-making, based on the study and enhanced with 15 years of experience from the field. 

It’s a simple questionnaire where you answer either YES or NO and the tool calculates whether it’s in favor or against your decision. If any question is irrelevant to you, just leave it blank.