Don’t compare automated testing with human testing
03.02.2020
Many times test automation is described as a time saver, more precisely, a saver of time spent on manual testing. Which is all true and good. Now our marketing team wanted to find out if we can use the knowledge of our people and make some educated guesses of how much it actually saves time.
Well, we couldn’t.
Man versus a machine, not actually even a battle in software testing
See, the case is much more complicated (which we should have known of course). We thought we could just make a survey and let all our test automation people answer how much time is test automation saving in their own projects. But no no. Our people weren’t fooled by this uneducated question. Most of the respondents in our survey just didn’t take it and refused to write down a number of the saved time.
If you’re not a pro yourself, it’s totally justifiable for you to ask why is that so? Isn’t test automation’s main purpose to automate stuff, i.e. save time? Well, yes and no.
Test automation definitely saves time. But as our experts pointed out with a crystal clear red pencil, the main purpose of test automation is not just to save time but to do a whole lot more. Here’s a reply from one of our experts which summarises the point nicely: “Human work and automated checks can not be compared 1 to 1. If a manual tester would run the same tests as the automation, he would also be able to check other things from the application at the same time.”
Naturally automation is time-saving by nature. If you wouldn’t automate a certain task, that task would be done manually (which is slower) or it wouldn’t be done at all (which is worse in general).
The distinction between man-made-testing versus machine-made-testing is perhaps the key here. They are not the same. A machine can never (at least as of yet) function as well as a human in certain tasks, like exploratively testing a part of a software’s functionality. In other words, test automation can and probably will increase efficiency of testing but it’s work is not comparable with human testing.
Conclusion
In terms of efficiency and quality, in an ideal world, there would be no such testing that a machine could not do. If a machine could test everything, it would be more efficient and more reliable than a human. But it can’t. And even if it could, a machine will never be able to explore and work creatively in a way that a human can.
We should stop comparing automated and manual testing. Automated testing is a very important part of software quality assurance and so is exploratory testing. Both of these should be used and they shouldn’t compete with each other. There is no answer to be found to the question, “how much time test automation saves”. There is no answer because test automation is an apple and a human testing the same thing is an orange.
