What software testing is and what it can achieve
31.10.2019
This is a continuation of my previous blog post: “Serious mistakes that could have been avoided: Lessons we can learn through software testing“.
Introduction
In my last post I wrote about one of the biggest issues relating to software testing in general. In my view it is the completely unrealistic set of expectations on what software testing is capable of or even what it is about to begin with. These, coupled with various human factors (more on these in the next blog post in the series) can have a serious impact on how, where, when and why testing is done – or left undone. These factors can also seriously affect what the results reveal or, sometimes, hide and, ultimately, what the quality of both the product and the project itself will be like.
This is a two-part blog post and in this first part I will cover some of the things that software testing is, in my view, whereas in the second part I will cover some of the things that it is not. Again: in my view.
However, before I get into it, I feel compelled to inform the reader that this blog post comes with a disclaimer:
There are many different schools of thought when it comes to software testing (four, at least, at the time of writing) but, in my view, there is only one to be taken seriously – the context-driven school of testing. Everything in this post is heavily inspired by that particular school because, to me, it seems like the most responsible one out there, and that will inevitably show in everything that I write.
Should you be interested to learn more about the context-driven school of testing, I will list a few eminent context-driven testers, along with links to their blogs, at the end of the second part of this two-part post.
Although I feel it should be self-evident I wish to emphasize that everything I write also comes with the disclaimer: “in my view”, even when I am not specifically stating so. I am not trying to force any “truths” on others – I am simply sharing my experiences and what I have come to know, believe and understand in my 15 years as a full-time software tester.
What testing is and what it can achieve
Exploration
First, and foremost, software testing is exploration. Every piece of software being developed nowadays, no matter how simple, is a complex myriad of interdependencies, differing states and state changes, interactions within and without the confines of the underlying operating system, and with other software running on the machine at the same time. Now, add any network-related functionalities, requirements, dependencies etc on top of that and I am sure you can see why it can be extremely challenging to keep track of everything that is going on.
For example, a developer’s primary job is to write code and this usually happens in a time-constrained environment. Because of this a developer will likely not be able to take all of the above points into consideration on a regular basis and still remain sufficiently productive in the limited time available.
This is why it is usually a good idea to bring in people whose sole focus will be on exploring as many of these factors, interactions and their combinations as possible, in addition to finding out what undesirable or unexpected side effects they might cause. In brief: testers help reduce the amount of unknown factors both inside and outside of the application context.
The above-mentioned complexity is also one of the main reasons why complete, exhaustive testing of any real-world software application is practically impossible or, at the very least, highly unfeasible. It would simply take too long and cost too much to “test everything” (which is never actually Everything™ – the possible variations are infinite, or close enough as to not matter). In addition to the complexity it is also because of diminishing returns – the benefits gained from testing the same things repeatedly decrease over time, relative to the cost and effort induced.
It is likely more useful and more valuable to prioritize and perform targeted, time-constrained testing sessions on a regular, continuous basis. These can and, in my view, should be done on one specific area at a time, approaching the testing with varying goals and angles of approach (functionality, data security, usability etc) in mind, before moving on to another area. This can effectively help reduce the risk of diminishing returns setting in.
Besides, simply repeating the exact same paths over and over again, such as when executing static “test cases”, you will lose the opportunity of finding out new information, which will necessarily make testing less valuable to the project than when a tester is allowed to explore more freely.
If a check1 is worth executing repeatedly, it is likely better to leave the task to a computer because they excel at quickly running through checklists. Humans, on the other hand, are far superior in everything that is exploratory (well, not really superior, actually, since there is simply no competition to begin with). Because of this, it would, in my view, be plain irresponsible waste of a valuable asset not to allow a tester to take full advantage of this human strength.
Critical assessment
Whereas the job of a developer is to write code and, often enough, also help design the product and its features so that it meets certain requirements, an important part of a tester’s job is to support the team by identifying omissions, conflicts, misunderstandings etc. Some of these issues may be in the code, some in the design and some may be caused by human factors such as people making automatic assumptions (more about this in a later post).
Testing is very much about critical thinking and asking questions about how the software might misbehave, or not work at all. What are the threats and risks both the product, and the project, may be facing because of a specific decision on design or implementation? What was left out entirely? What was not thought through carefully enough? How might the software not work the way we want it to? How could it be misused, abused, or exploited? What do we actually not know but simply assume? Do we have a problem? These are all very important questions that need to be asked on a regular basis in order to improve a project’s chances of success.
In addition to critical thinking, vivid imagination, a little bit of daring and arrogance towards the software and, at least in my books, also a crooked sense of humor can all be highly useful tools in a tester’s tool box. They can help explore the unknowns both inside and outside of the application, as well as help find new angles of approach, and completely new kinds of issues.
I just love it when I discover a serious new issue simply by trying out something a little esoteric or unorthodox while testing. You know, something like discovering a denial of service attack that crashes the entire server, simply by using an 8 megabytes long username when creating a new account in an online service, because the username field’s length was not limited (one of the cases I covered in my previous post).
Learning and sharing information
This has already been partially covered in the previous two chapters but let me reiterate some of those points: Testing is about finding out information. It is about discovering the actual current state of the software as opposed to what a (written) specification says it should be like. These two are very rarely the same thing – especially so because written requirements are practically never the same thing as The Requirements™. There will invariably be implicit requirements, tacit knowledge and automatic assumptions. There will be communication errors, misunderstandings, accidental references to outdated versions of documents or software components, regression issues caused by e.g. someone forgetting to merge a specific branch to master in GitHub and so on. The list is endless.
All of these factors can add to uncertainty about the current state of the product. That is why testers go through all of the possible data at their disposal, including identifying any missing data because that can be a meaningful test result as well. On top of that, testers also exercise the software itself, to learn more about it, so they can communicate what they have learned to others. This sharing of information is just as important as actually obtaining it – management needs to be able to make informed decisions and they can not do that if they do not have all the relevant information available.
An important point to make here: testing is in no way limited to just exercising the software implementation itself – one can test ideas, designs, expectations, specifications, reality even e.g. in the form of going through debug logs, analytics data, user feedback etc. All of these can result in gaining new, valuable knowledge and insights about the product – what it is or what it should be like, and what it is not or what it should not be like.
Information that competent testers are very well-equipped to provide through learning.
Summary
In this first part of the two-part blog post, I covered some of the things that software testing is, in addition to just finding bugs:
- Exploration,
- Critical assessment, and
- Both learning and sharing information.
All of these need to be performed on a continuous basis, to maximize a project’s chances of creating a high-quality product.
In the second part, I will cover a couple of slightly more controversial topics under the title: “what testing is not and what it can not achieve”.
References:
1: https://www.developsense.com/blog/2009/08/testing-vs-checking/
About the writer
Petteri Lyytinen
Software Detective
Petteri is a software testing enthusiast who has been doing testing professionally since 2004. He is an active member in the global testing community and has authored various publications ranging from blog posts to a joint book project with other testing professionals from around the world. Petteri was “Finnish Tester of the Year” nominee in 2011. He currently lives in Estonia but quite often works in the Helsinki metropolitan area.
