Serious mistakes that could have been avoided: lessons we can learn through software testing
16.10.2019
Introduction
Software testing — a subject a lot of people have opinions on, and many of them unfounded, misguided, or just plain wrong. Every professional software tester has probably, at some point or another, heard the phrase “anyone can test”. Well, newsflash: it’s true — but only if you don’t care about the end results. In exactly the same way, I can cook a mean lasagne but that doesn’t mean I would make a serious candidate for master chef in Atelier (a 3-Michelin star restaurant in Germany). Similarly, I am quite the shower tenor but I don’t think anyone would want me on the stage of Teatro dell’Opera di Roma as lead singer for Aida. I am simply not good enough for those positions even if I can do both, cook and sing.
What I am trying to say here is that mastering software testing, like any other serious craft in the world, is hard. It takes time, it takes effort, and it takes lots and lots of practice through trial and error. Anyone CAN do it, however, if they really want to. Just don’t expect brilliant results from day one because you think you know what testing is all about. Below, I am giving you a couple of real-life examples from my own personal working life (before VALA) where people were confident that everything was just fine but, surprise, it turned out that this was not the case at all in any of them.
Case 1: The hotel reservation system that allowed for you to get paid for staying in the hotel
Sounds nice, doesn’t it! In this particular case the system was built so that the entire reservation object, with all of the reservation data, was loaded and handled locally in the browser. I can already hear experienced developers groaning in pain at that but it gets worse! The key thing that we discovered was that the system’s session handling was seriously broken (read: non-existent): if you loaded the reservation object in two (or more) different browsers at the same time, you could actually make the hotel pay you for staying there.
Here’s how: First, you made a normal reservation for e.g. 7 nights. Then, you started modifying that reservation in one browser instance (let’s call this browser 1). Then you did the same in another browser instance (browser 2). Shortening your stay from 7 nights to e.g. 1 night in browser 1, and then saving the change, immediately refunded you for 6 nights. So, at this point, you had only paid for 1 night. Then, in browser 2, where the unmodified reservation object was still stored as it originally was, you made the same change and voilà: the system just overwrote the previous data without question and immediately triggered another refund for 6 nights.
In short: you paid for 7 nights, but got refunded for 12 nights, while still getting to stay in the hotel for 1 night. That’s the kind of hotel I’d like to visit!
Case 2: The login system that allowed you to become admin with your own password
Quite often, a login page is the first and, in some cases the only, barrier keeping unwanted visitors outside of a system that might contain either sensitive or business critical data. While it can be debated in which cases this is or isn’t enough, I don’t think it’s unreasonable to say that making the login system a hacker’s best friend is probably not a very good idea in ANY case.
This, however, was exactly the situation in one system that I tested years back. Quite simply put: when a user logged in, the system checked whether or not the entered username existed and the account was active. Then the system checked whether or not the entered password was found in the database. What it did NOT check, however, was whether or not the entered password belonged to the account that was used to login.
Still, to this day, I do not understand how anyone could make such an amateur-level mistake in a serious software development project and be able to carry the code all the way to production but there it was. I stared blankly at the screen, at a loss for words, when I realized that if I created a user account for myself I could then login as administrator and have full, unrestricted, access in the system simply by entering “admin” as the username and then using my own account’s password — the one I had just chosen a minute earlier. An utterly baffling mistake to make in a commercial software project if you ask me.
Case 3: The denial-of-service user registration form
Yep, I am a huge fan of HTML forms because they can almost always be used for all kinds of unexpected and undesired malice (in a controlled environment, of course, since I am always there helping customers strengthen their systems by finding weaknesses before The Bad Guys™ do).
In this particular case, which was for a large media company’s content management system, I just half-accidentally noticed that the maximum length of the username field on a new user registration form was not limited. A very bad idea, if you have me as your tester. So, what else would a tester with a vivid imagination and more-than-a-little crooked sense of humour do but choose to use an 8-megabyte username for himself? I mean, duh, c’mon.
What was not expected, however, was the full scale and severity this particular, innocent, little test would have on the system: it froze, and I’m not talking about a piece of code regurgitating a Java stack trace on a user’s screen. No, I mean the entire system froze. Completely. They had to physically reboot the server in order for it to recover, only to have it freeze up again the moment any part of the system tried to retrieve user data from the database. My user account had to be removed from the database manually, through a command-line interface, before the system could be used again.
Quite a trivial, and effective, denial-of-service attack considering it was made possible simply by leaving out the maximum length limitation on a single HTML form field (and, obviously, a similar check in the backend in the form data handling).
Final words
To summarize the above and, to complement on the introduction, one of the reasons why mastering software testing requires time, dedication and effort, just like any other serious craft in the world, is the complexity of the systems being developed. No real-life software project can ever be fully tested because the possible variations are infinite. Complacency, cognitive laziness, false confidence and, especially, automatic assumptions are also reasons why issues like the ones described above can be found in software systems on a more or less regular basis. I might write more on those subjects later on because they would definitely warrant being explained in more detail.
There is always an element of uncertainty present in any system because it is impossible to take everything into account. My friend and colleague Michael Bolton (no, not the singer one) likes to say: “We testers are in the (false) confidence demolition business”. Hear hear, says I. Even though people don’t like to hear it, the cold fact is that we can never know for sure when everything is fine — like the people in the projects above thought was the case.
That’s why you need experienced, dedicated, professional testers: to go on a hunt for the unknown, and the unexpected — not just to tell you that the system CAN work through the so-called happy path scenarios where all inputs and user behaviors are exactly as expected.
Now that is another topic that would warrant a blog post of its own. To be continued…
About the writer
Petteri Lyytinen
Software Detective
Petteri is a software testing enthusiast who has been doing testing professionally since 2004. He is an active member in the global testing community and has authored various publications ranging from blog posts to a joint book project with other testing professionals from around the world. Petteri was “Finnish Tester of the Year” nominee in 2011. He currently lives in Estonia but quite often works in the Helsinki metropolitan area.
