Our test automation and test management communities needed some intel about what tools VALA people have experience of and what kind of positive and negative user experiences and observations they relate to those.  For these purposes, we made a survey for our test automation and test management people at the end of 2020.

Since the survey respondents represented a nice mix of testing professionals and the number of responses (N 26) was somewhat satisfactory, we thought it would be nice to share the results here too. The results portray individual people’s opinions, not VALA Group’s. 

Important note: Ranking test management or test automation tools is not that easy. It is influenced by many different factors, so it’s a good idea to think about your own needs before choosing the right and suitable tool. Influencing factors can be for example: number of users, different development team needs, size of the projects, number of different vendors, other tools used e.g for development management, licensing costs, manual testing or automation testing or both and the list goes on…

Maybe that’s enough of disclaimers and we can move on.

Experience of VALA people and the tools they’ve used

Let’s get to it. First a few words about the respondents. The median of the experience years in software testing was 5-10 so pretty experienced people to say the least.  

The respondent group has experience of various test management and test automation tools. Here are the top five’s of both. 

Test Management

1. Jira without additional plugins 16
2. Microsoft Excel 😉 15
3. Jenkins 13
4. TestRail 11
5. HP ALM Quality Center 11

Test automation

1. Robot Framework 22
2. Selenium 18 
3. SoapUI 15
4. Postman 12
5. Appium 5

Recommended test management tools

Our people were asked, “What testing tool would you recommend for managing and reporting testing?”. Here are the answers. 

Jira was the clear winner with five people mentioning it as a preferred tool by itself and six more with a plugin of choice. Also the second most preferred tool was pretty clear as TestRail clinched three votes (plus one as a combination with Jira).  After these two we had two mentions for Robot Framework (yes, it can be used for test management purposes too) and all the rest were mentioned only once. 

Many of the respondents raised their concern relating to the term ”recommend”, as they (rightfully) thought that one can not really recommend a tool without knowing the situation at hand. So, once again, remember that these are not VALA’s recommendations but more like individual people’s number one choices if they could pick only one. 

Pros and Cons of the selected test management tools

Next we asked: What are the best and worst aspects of the test management tool that you recommended? Here are all the replies. 

Jira Pros

• Single point of truth for all development activities. Can be integrated to many other systems.
• Flexibility, and everyone is using the same tool – single point of truth is visible to all. I especially like the dashboards, but it is a very unused feature.
• Easy to use
• Integration with requirements, development and testing . For example documentation on con,uence for the requirements which can be linked to Jira development tasks and testing can be done using Jira plugin to see current testing status for the whole project. Defect would be reported to Jira
• Flexible, easy to configure, cheap
• Configurable
• Easy to follow tickets in sprint board and easy to plan tasks for sprint
• Maintainability of JIRA can get a bit complex.

Jira Cons

• Flexibility. 😉 If you are not disciplined in maintaining the structure, the structure will be corrupted very soon and become a ticket graveyard.
• Updating tool causes problems
• No integrations between different systems and you would need to do double work to maintain different thinks like tests or defects
• Somewhat limited, I don’t have experience of traceability so can’t comment on that
• Tickets can get lost in backlog if there are too many of them

Test Rail Pros

• Designed for testing, integrates with JIRA etc.
• Quick to add test cases, clear structure and everything in one place (sees the whole picture and can follow the testing situation easily), can add images to test cases, can make graphs, can connect to Jira and e.g. Robot Framework test results can also be added to TestRail (manager sees everything from one tool)
• Easy to use and modify. Flexible. Can be used together with Jira if wanted ( links defects for example)
• Easy to use, versatile

Test Rail Cons

• I do not know about the licences and prices
• Requires a licence
• hard to say. Enough for our needs
• It’s a test management tool

Meliora Test Lab Pros and Cons

• Visual, modern tool. Support for test automation is there as well as interfaces towards JIRA
• Not so mature yet so small issues popping up. Nothing major though

GitLab Pros and Cons

• Gitlab is the closest to the developers.
• Gitlab requires pre-customization, creation of templates.

HP ALM Quality Center Pros and Cons

• Comprehensive, includes test management and bug management, easy to use
• Expensive

Jenkins Pros and Cons

• Easy to install and use also at home lab.
• To find the good plugins. Too many plugins.

Worth noting here is that these are the preferred tools of VALA people so naturally this might affect the number of cons listed in the answers. 

Recommended test automation tools

We also asked about the preferred test automation tools. Here are the answers. 

A landslide win for Robot Framework! 

What tools to use in what kind of testing?

One major reason for the success of Robot Framework is without a doubt it’s versatility. In the survey people were asked “What would you use your preferred TA tool in?”. Here are the results which clearly display how versatile Robot Framework is. 

Mobile app

• Robot Framework
• XCTest/Espresso

API

• Robot Framework
• SoapUI
• Postman
• UnitTests++
• Googletest

Web UI

• Robot Framework
• Selenium

Client UI (for example Windows app)

• Robot Framework
• TestComplete

OS level

• Robot Framework

Embedded Linux SW

• UnitTests++ 
• Googletest

So, how to sum up this all. Well, the more we dug into this, the more we realised how dependent on their personal history the choices of an individual are. Even with many years of experience, people might have used only a few different tools. Moreover the most used tool seemed to be the number one choice for many. 

For us this is merely a starting point from where we go deeper. Stay tuned for more detailed analysis!

• Blog
|
• Software testing
|
• Test automation