Choosing the best option to oneself rarely is an easy one. The problem is similar when you are buying jeans for yourself and when you are choosing a software for your company – the salesman tells the brand they represent is the best option and probably recommends the more expensive options. The price often reflects the quality, but does not guarantee it. If you want the best bang for your buck, you need to know about the product you are purchasing. In the case of test management tool the choice can be made easier with some preparation. This article is about what you should remember in order to be able to choose the best tool for your purposes.
Different approaches for choosing the tool
One option you have is to outsource the problem. This can be a good approach especially if QA tools are far from your organization’s field of expertise. You will easily find QA companies or companies specialized in selling software to do the research for you. If you use this approach, remember to be a little bit skeptical about the advice you get. Not all research conducted this way is without ulterior motives. Companies recommending solutions to you might have a monetary interest to recommend a specific solution or simply they might have more experience with one tool over the others and thus they might recommend it even if it would not be the best one for you.
Another thing you should think early on is the approach of choosing the tool and taking it into use. Usually, you’ll the most benefit from your tool by using a single tool with long term benefits in mind, but consider piloting the test management tool in a real project before making the final decision. This way you will get real feedback from your own professionals and have a possibility to withdraw from using the tool if it doesn’t suit your needs.
The tool as a service or installed on own server?
Do you already know if you want the tool installed on your own premises or do you know you want it as a care-free service? Usually, the reason for installing the tool on your own premises is a concern about security. Another thing is that not all SaaS solutions offer you a possibility to integrate into your other systems. As a rule of thumb, if you do not have a reason to have the solution in your own premises, you will have it easier as a service. Most important benefits in SaaS installations are
- No need to install or maintain the software. You can use that time for productive work
- You always have the latest version.
- No need for long commitments. Buy what you need.
When you choose to sign up for a service make sure you have a way to export your data relevant to yourself in a format you can use later on. Optimally this means some easily usable text-based format such as CSV or XML. To be on the safe side, contact your vendor and ask if you have a possibility of getting all your data if you decide to give up using the service. Even after the project is over, you might need the data. For example, when developing a new version of the software you might want to revise previous requirements/user stories and not to start from scratch and maybe check the found defects not to make same mistakes again.
What features do you value?
Test management tool comparison sheet
When purchasing just about anything, including these jeans or test management tool, the main point is it should suit your needs. When deciding a tool a commonly used practice is to first list your requirements for the tool and then see how the tools fare against your requirements; The idea is that the tool should support your way of work.
A good practice is to look at the tool features with open eyes meaning that you should be open to any features which you originally weren’t looking for. Sometimes you might find features that really help you even that you didn’t list it beforehand or the tool might not have a certain feature, but another good way to accomplish the thing why you wanted the feature in the first place. Making a really strict rating system for the evaluation probably guides you towards features you are already familiar with without taking into account that the tools always develop new features over time. Then again, you should value the features you really benefit from and give less attention to features that seem nice but you are not sure if you really need them.
The above said I have included here an evaluation sheet to help you with your evaluation efforts. Usually evaluating many tools takes time and this sheet also helps you compare different features of the tools. When giving points to different features you can give higher points to solutions that are more usable. For example, if your organization needs to import data often you will value the ability to do it as easily as possible without needing to install additional plugins or software. In the end, you don’t need to choose the one that got the most points if your gut feeling tells otherwise.
You should also prepare for future needs. For example, if you have not used formal requirements in conjunction with testing, you might want to ensure that your tool supports requirement management. As features, it is also best to take into account the general use and maintenance of the tool. If the tool requires the installation of client software or plugins, do those support your possibly heterogeneous workstation environment? If installations are needed, how do you deliver them to users’ workstations?
One possibly important feature group that is closely tied to your way of working is the capability for integrations. Which other tools do you use in your development and which of those should be tied to testing? Generally, you benefit most from the integrations that support your daily work. For example, if your developers use continuous integration, you might want to have integration between the tools so that your testers will see what unit tests there are and what is their run status at any given moment. This feature helps people to work together and thus saves precious time.
The architecture of the tool does not affect you directly, but you should look at how often the tool is updated. The tools with steady update history are more probably to have updates also in the future.
What do the tools cost in the end?
One prime factor in our buying decision is, of course, the overall cost of the service. When we buy jeans, we see pretty accurately the price we are going to pay. With development-oriented software, the final price is a bit more complicated. In addition to the actual cost of the tool, we have to take into account the hosting costs and how much time people need to spend on the maintenance.
Different tools have different pricing structures. If we pay for named users we need a lot more of them than floating licenses depending on the way of working in your organization (typically, one in three of your named users should usually be enough floating licenses in comparison). If you buy software as a service, the yearly costs are simply twelve times the monthly fee. For on-premise installations, the best thing for you to do is to approximate the yearly costs including the hosting, maintenance and updates. Approximate how much time the tool maintenance requires and take it into account too. If you buy the software, take also the maintenance fees into account.
Think what is the value of minimum up-front investments for you. With monthly fee pricing, you can stop using or downgrade the licenses when you need to. Then again, by committing to a longer subscription you might get a discount.
What goes in comes out – and the format matters
All testing tools offer some kind of reporting system, but they are not equal. Make sure you get the information reported that you need. A good reporting system is easy to use and gives you the information you need in a format that is easy to understand. Some tools have a reporting system that is comprehensive but requires a lot of work to set up and results might be easily misinterpreted. Then again, some tools allow you to see the basic information, but they do not really tie it together to meaningful reports to help you make decisions based on them. Ok, you might know which cases failed and passed on the last run, but what is the big picture – for example how thoroughly have you tested certain areas of your software project? Or you might know the number of issues, but not the rate which in the defects are closed.
Avoid shotgun marriages with the tool
You should also think of some sort of exit-strategy. What happens when you want to change the tool or keep a pause in its usage? Will you get your data out in a format that can be used later? If not, it is harder for you to change the tool if you would grow unsatisfied with the service you get. The problem is almost the same in SaaS and installed services. If you don’t get the data in a usable format, which today means a .csv -file, you can not import it straight to another tool.
What do you need from your vendor and can the vendor deliver you that?
A working tool as itself is never enough – support and service level your vendor can deliver is as important. If you need support, will you get help from the vendor? And if you do, what is the procedure? With some vendors getting real help for technical problems is notoriously hard as you need to fight through different tiers of customer support before getting to someone who can really help. And that takes time. Then again, if you go for a hosted service, you will need technical support less as the tool is maintained by the vendor. Some companies also offer a more consultative touch to their support: How should you use your requirements effectively to support testing? What sort of workflow should you use in a certain project? This can be a lifesaver for some companies where all the involved people are not experts as you get real-world expert answers when you need them with reasonable costs.
If you need training, consultation, and support, you should give value to what the vendor is capable of offering. Can the vendor or someone else you trust provide you that? Do you have a need to customize the tool now or possibly later? For example, if you would like to integrate your tool to your own third party product – can the vendor deliver that, and if yes, for what kind of price? Best vendors can support you in all your needs, and still do not cost you your arm and leg.
Try before you buy
In evaluation, it is essential that you have the possibility to try out the tool in real-world scenario before making the decision. That is the only way to be sure that the tool works as you have expected it to work. That can be sometimes a bit tricky as some features will reveal how they really work only after using the tool in real projects. Many tools offer you a trial period to test the software which is good, but testing the software in real-world scenario often requires more time. If you can, consider piloting the chosen tool with a project or two before committing yourself to big investments.