Test management is not as simple as it first seems. Different versions of test cases, different versions of the product under test and even different configurations to test against. This matrix becomes ever more complicated to manage as your test projects grow. Ensuring you have a process and the tools to manage this is key to your success. Below we list the 7 key complexities that we believe you should consider when implementing a process or selecting a test management tool.
At some point in the process we need to assign a test to a tester. The question is are you looking to assign it at the step, case or set level? Do you need to allocate different tests within a set to different QA engineers? How do you track the difference between assignment for writing and executing the test case? All of these points complicate a seemingly simple concept. When you look at the different ways different tools support this capability you’ll find it’s not quite as straightforward as you might first think.
We version control the code we release. In many cases we need to be controlling versions of the test cases we write and execute too. When working with version control you need to consider if the replication of different versions impacts already executed tests. You need to understand if you need to be able to choose the different versions you use at execution time. Finally can you compare the different versions easily?
Writing similar test cases repetitively becomes tedious very quickly. It’s open to mistakes and is also a poor use of the QA engineers time. To help address this some tools provide the ability to parameterise test cases. You write one test case then scale up with data that creates many permutations. It’s a nice concept but one that’s difficult to execute effectively. This difficulty is sometimes compounded by the way in which the tools implement this feature.
The concept of having a library of reusable test cases is as old as the discipline of software testing itself. How your test management tool allows you to manage this library makes a big difference in your capability to implement your QA process. Can updates in the library be replicated to test cases already assigned? Can you replicate updates across projects? Is the version control on the library linked into the executed test cases? All points worth considering when you decide how best to administer your test library.
The software products that we’re working on are complex. The complexity of these products needs to be modeled within our test setup to a degree. Core modules of the product we’re QA’ing will have a set of core test cases. Specific modules then have specific test cases. When modeling this within your test management tool, there comes a point where your ability to report on results becomes too complex. Can you aggregate results to deliver the high level reports you need for your clients?
Tests fail. Bugs get raised. It’s a clear cut part of the process. What’s not quite so clear cut is how we approach retesting of the fixed bugs. Do we delve into our defect tracking tool and look for all bugs that are fixed? Then write test cases for each fixed defect? Or do we search for failed test cases and then re run these? Sounds simple but pulling out that information from your test management tool isn’t always that easy.
Every test is run against a release of the product and a specific configuration of the product. Sounds simple. Then you start looking a daily builds, different platforms and operating systems to run the tests on. Before you know it a few variables start to result in thousands of permutations. Keeping track of this is absolutely key to your test process. How easy is this to track? It largely depends on the capabilities of the tools you use.
Most of these points cover core processes that we all follow as a matter of course. There’s more to most of them than first meets the eye though. When we look to support that process with a tool we can find that the process the tool imposes doesn’t quite match our requirements. This is the reason it’s important to examine these aspects prior to implementing a test management tool.
