Using a test management tool to control different versions of your testcases can seem like quite an innocuous point on the surface. However, there are many complexities to consider when you get below the surface. These complexities can have a serious impact your QA process. Ultimately you need to be able to ensure that the testcase you ran, and the associated result, is an accurate record of what actually happened.
What we mean by this is that if you execute a testcase and then log a result, any future changes should not get applied to the occurrence of the testcase that you have already run. If the changes are applied to the already run instance then you invalidate the result. For example take a scenario where the initial execution produced a pass. Then the steps changed due to an update. Yet these changes could now cause a fail if it is repeated under the same conditions. That innocuous change to the steps results in a completely different outcome. Thus invalidating the results from the initial execution.
To deal with this issue test management tools approach this in a variety of ways. These approaches are outlined below:
Full version control
With this approach the test management tool has separate instances of the same test. Each occurrence has it’s own associated result record. Each can then be modified independently without impacting, or invalidating, other instances. So there is a one to one relationship between a testcase and the result record.
When implemented well there is a master instance which is linked to other occurrences of the testcase. A change to the master can be propagated to other occurrences (when needed). This is usually at the discretion of the user so he/she can opt to update instances of the where it hasn’t been executed. The user would not update testcases where they have already been run. This update is at the discretion of the user so there is no enforced process within the test management tool which prevents you updating and invalidating a previous run.
Partial version control
This approach is similar to the full version control approach in as much as there are separate occurrences of the testcase. There is a one to many relationship between the test and the results still. However, there is no ability to link similar instances of the same case. So if you update one occurrence you can’t automatically update other occurrences. You need to update those occurrences manually.
This test management approach does have it’s advantages. You can’t accidentally update a testcase automatically and invalidate the result. However, it is cumbersome if you want to make minor changes to all instances of the same case as you have to manually update all instances by hand.
No version control
With this approach there is a master testcase and that single instance has multiple results logged against it. For example different results to cover different occasions when the test was run against different builds/versions of the product. In this scenario any change to a testcase then applies to every results logged against it. Here there is a one to many relationship between the test and the results.
This is probably the simplest approach to manage from the users perspective. Clearly though it’s limited in terms of how changes are related to results which have already been run. For example you can change the original steps significantly and those change apply to results which may already be passed or failed. This then invalidates previous results. This approach can be useful in environments where you don’t care much for the history of previous results. It can be a disastrous approach where you do need that history, perhaps in environments where traceability, repeatability and accountability are important (e.g. air traffic control, medical, nuclear, etc).
When implementing a test management tool it’s important to pay close attention to which approach of version control you are going to be using. None of the tools available today give you the choice. So the tool you pick will enforce one of the approaches above, whether you like it or not. And once you’ve invested a significant amount of time setting up your test management tool you’ll be stuck with the approach you’ve chosen. Choose wisely.
