In our test management tool side-by-side feature comparison we identified well over 100 features that you can use to compare the different tools. Not all of the features are significant though. In this review we pick a few of those features and take a look looking at how Zephyr Scale addresses 7 complexities of test management.
Those complexities and core features are:
Test cases are executed in cycles within Zephyr Scale. The cycle has an owner at the top level. Then you can assign individual test cases can to specific testers for execution. This is also supported with an 'Executed by' field that shows who actually ran the test. Then you also have the “owner”. The person responsible for developing and maintaining the test case. In short Zephyr Scale will let you record the owner, assignee and executed by values all independently.
You can filter the assignee and bulk update. It’s pretty simple to update the assigned tester in bulk. It is also possible to update the ‘executed by’ value retrospectively. That may or may not be a good thing depending on how rigorously or flexible you want to be.
Assigning the owner of a test case
In Zephyr Scale, you have the ability to increment the version of a test case. In reality this creates another test case instance. From this you can see the different versions that have been run in the Execution view which provides an accurate record of exactly which test case has been run. The history view also gives a good view of the changes that are made across various versions.
When you update a version that's already been run, the instance in the test run isn't updated. That’s good. You will get a warning that the script is outdated and you can easily update if you want to. That’s good too.
When you add a test case to a cycle, you get to pick the version you need to add. You can use old versions if required or just pick the latest. When you view the executions in the test case, you’ll see the exact version that was run in the test cycle.
The approach Zephyr Scale takes to version control of test cases is well thought out and works well.
Adding versions to test cycle
Parameters offer simple option where you can define variables and set values in the test case. The values set can be used across the test steps. When you run the test, you don't get different permutations of test steps, you get the test steps you defined with the value update.
You can also add test data at the project config level and reuse across many test cases. This is a smart way to reuse and distribute test data. When you run the test cycle, you get to see the parameter values clearly in the test script.
The two options for parameterizing test cases and test steps
Zephyr Scale manages the pool of test cases that you write with folders and labels. You can categorise by the 'component' defined within Jira, apply a status, and configure status values.
Test cases are re-used in cycles either by directly adding when viewing the cycle or creating a new test cycle. The UI for managing your libraries of tests is well laid out and works well. Simple to get to grips with and very effective.
Creating test cycle
You have the ability to aggregate results from multiple versions or builds using the “Result” type feature in the reports. Pick either the ‘Latest test execution’ result or ‘All test execution’ results. So you may have a test case executed multiple times in different test cycles, but you can choose to just have the latest result.
Reporting latest test execution
You may have a failed test in the last release and you want to re-run it. The 'start a new test execution' feature handles this. You’ll log the result of the test against a different version or build after rerunning the failed test again in the same test cycle. You can order by the last test result execution, add them to a new test cycle, and run them as a group to see if the re-tests pass.
If the previous failures display in any reports where you have run the same test case multiple times, you'll have to use the 'start new execution’ capability within the test cycle you're executing. This way you can log that individual test case against a different version for accuracy.
Rerunning a failed test
Configurations or environments are defined in the Zephyr configuration area. When you build the Test Cycle, you can allocate individual test cases to an environment and bulk assign tests to environments.
You can add a test case multiple times to a test cycle and allocate to different environments or duplicate a test cycle and run each cycle against different environments.
Bulk updating and assigning tests to an environment
Zephyr Scale handles most of the complexities of test management well and avoids unnecessary complexity. Aggregating test results across releases or builds is the only area that is slightly limited but this is an area where very few tools provide a good solution. This is not really a problem, it’s something that could be solved with a custom reporting engine if and when the need arises. On the whole this is one of the most powerful and well thought out Jira ad-ins on the market.
