The functional analysis of test management tools can be broken down into more than 100 separate features. In fact, our our side-by-side test management tool feature comparison has identified well over 100 features. Having lots of features isn’t always a good thing. Let’s face it not all of them are significant. However, there are 7 core and complex features that every test management solution has to address in one way or another.
The core features are:
In this write-up, we’ll be seeing how Xray stands up to the 7 criteria listed above.
Xray implements test cases as Jira issue types. When you start out with Xray, you start with the standard Jira issue that includes things like the reporter and assignee fields on your test cases. That reporter and assignee show up on the test set issue too. Again the Set entity in Xray is a standard Jira issue type. You can assign tests and sets to people just like you do standard Jira bugs.
Individual tests or sets can be added to a plan and you can assign an execution to a particular tester. Individual test cases can also be assigned to a tester.
Assigning individual test cases to a different tester
Xray implements version control on test cases. If you update a test case, it doesn’t update the test case details that were used in previous runs. You get the option to update or merge details into the existing runs. Sounds like an obvious feature but it’s not available on some Jira plugins. It’s a crucial feature if you’re serious about implementing an effective process.
As part of this you also get the case history or the Xray history as it’s labeled. Every detail, every change, to your test case is tracked.
Tracking changes in Xray
You can add datasets in Xray. When you do this you get three options. When defining the parameters or values within that dataset you can add:
Once you've defined the values, you have the ability to make your data sets combinatorial. This allows you to easily create many test steps with permutations that are automatically created as combinations from the values you provide. You can add datasets at the test case, test plan, and text execution level. That’s quite an advanced feature, not available in most solutions.
Creating parameter or dataset
With Xray, you have the ability to build a repository by folders, and filter using the standard jira issue type fields like labels, components and much more. You also have the ability to filter using JQL.
Tests can then be added directly to plans and you can group them into Sets, which gives you another logical way of grouping test cases. You can also bypass the creation of sets or plans and add tests directly to an execution. This is useful if you don’t want to update a test plan but need a few additional tests in a test run.
Repository of Library
Xray has the concept of a case having a status for various different scopes. For each scope the latest executed test result can be viewed.
Maybe you have several executions of the same test. Let’s say the first execution was completed last week and you have another execution lined up for this week. This week’s run is still in a Todo state. For your report, you want to pick up the last result that you’ve completed and ignore the Todo state or non-final state. You can view the result as it is now (e.g. To-do) or look at the last executed result (e.g. pass or fail).
Displaying test status value
At some point, you'll need to identify all the failed test cases, and possibly group them into a dedicated execution run and re-test. This can be easily done using Xray.
If you're just looking at re-running from a Test Plan, it’s a simple thing to do and it can be done by:
Creating test execution
Xray implements the concept of configurations and releases effectively. You can track your testing against configurations by defining environments. You can also track your testing against releases with the standard Jira releases/versions.
Defining test environments or configurations under Xray settings
Xray’s ability to implement everything as a standard Jira issue type brings about power and flexibility. Consider though that some advanced JQL searches only come with locally installed versions of Jira. Hosted solutions don’t support some of these advanced JQL queries. Not a huge issue so don’t let it distract from the power of this solution. Xray is a comprehensive test management solution embedded right in your project planning and issue management tool of choice with Xray. It has to be a serious contender in your list of solutions to trial and evaluate.
