You’ve configured your test hosts (Bringing Manual and Automated Tests Together) and you’ve set up and run automated tests from QAComplete (Setting Up Automated Tests in QAComplete). Now we’re running these automated tests how do we track the results and how do we follow the traceability back to our requirements?
For many of our automated tests we’ll want to track traceability back to the requirements they are providing us with test coverage on. From an QAComplete implementation point of view automated test cases are no different to manual test cases. Automated test cases are defined in QAComplete and linked back to the requirement. This traceability can be seen and modified when you view the test case in the test library.
Once we have this traceability setup the automated test can be run as part of a test set and a test run record created. In this next section, Run History, we talk about tracking the run history and then in the Dashboards section we look at how to create traceability reports.
So each test set run (which may or may not contain automated test cases) has a result set stored in the run history record. This can be found here:
You can search through and group the run records with the ‘Group By’ and Quick Search features. Bear in mind that you can’t filter on execution type and see just the automated tests. This is because each test run relates to a test set that can have multiple test cases included in it. Multiple test cases in a test run can be a mix of automated and manual tests.
You can check the test type (either manual or automated) by drilling down into the individual tests themselves. To do this view the test runs and then open up the run view, followed by the review test view. In the test view, under the status info section, you’ll see the execution type.
To start looking at test runs that focus specifically on automated tests we’ll need to create filters and look at some of our test dashboards.
If you want to see test run results related to other artifacts (like requirements) you can view the “Test Runs by Requirement” dashboard. This shows the individual test case results per requirement.
In this example three test cases have been executed (across one or more test sets) where all three test cases have been linked to the requirement. The test results shown here will be combined results from both the manual and automated tests that have been executed (not forgetting that test sets can contain both manual and automated tests).
In another example you can see the Test Runs by Configuration dashboard. As long as the test set (that contains the automated tests) is linked to a configuration and the test set run is actually run on that configuration then the run results will show up on the ‘Test Runs by Configuration’ dashboard. It worth pointing out that this dashboard view doesn’t list individual hosts just the type of host configuration that the test set was linked to.
If you are looking to report on just test cases that are automated then the best approach is to create a filter. Create a filter that based on the field ‘automated’ and set the condition to ‘true’.
With a filter setup up you can now view the dashboards and apply the filter. For example this ‘Test Library’ dashboard shows the last run status for tests but has the ‘Automated’ filter applied. So the result is the display of last run status only for automated test cases.
QAComplete lets you mix automated and manual tests in any way you like. This flexibility can be good, if for example, your focus is requirements coverage. If you are focusing on requirements coverage and all you want to see is test results (whether they are automated or manual) then mixing these different test types in Test Sets that are requirements focused is the way to go. So for example you create a Test Set containing manual and automated tests that cover one particular requirement.
For some QA teams though there’s a need to keep manual and automated tests separate. Perhaps your automated tests are run just to cover the regression aspects of your testing. In which case you may not be so worried about tracking the requirements coverage. So you’d take a slightly different approach to setting up QAComplete in this case.
If you want to keep your automated tests separate then you need to approach the construction of your test sets in such a way that a test set only contains automated tests. The best way to do this is as follows.
First, in the Test Library, you need to identify test cases that are automated. Here again we can use the ‘Automated’ flag that signifies if a test case is automated. Add the ‘Automated’ flag to the list of fields displayed.
Then make sure you have the ‘Automated’ flag selected.
This will help you identify the automated test cases from the manual test cases.
Using this in conjunction with a filter setup with the ‘Automated tests flag’ and you can pull out the specific automated test cases you need to create a dedicated automated test set. Once you have those test cases listed use the ‘Create Test Set’ button.
The key here is to make sure you create a folder structure for your Test Sets that allows you to separate out automated and manual test sets. Also make sure that you have a naming convention for your test sets that identifies the test set as an automation set (the reasons for this will become clear in a minute).
Then when you create test sets that only contain automated tests you make sure you store those test sets in the ‘Automated Test Sets’ folder. Also make sure that the title/name for the ‘Automated Test Sets’ contains a string like ‘AUTO:’.
When setup like this we’re in a position to filter our test runs and dashboards based on this folder and/or title string criteria. For example if you now view the Run History you can create a filter that pulls out only the Test Runs for Test Sets where the name contains ‘AUTO’.
If you then group by ‘Date Started’ and apply your ‘Automation Runs’ filter you’ll see all your automation runs filtered and grouped by the run dates.
The ability to define automated and manual test cases in one test library location now gives QAComplete the ability to deliver full traceability from all test types back to requirements and other project artifacts. This then gives you the ability to report on test coverage for both manual and automated tests in a consistent way.
As we’ll see in future posts this capability with TestComplete integration, the new Selenium integration (more on this soon) and ultimately integration with SoapUI/Ready!API makes QAComplete the hub of your test execution and reporting capabilities.
