In this module we start to put together some API tests to check the REST Api that is delivered as part of a Rocket Chat install. The moment the Selenium tests complete Jenkins will trigger SoapUI to run a number of REST based API tests from one of the Windows AWS client machines. In setting this up we’ll cover a few key areas:
1. Looking at the Rocket Chat API calls
2. Installing and writing SoapUI REST API tests
3. Driving SoapUI from Jenkins
4. Using the Jenkins xUnit plugin to display test results
We’ll run in to a few small challenges whilst setting this up. For example the rocket Chat Api returns some security credentials after the initial login. In order to successfully run all the other REST tests we need to pass those credentials between all the different test cases. We’ll look at how you can configure this in SoapUI.
Once we have SoapUI running the tests we’ll see how we can save the results in JUnit format. Then using the Jenkins xUnit plugin you’ll learn how to capture those test results and display them from within Jenkins. This is a great way to introduce you to an important plugin that has a multitude of uses when tracking xUnit results from various sources.
What You’ll Learn
In this module we’re going to look at creating and running some REST API tests. We’re going to use the open source tool SoapUI to run these tests. Of course these tests will all be initiated from Jenkins once the build of the Application Under Test (AUT) is complete. We’ll look to run these SoapUI API tests in serial from the same test machine that we’re running the Selenium tests from. The Selenium tests will complete then the SoapUI tests will be kicked off.
Once this is configured we’ll have these pieces in place on our test automation rig:
This is an important module because we’re starting to build out the bredth of our test coverage now. You’ll find the configuration of Jenkins quite straight forward now. However, we’ll show you how to overcome a few key challenges like passing parameters in SoapUI and displaying JUnit results within Jenkins.
The Tools We’ve Chosen
We’ve picked SoapUI. Mainly because of it’s popularity but also because it’s GUI is easy to get started with. It’s also simple to covert the tests we write into a test suite that can be run from the command line – which is what Jenkins needs.
SoapUI also provides us with the capability to record test results in JUnit format. This format can be read by the Jenkins xUnit plugin which allows us to display all of our REST Api test results from within Jenkins.
Worth pointing out here that the Rocket Chat API is a REST api. SoapUI might imply support for SOAP api’s but it does support REST too.
Prerequisites
If you’ve followed Modules 1, 2 and 3 you should already have your Amazon Virtual machine environment up and running along with Jenkins and Selenium. This setup gives us the 3 machines we’ll need to use in this module.
1. Windows Master machine: this is running Jenkins and controlls all our other machines (including the installation of the AUT and the exection of our Selenium tests). This machine will be responsible for kicking off our SoapUI API tests
2. Linux Client machine: this Ubuntu linux machine is run up on demand by Jenkins and then has the AUT (Rocket Chat) autoamtically installed on it. This machine provides the web interface for the Rocket Chat application and the API for the Rocket Chat application.
3. Windows Client machine: this Windows Server 2012 machine has 3 browsers (Ie, Firefox and Chrome) installed on it along with Selenium. The Windows master machine kicks off the Selenium tests that drive the 3 different browser tests against the Rocket Chat application.
Check the Status of your AWS Machines
Both of the windows machines should already be running. The Linux machine may or may not be running. The Linux machine is run up automatically by Jenkins so it’s fine if it’s not running. Whatever the state of the linux machine you should see the Windows machines status in the AWS console as follows:
Open an RDP Terminal Session on the Windows Master Machine
With these windows machines running you’ll need to open an RDP session on the Windows Master machine. This is where we’ll be configuring Jenkins.
Then enter the password (you may need your .pem private key file to Decrypt the password if you’ve forgotten it) to open up the desktop session.
Start the Linux Client Machine
IF the Linux machine isn’t running with the AUT installed then we need to start it. We can get Jenkins to do this for us. Once you have an RDP session open to your Windows Master machine you should have the Jenkins home pages displayed. If not open a browser on this machine and go to this URL:
http://localhost:8080/
From here you can start the ‘BuildRocketChatOnNode’ job and start up the AUT.
Once RocketChat is up and running we’ll need to know the host name that Amazon has given our new Linux instance. We save this in our ‘publicHostname.txt’ file that is archived as part of our build job. So if you go to this directory using Explorer
C:\Program Files (x86)\Jenkins\jobs\BuildRocketChatOnNode\builds\lastSuccessfulBuild\archive
You should find this publicHostname.txt file…
Open this with notepad and make a note of the hostname. We’ll need this while we setup our API tests.
Open an RDP Terminal Session on the Windows Client Machine
As we’re utalising our existing Clicent Windows 2012 R2 test machine we don’t need to configure any new Amazon instances or machines. We have what we need running already. We just need to open an RDP terminal and install the software we need. To open the RDP terminal:
Again enter the password (you’ll need the .pem private key file again if you need to Decrypt it) to open up the desktop session.
At this point you should have…
1. An RDP session open to your Windows Master machine
2. An RDP session open to your Windows Client machine
3. Your Linux Ubuntu machine running with Rocket Chat installed
From here we’ll start to setup our Windows test machine and configure Jenkins on the Master machine to run the tests.
Part 1: Install SoapUI
We’re going to use our existing Windows client machine for this. So once we have the RDP session open we can download SoapUI and install it on our Windows Server 2012 R2 64 bit machine.
Only a few steps to follow here then, all to be carried out on the
Windows client machine:
1. Open a browser
2. Enter the following address:
https://www.soapui.org/downloads/open-source.html
3. Download the ‘Windows Installer (64 bit)’ package
4. Run the installer, accept the agrements and accept all the default options…
Once you’ve finished you should be presented with the SoapUI user interface:
Now we have SoapUI running we can check the connection to the Rocket Chat Rest API.
Part 2: Connect to the Rocket Chat API
So Rocket Chat, our AUT, comes installed with a REST Api. This API is documented here:
https://github.com/RocketChat/Rocket.Chat/wiki/REST-APIs
On
your install of Rocket Chat you should be able to connect to the Api directly with a browser here…
http://<your-amazon-aws-instance >:3000/api/
Remember earlier we said you could get your host name from the publicHostname.txt file on the Windows Jenkins master machine. So if you replace the <your-amazon-aws-instance> text with your public hostname you should be able to connect to this Api from any remote machine. You’ll see something like this:
If you have trouble connecting from outside your Amazon Virtual Private Cloud you’ll need to check your Security Group settings in your AWS control panel. If all else fails connect from the Windows Client machine (the one with SoapUI now installed).
One final check before we start writing some SoapUI tests. Lets check we can the version of Rocket Chat that’s running from the API. We’ll do this from the Windows client machine. So in the Windows Client machine RDP session…
1. open Chrome (it must be Chrome)
2. browse to this URL
http://<your-amazon-aws-instance >:3000/api/version
3. check the Rest response
You should see this Json structure returned in the Rest response:
The reason we had to use Chrome there is that Ie isn’t that happy displaying Json without messing around a bit. Easier just to use Chrome.
We’ve confirmed that we can connect to the Rocket Chat API and that we can get a valid response. Lets start writing some tests.
Part 3: Creating Some REST Requests
Now we have access to the Rocket Chat API we can start creating some REST requests. These requests will be written in SoapUI and executed on the Windows Client machine. Eventually we’ll have these tests kick off automatically from Jenkins.
For now though lets write some requests in the SoapUI gui and get them working. We need to create a new REST project and …
1. create a new REST project and enter the Rest URI
and then enter the URI for your Rocket Chat Rest API and click ok
http://<your-amazon-aws-instance >:3000/api/version
This should give you something like this…
The navigator section here showing you the hierarchy of project, endpoint, resource, method and finally request. We’ll look at each of these in detail next.
2. Define the request
In the request window we’ll configure a request to check the version number. So set the following:
Method: GET
Endpoint: (already set)
Resource: /api/version
It should look like this:
This particular request doesn’t have any parameter or body send. So we’ve defined enough now to check this. Before we move on though a few basic points about SoapUI
In SoapUI we basically have 3 main panels for each request:
1. Request details: the method (eg. GET, POST, etc), Endpoint, Resource and Parameters
2. Request body: any additional content (e.g. body or parameters) that need to be included in the request
3. Response: the response that comes back from the Endpoint after the request has been sent
These different panels are shown here…
In addition to this we have our green ‘Submit Request’ button that sends the request we define. Let’s see how this works.
3. Test an Initial Request and Check the Response
We can now click on the ‘Submit Request’ button …
And check the response. The result? Probably not very exciting. You probably got the error message…
“The content you are trying to view cannot be viewed as XML”
We already know, from our earlier call using a browser, that the response is in Json format. Just need to click the ‘Json’ tab to view the response then:
We can also check the raw response and make sure we received an Http 200 Ok response code:
Next then we’ll add a few more requests to login, list the rooms and log out again. From there we can add all these requests to a test suite and define some assertions.
4. Add More Requests
At this point we need to add three more requests. Requests to login, list the rooms and log out again. To add each of these 3 requests we’re going to repeat these steps:
1. Clone Resource
2. Name Resource
3. Open the request (double click)
4. Set the Method
5. Set the resource
6. Set the parameters
You’ll need to click in the parameters field here then the dialogue box opens up where you can enter the parameters
7. Specify the Post data
On some requests you need more than just the end point and resource. You’ll need to send additional data (e.g. like a browser does when you complete and html form and press the submit button). You’ll need to define that data and pass the data using a specific content-type
8. Set the header
You only need to set these values once you’ve completed the login request (this is NOT needed on the login request). Basically you run the login request, an auth token and userId are returned from that request. In all sebsequent requests you’ll need that ‘auth token’ and ‘userId’ to complete the requests.
You’ll need to repeat the above steps and create these three requests:
Request 1 – Login
Resource Name: Login
Method: Post
Resource: /api/login
Data (1): user=admin
Data (2): password=tester123
Post QueryString: <checked>
Create this request then run it.
You should have something like this after you’ve configured and run the request:
And the Json response containing data like this:
{
“status”: “success”,
“data”: {
“authToken”: “-7GGh68UQ5B0zbuLAEj29DO95H3uNg2HvRgZg63-nMF”,
“userId”: “bNitLrJ3rhZq5uLMR”
}
}
You need then to take the authToken and userId and use them in the next requests.
Request 2 – List the Chat Rooms
So go back and repeat steps 1 to 8 above using this data:
Resource Name: publicRooms
Method: Get
Resource: /api/publicRooms
Data: <remove data fields>
Post QueryString: <unchecked>
Header(1): X-Auth-Token=<authToken>
Header(2): X-User-Id=<userId>
Create this request then run it.
You should get a response like this:
And Json data equivalent to this:
{
“status”: “success”,
“rooms”: [ {
“_id”: “GENERAL”,
“ts”: “2016-03-01T10:57:10.758Z”,
“t”: “c”,
“name”: “general”,
“usernames”: [“admin”],
“msgs”: 0,
“default”: true
}]
}
Request 3 – Logout
So go back and repeat steps 1 to 8 above using the data below. Make sure you clone the ‘publicRooms’ resource as this one already has the headers defined:
Resource Name: logout
Method: Post
Resource: /api/logout
Data: <remove data fields>
Post QueryString: <unchecked>
Header(1): X-Auth-Token=<authToken>
Header(2): X-User-Id=<userId>
Create this request then run it.
You should get a response like this:
And a Json response containing this data:
{
“status”: “success”,
“data”: {“message”: “You’ve been logged out!”}
}
Don’t forget if you want to expand all these tests and create your own you can refer to the list of supported requests here:
https://github.com/RocketChat/Rocket.Chat/wiki/REST-APIs
We now have a few requests that work. We just need to convert them to tests and create some assertions that check the responses.
Part 4: Converting the REST Requests to Tests
From here then we have 4 REST requests that all give us valid responses. With SoapUI the next step is to convert them to tests and create the assertions that validate the responses. Two ways you can do this:
1. Create individual test cases for each Request
NOTE: you can do this one by one (as listed below) or you can jump to section 2 below and convert the whole lot in one go. The choice is yours.
i. Right click on the ‘request’
ii. Select ‘Add to TestCase’
iii. Create the Test Suite (just give it a test suite name)
iv. Create the Test Case (just give it a test case name)
v. Accept the default options for adding the test case
Easier to use method 2 though…
2. Create test cases within a test suite for all resources.
i. Right click on the ‘end point’
ii. Select ‘Generate Test Suite’
iii. Select <create> test suite and create ‘One Test Case for each Resource’
iv. Give the Test Suite a name (e.g. RCTestSuite)
In the navigator panel you should then see something like this…
We now need to update each Test Case so that it has some assertions that check the results from the test run. We’ll also need to look at passing some values between test cases. We’ll need to pass values because the login test case response has the ‘authToken’ and ‘userId’ values that subsequent test cases need.
3. Add the Test Case Assertions
We can select the first test case (the one that gets the version of Rocket Chat) and add our first assertion. Make sure you have the Request work space open:
Then click on the assertion button to open the assertion panel:
Then we’ll add our first assertion. For this test case it’ll just be that the response is a valid HTTP Status Code 200 Ok:
After you’ve clicked the ‘Add’ button just enter 200 as the code:
And then click okay. You can confirm that this works by clicking the green ‘Submit’ button for the request and then check that your assertion passes:
I’ll show you one more example of an assertion that checks the content of a Json response. Then you’re on your own for adding a range of assertions for all of the test cases.
Next select the Login TestCase Request and open the work space (double click the request):
This time you’ll notice that I’ve run the request before I’ve created the assertion. You can see the Json response that I want to create the assertion for now.
Click the add assertion button and select the ‘JsonPath Count Match’:
We know that if the login is successful then we’ll see this in the response…
{
“status”: “success”,
“data”: {
“authToken”: “-7GGh68UQ5B0zbuLAEj29DO95H3uNg2HvRgZg63-nMF”,
“userId”: “bNitLrJ3rhZq5uLMR”
}
}
What is absolutely crucial is that we get back this ‘data’ structure because we need the ‘authToken’ and ‘userId’. We need these for the subsequent requests we’re going to send. So let’s create this assertion so that it checks for the ‘data’ structure. We’ll enter these values in the dialogue box:
JSONPath Expression: $.data
Expected Result: 1
You can click the test button to check this in the dialogue box too:
From here you know enough to start experimenting with adding further assertions to the other requests. Just follow exactly the same process we’ve covered above.
What’s important now is to string these Test Cases together and pass the ‘authToken’ and ‘userId’ from one Test Case to the next.
NOTE: might be worth saving your project at this point. File menu, Save All Projects. You can save it with the default name of “REST Project 1”
4. Passing Parameters to our Test Cases
What we need to do now is work at the Test Suite level, link all these Test Cases together in a chain and pass some parameters between the Test Cases. So double click on the ‘Test Suite’ node and open the ‘Test Suite workspace’. Then click on the ‘Properties’ button to open up the properties panel.
We need to create a couple of properties at the Test Suite level. So click the ‘plus’ button and add ‘userId’ and ‘authToken’ properties.
These properties will be set by the Login test case. Then these properties can be used by the other test cases as part of the Headers to enable access for each request. Next step is to set these Test Suite level property values.
Right click on the ‘Login’ TestCase and add a ‘Property Transfer’ test step to this test case. You can give this test step the default name of ‘Property Transfer’:
Then open the workspace for this Property Transfer Test Step (double click). Then add 2 Transfer values, userId and authToken:
Now all we need to do is set the source and target for these property transfers. The source will be the Login Test Case response that returns the ‘userId’ and ‘authToken’. The Target will be the Test Suite level properties (which then makes these values available to ALL test cases in the suite).
Set the authToken first by setting these values…
authToken
Source: Request 1
Property: Response
Path Language: JSONPath
Path: $.data.authToken
Target: TestSuite
Property: authToken
Path Language: JSONPath
It should look like this…
Then set the userId …
userId
Source: Request 1
Property: Response
Path Language: JSONPath
Path: $.data.userId
Target: TestSuite
Property: userId
Path Language: JSONPath
Which should look like this…
We can check this setup by clicking on the ‘Run All Property Transfers’ button. From the Transfer log we should see these values picked up from previous responses from the login request:
And now the properties are available at the Test Suite level we can use them to run the other test cases. Just need to tweak the other test cases where the ‘authToken’ and ‘userId’ are currently hard coded.
Edit the ‘publicRooms’ test case and click on the ‘Header’ button so that we can add these TestSuite parameters to the Header values. We’ll need to add these property strings to get the Header values to use our TestSuite properties:
X-User-Id: ${#TestSuite#userId}
X-Auth-Token: ${#TestSuite#authToken}
So replace the hardcoded values with these strings so that you have this:
You can check this setup by clicking the green submit request button for this test case. Assuming it works what we now have is the ability to run the Login test, get the dynamic authentication variables that Rocket Chat passes us back in the response, and use these variables in all the other test cases.
All you need to do now is update the Logout Test Case to. The logout test case should look like this:
Next step then is to get these TestCases to all run in series, end to end.
5. Running the Test Suite
Well this bit is probably easier than you’re expecting. Just open the TestSuite workspace and click on the ‘Run’ button. You should see each test case within the Test Suite run in turn. Each test case highlighted in green to indicate that it’s passed.
All we need to do now is get this running from the command line and get Jenkins to kick the API test run off.
Oh! Probably worth saving your SoapUI project at this point!
Part 5: Running the SoapUI Test Suite from the Command Line
There is an easy way to work out how to run this Test Suite from the command line. You can initiate a command line run from the SoapUI gui and then copy the command line that the GUI gives you.
1. Launch the TestRunner
To do this, from the ‘Project’ menu select ‘Launch TestRunner’:
This will give you the test runner dialogue box. One the ‘Basic’ tab all we need to define is the path to the TestRunner. When we run these tests from the command line we’re basically running a windows batch file called ‘testrunner.bat’. You just need to set the path to this file to:
TestRunner Path: C:\Program Files\SmartBear\SoapUI-5.2.1\bin
The version number in the directory may change for you but it should be like this…
Then on the ‘Overrides’ tab we just want to select our Endpoint. We need to see how to pass the value in on the command line so setting it here will get SoapUI to use this value as a parameter.
Then click ‘Launch’ to run this. A successful run should look like this…
A few exceptions in there with this release of SoapUI. Nothing to worry about. We can fix these by stopping SoapUI trying to load these plugins.
2. Clear the Exceptions (optional step)
To clear all these unwanted exceptions open explorer and go to this directory
C:\Users\Administrator.soapuios
Close SoapUI after you’ve saved your project. Then rename the plugins directory to something like ‘plugins-old’. Like this…
3. The Command Line
What we’re really interested in though is the command that was run by the TestRunner. If necessary re-run the test runner from SoapUI. Then click ‘OK’ and scroll back up to the top of the Test Runner windows. You need to locate this…
From this you can pick out and copy the command line you need. Like this…
directory: C:\Program Files\SmartBear\SoapUI-5.2.1\bin
command: cmd.exe /C testrunner.bat -ehttp://<your-aut-host>:3000 C:\Users\Administrator\REST-Project-1-soapui-project.xml
You’ll want to modify this slightly so that you have this format…
cmd.exe /C “C:\Program Files\SmartBear\SoapUI-5.2.1\bin\testrunner.bat” -ehttp://<your-aut-host>:3000 C:\Users\Administrator\REST-Project-1-soapui-project.xml
Note the path in front of the bat command and the inverted commas. This is what we’ll test before we add it to a Jenkins job. So lets check this from a command line prompt. Open a command window and run this command:
When it’s completed check the error level with a command like ‘error %errorlevel’:
If all the tests passed then the error level should be ‘0’. If any failed then the response should be ‘-1’. You can test this if you like by slightly modifying one of your test cases in SoapUI to make it fail (change the assertions on one of the TestCases or something). Then run the command line command again and check the error level is ‘-1’.
Nearly there now. Just need to plug this all into a Jenkins job.
Part 6: Running the SoapUI Test Suite from Jenkins
First we’ll need that SoapUI command from the previous step. So you’ll have something like this…
cmd.exe /C “C:\Program Files\SmartBear\SoapUI-5.2.1\bin\testrunner.bat” -ehttp://<your-aut-host>:3000 C:\Users\Administrator\REST-Project-1-soapui-project.xml
Back on to our Windows Master machine that’s running Jenkins (open the RDP session if necessary). Then bring up the Jenkins home page. From here we need to create a new job / new item:
We’ll make this a ‘Freestyle Project’ and call the job ‘RunApiTests’
We have a few key things to configure on this new job:
1. Restrict where this project can be run: SeleniumTestClient
2. Build Triggers:
Build after other projects are built
Projects to watch: RunSeleniumTests
Trigger only if build is stable
Then we need to add 3 new build steps. To create each one click on the ‘Add build step’ button, select the build step type and set the parameters (as set out below)
1. Copy artifacts from another project
We need this to pull in the publicHostsname.txt file from the ‘BuildRocketChatOnNode’ job. This way this build job will know where Rocket Chat is running.
2. Inject environment variables
We’ll take the contents of the publicHostname.txt file, that contains the host name where Rocket Chat is running, and use this value in the next batch command job.
3. Execute Windows batch command
This is where we’ll run our SoapUi command on the remote machine (the machine labled SeleniumTestClient). Note that we don’t need to hard code the host name. We can use the %PUBLIC_HOSTNAME% variable that’s set from the publicHOstsname.txt file. So enter your command like this:
Using this command:
cmd.exe /C “C:\Program Files\SmartBear\SoapUI-5.2.1\bin\testrunner.bat” -e%PUBLIC_HOSTNAME% C:\Users\Administrator\REST-Project-1-soapui-project.xml
No post build actions at this stage. So we can just save this job and return to the Jenkins home page.
Before we run this though lets setup things up so that Jenkins can actually report on the status of the tests run by SoapUI. To do this we’ll need a new plugin installed in Jenkins.
Part 7: Reporting the SoapUI Test Results in Jenkins
What we want to do now is see the individual test case results displayed in Jenkins. SoapUi can save the test results in a JUnit style report format. Then we can get the Jenkins ‘xUnit’ Plugin to display the test results. Lets install the plugin first.
1. Install the xUnit Jenkins Plugin
Just go to the Jenkins ‘Manage Plugins’ page, select the available tag and search for “xUnit”.
Click install without restart. Once the plugin is installed we can configure our SoapUI project to save the test results in JUnit format.
2. Modify the Jenkins ‘RunApiTests’ Job
Back on the Jenkins home page configure the ‘RunApiTests’ job.
What we need to do now is configure our SoapUI command line so that it save the test results in a JUnit format. We can do this by changing our ‘Execute Windows batch command’ from this:
cmd.exe /C “C:\Program Files\SmartBear\SoapUI-5.2.1\bin\testrunner.bat” -e%PUBLIC_HOSTNAME% C:\Users\Administrator\REST-Project-1-soapui-project.xml
to this…
cmd.exe /C “C:\Program Files\SmartBear\SoapUI-5.2.1\bin\testrunner.bat” -e%PUBLIC_HOSTNAME% -sTestSuite -j -fc:\jenkins\workspace\RunApiTests\ C:\Users\Administrator\REST-Project-1-soapui-project.xml
Note that we’ve added three parameters to this command:
-sTestSuite: defines the TestSuite to run (which we called ‘TestSuite’)
-j: Turns on exporting of JUnit-compatible reports
-fc:\jenkins\workspace\RunApiTests: which specifies the directory to save the report in
So now when we run this job we should get this file created on the client machine that will contain the individual test results:
C:\jenkins\workspace\RunApiTests\
It’s important to save this in the Jenkins workspace on the test machine. This way Jenkins will automatically make the JUnit files available to the master Jenkins machine for processing.
Don’t save this job yet. Still one more thing to configure. We have to get Jenkins to read those results and publish them.
3. Publish xUnit Test Results
Whilst we’re still editing the ‘RunApiTests’ job we can add a post build action. Click on the ‘Add post-build action’ button and select ‘Publish xUnit test result report’
Once this is added we need to add an xUnit type of report. Click on ‘Add’ and select ‘JUnit’
From here we can set the file names Jenkins needs to look for where the JUnit test results reside. Well SoapUI will be creating a JUnit file called TEST-TestSuite.xml. So we’ll need to configure this ‘Publish xUnit test result report’ to look for TEST*.xml files. So set these parameters:
JUnit Pattern: TEST*.xml
Skip if there are no test files: un-checked
Fail the build if test results were not updated this run: checked
Delete temporary JUnit files: checked
Stop and set the build to ‘failed’: checked
And that it’s really nothing else to set here (you don’t even need to set any of the test thresholds).
Save this job now and lets run the full set of Jenkins jobs.
On the Jenkins home page click the “Run now” button for the ‘BuildRocketChatOnNode’ job. This should end up running our full set of 5 jobs. And once complete we can check the console output for our ‘RunApiTests’ job.
That console ouptput should show you something like this at the end:
c:\jenkins\workspace\RunApiTests>exit 0
[xUnit] [INFO] – Starting to record.
[xUnit] [INFO] – Processing JUnit
[xUnit] [INFO] – [JUnit] – 1 test report file(s) were found with the pattern ‘*.xml’ relative to ‘c:\jenkins\workspace\RunApiTests’ for the testing framework ‘JUnit’.
[xUnit] [INFO] – Check ‘Failed Tests’ threshold.
[xUnit] [INFO] – Check ‘Skipped Tests’ threshold.
[xUnit] [INFO] – Setting the build status to SUCCESS
[xUnit] [INFO] – Stopping recording.
Warning: you have no plugins providing access control for builds, so falling back to legacy behavior of permitting any downstream builds to be triggered
Triggering a new build of CloseClientRDPSession
Finished: SUCCESS
If you get warnings about clocks being out of sync check the next section. If there results are okay then move on to ‘Reviewing the JUnit Results’.
4. What to do if Clocks are Out of Sync
The ‘xUnit’ plugin depends on the clocks, between master and slave nodes, to be in sync. If they’re not in sync then you might see an error like this in the console log for the ‘RunApiTests’ job
[xUnit] [ERROR] – Clock on this slave is out of sync with the master, and therefore I can’t figure out what test results are new and what are old.
Please keep the slave clock in sync with the master.
[xUnit] [INFO] – Failing BUILD.
If that’s the case take a look at the Jenkins configuration to confirm this.
So indeed on this system the client machines clock is running 17 seconds behind. To resolve this you’ll need to run these 4 commands on the ‘Client’ machine:
net stop w32time
w32tm /config /syncfromflags:manual /manualpeerlist:”time-a.nist.gov, time-b.nist.gov, time-c.nist.gov, time-d.nist.gov”
w32tm /config /reliable:yes
net start w32time
Like this…
Then wait a minute or two. Go back to the Jenkins master machine and refresh the status:
And clocks are all back in sync.
5. Reviewing the JUnit Results
We’ve set the xUnit reporter to pass the build/job if all the tests fail. What happens if we see a failure though? Well if you first check the workspace for the job:
You’ll see the TEST-TestSuite.xml file and the other log files that were saved from the SoapUI test run:
From here if you look at the drop down menu for the last test run you’ll see an option for viewing the test results:
Click on this and you can then start drilling down through all of the unit test results:
And if you click the ‘History’ item in the right hand menu bar you’ll see a track record of all the SoapUI JUnit test results.
Conclusion
Well nothing too difficult on the Jenkins side of things this time round. By now you should be experienced enough to configure the Jenkins build job easily to handle the execution of the SoapUI test run. The only new thing we introduced here for Jenkins is the capability to process JUnit test results and display them. A key capability for tracking test results in many other tools controlled by Jenkins too.
We delved into a lot of SoapUI features to create a handful of REST based Api test cases. The only real complexity here being the passing of parameters between test cases. Once you’ve seen how to do this it not difficult to repeat the process and build up a good pack of Api test cases. Key to the success of this module though is the ability to run the SoapUI tests from the command line and get Jenkins to pick up the JUnit test result file. As long as you know where to pick up the SoapUI test runner application and configure the xUnit plugin none of this should have been too challenging.
At the end of this module then we have a test automation environment that covers installing our appliation under test, running a suite of GUI based browser tests with Selenium and now running a handful of API test cases using SoapUI. All started and executed with ‘one click’ on a ‘Run Now’ button.
In the next module we’ll look at how we can build in some performance tests with JMeter driven from Jenkins too.
