My SoapUI Thoughts on Effective Testing in SoapUI

Implementing a Datasink

A common use-case in automated testing is to create a log of the progress of the tests. In SoapUI and other tools, this capability is often implemented by means of a datasink, available though built-in features in the Pro version of SoapUI and Ready!API. In this article I take a look at what’s involved in implementing an alternative via a script in the open-source version of SoapUI.

Prerequisites

  • A test case configured to test the todos resource.

Goals

  • A test case which implements logging of the users resource test via a datasink.
  • An implementation which reads test data from a JSON file.

The good news is that a datasink can be implemented with very little effort. In this example, I’ll test the users resource via test data stored as a JSON file. Go to the JSON Placeholder users page and copy the text directly into a file called users.json. For the purposes of this test, I restricted the file to the first four user records.

To process this data, we need to derive a class from Datasource as we did for the CSV and Excel formats. Let’s call it JSONDatasource, implemented as follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import groovy.json.JsonSlurper

@InheritConstructors
class JSONDatasource extends Datasource
{
	boolean initialise()
	{
		try {
			final def rawData = new File( fileName )?.text,
				   JSONData = new JsonSlurper().parseText( rawData )

			def line = []
			JSONData[ 0 ]?.each { line += it.key?.toString() }
			testData.add( line )

			JSONData?.each { record ->
				line = []
				record?.each { line += it.value?.toString() }
				testData.add( line )
			}
		}
		catch( Exception e ) { testRunner.cancel( "$e" ) }
		true
	}
}

As with the other derived classes, the JSON processing is pretty straightfoward. Groovy’s JsonSlurper class makes processing JSON data very easy, but note that the sequence is slightly different from what we saw in the CSV and XLS classes. With JSON, we’re dealing with a hierarchical format where the field names are repeated in each record; the initialise() method here uses the first record to collect the property names, and then iterates over each record including the first to collect the property values. The processing for XML data is very similar.

To access this class via an external script, you need to create a file called JSONDatasourceFull.groovy with the JSONDatasource class as the first class in the file, followed by our familiar Datasource base class. Don’t forget to include the usual import statements together with the import required for the JsonSlurper class as shown above. You can then clone one of the existing test cases, making the changes required to point it to the users data and the JSONDatasourceFull.groovy script.

With that in place, let’s look at the changes to the base class required to implement a datasink designed to log the progress of the test case.

What our log should look like

Given that the test case might process thousands of data records, the resulting log should be fairly concise while still bearing enough information to be useful. I’ve found that a log with lines in the following format provides me with enough information to trace any problems back to the original data.

29-Dec-2016 16:22:59.688: [Get User by Index (id = 10)] GET http://jsonplaceholder.typicode.com/users?id=10 Status=200 (26ms)

This line is made up of the following fields:

  • The timestamp showing when the call was made
  • The name of the test step
  • The name of the HTTP method
  • The endpoint, with all the template parameters resolved at the time of the call
  • The HTTP return status
  • The response time

Additionally, each call should provide a list of any failing assertions so that we can investigate, as follows.

29-Dec-2016 16:36:47.734: [Get User by Index (id = 10)] GET http://jsonplaceholder.typicode.com/users?id=10 Status=200 (25ms)
	[HTTP OK] FAIL: [Response status code:200 is not in acceptable list of status codes]
	[1 Result Returned] FAIL: [Comparison failed for path [$], expecting [17], actual was [1]]
	[Response time] FAIL: [Response did not meet SLA 25/9]  

Each of these failed assertions includes the name of the assertion and the reason(s) for its failure. SoapUI provides this information in a way that we can access reasonably efficiently.

Implementing the Datasink

In this implementation, most of the work is done in the runTests() method of the Datasource base class.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
	public void runTests()
	{
		final def tc = testRunner.testCase,
			propsStep = tc.getTestStepByName( "Props" ) ?:
			tc.addTestStep( PropertiesStepFactory.PROPERTIES_TYPE, "Props" ),
				testSteps = tc.getTestStepsOfType( RestTestRequestStep.class )
			testSteps.removeAll{ it.disabled }

		// Populate the properties step and run the enabled test steps.
		final def names = testData[ 0 ]
		( 1 ..< testData.size ).each { rowNo ->
			testData[ rowNo ].eachWithIndex { it, i -> propsStep.setPropertyValue( 
				names[ i ], it ) }

			final def logProp = propsStep.getPropertyList()?.getAt( 0 )
			testSteps.each { ts ->
				final def stepName = ts.name
				ts.name += " (${logProp?.name} = ${logProp?.value})"
				final def stepResult = testRunner.runTestStep( ts ),
						  exchanges = stepResult?.getMessageExchanges()

				exchanges?.each { exchange ->
					exchange.getResponse()?.with {
						Calendar.instance.with {
							setTimeInMillis( getTimestamp() )
							final def timeStamp = format( 'dd-MMM-yyyy HH:mm:ss.SSS' ) 
							this.datasinkLines += "$timeStamp: [${ts.name}] " +
								"${getMethod()} ${getURL()} " + 
								"Status=${getStatusCode()} (${getTimeTaken()}ms)"
						}
					}
				}
				sleep( 20 )
				ts.name = stepName
			
				// Reference any failed assertions in our results datasink.
				if( stepResult?.getStatus() != TestStepResult.TestStepStatus.OK ) {
					ts.getAssertionList()?.each {
						it.failed ? datasinkLines += 
							"\t[${it.name}] FAIL: ${it.getErrors()}" : null
					}
				}
			}
			datasinkLines += '' // Leave a blank line between each run of the test case.
		}

		testRunner.gotoStepByName( propsStep.name )
	}

This script debunks the misconception that the messageExchange variable is available only in script assertions. In fact, whenever the testRunner runs a test step, it returns a TestResult object from which the messageExchange list can be obtained. In turn, each exchange contains the HTTP response, and you can see from the processing above that the response includes most of the items we need for our datasink entry, being the timestamp, HTTP method, URL, status code and response time. We already have the name of the test step at our disposal and we can use Groovy’s Calendar object to convert the raw timestamp value in the response from milliseconds into a human-readable format.

Having collected the information relating to the test step call, the runTests() method then consults the list of assertions if the test step failed. For the assertions that failed (which may be only some of the list of assertions on a test step) the code creates an entry in the datasink.

Finally, the code adds a blank line to the datasink to help distinguish between successive runs of the test case. You’ll see that I have stored all of the datasink information in a list variable called datasinkLines which is a member of the Datasource class. I’ve elevated this to the class level rather than keeping it local to the runTests() method so that the actual writing of the datasink file can take place in the tearDown() method. This is a common use of a tearDown function.

When I initially implemented this feature, I used the Setup and Teardown scripts as part of the test case, storing the content of the datasinkLines as a member of the test case context variable. While there’s nothing wrong with this approach, I always feel a little perturbed at having logic in the Setup and Teardown scripts. This is partly because the SoapUI user interface doesn’t indicate that there’s a script hiding in there unless you happen to open the script inspector. If you copy test cases around, you can quite easily forget that there’s a script waiting to be executed. This can lead to some quite unexpected processing.

The second reason I moved the implementation of the datasink into the Datasource class is that the class already provides a tearDown() method, so it’s probably best to use it, rather than to divide the processing between the class and the test case.

Finally, if the processing is provided by the class then any test case can use it without needing to duplicate the Teardown script.

Writing the datasink

Having collected the loging information as the test runs, we need to create the log file. Let’s take a look at how the tearDown() method does this.

1
2
3
4
5
6
7
8
9
	public void tearDown() {
		final def fileName = context.getTestCase()?.getPropertyValue( "datasinkFile" )
		if( fileName ) {
			fileName = context.expand( '${projectDir}' ) + "/" + fileName
			new File( fileName ).withWriter{ dataSink ->
				datasinkLines.each{ dataSink.println it }
			}
		}
	}

The method first consults the test case for a property called datasinkFile. Apart from allowing us to specify the name of the file, this allows us to turn the datasink on or off without modifying the script’s code. The processing won’t proceed if no property called datasinkFile is found. The method then opens the datasink file and writes the datasink lines to it via a withWriter closure, creating the file in a single operation.

It’s probably worth summarising the layout of the JSONDatasourceFull.groovy file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
///////////////////////////////////////////////////////////////////////////////
//	Datasource driver - an opensource driver implementation.
//	Read the properties provided in JSON format.
//

import groovy.transform.InheritConstructors
import com.eviware.soapui.impl.wsdl.teststeps.*
import com.eviware.soapui.impl.wsdl.teststeps.registry.PropertiesStepFactory
import com.eviware.soapui.model.testsuite.*
import groovy.json.JsonSlurper

@InheritConstructors
class JSONDatasource extends Datasource
{
	public boolean initialise()
	{
		.
		.	As above
		.
	}
}

abstract class Datasource
{
	Datasource( log, testRunner, context )
	{
		testRunner 	= testRunner
		context	= context
		log		= log
		testData	= []
		datasinkLines = []

		fileName	= context.expand( '${projectDir}' ) + '/' + 
			testRunner.testCase.getPropertyValue( "PropertiesFile" )
	}

	abstract boolean initialise()

	public void runTests()
	{
		.
		.	As above
		.
	}

	public void tearDown() 
	{
		.
		.	As above
		.
	}

	protected testRunner
	protected context
	protected log 
	protected testData
	protected fileName
	protected datasinkLines
}

At this point you should have a fully-implemented datasink. To use it, create a property on the test case called datasinkFile and provide it with the name of the datasink file you want to create. It’s probably wise to name the file with a *.txt extension, like usersLog.txt.

When you execute the test case, you should see a new file created in the SoapUI project directory with contents like the following:

01-Jan-2017 23:38:28.206: [User Options (id = 1) (id = 1)] OPTIONS http://jsonplaceholder.typicode.com/users/ Status=204 (798ms)
01-Jan-2017 23:38:28.858: [Post User (id = 1)] POST http://jsonplaceholder.typicode.com/users/ Status=201 (529ms)
01-Jan-2017 23:38:29.743: [Delete User (id = 1)] DELETE http://jsonplaceholder.typicode.com/users/1 Status=200 (566ms)

01-Jan-2017 23:38:30.362: [User Options (id = 1) (id = 4)] OPTIONS http://jsonplaceholder.typicode.com/users/ Status=204 (580ms)
01-Jan-2017 23:38:30.875: [Post User (id = 4)] POST http://jsonplaceholder.typicode.com/users/ Status=201 (483ms)
01-Jan-2017 23:38:32.008: [Delete User (id = 4)] DELETE http://jsonplaceholder.typicode.com/users/4 Status=200 (927ms)

01-Jan-2017 23:38:32.753: [User Options (id = 1) (id = 7)] OPTIONS http://jsonplaceholder.typicode.com/users/ Status=204 (698ms)
01-Jan-2017 23:38:33.251: [Post User (id = 7)] POST http://jsonplaceholder.typicode.com/users/ Status=201 (466ms)
01-Jan-2017 23:38:34.051: [Delete User (id = 7)] DELETE http://jsonplaceholder.typicode.com/users/7 Status=200 (584ms)

01-Jan-2017 23:38:34.544: [User Options (id = 1) (id = 10)] OPTIONS http://jsonplaceholder.typicode.com/users/ Status=204 (460ms)
01-Jan-2017 23:38:35.150: [Post User (id = 10)] POST http://jsonplaceholder.typicode.com/users/ Status=201 (566ms)
01-Jan-2017 23:38:35.882: [Delete User (id = 10)] DELETE http://jsonplaceholder.typicode.com/users/10 Status=200 (531ms)

Compiling classes with Java

In implementing support for JSON data, we’ve extended (and copied)the Datasource class three times. Let’s look at how to derive multiple classes from a single base class via compilation.