My SoapUI Thoughts on Effective Testing in SoapUI

Data-driven Testing

Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.

Fred Brooks, The Mythical Man-Month


Data-driven testing is a pretty well-known subject. We’ve all heard about the advantanges of keeping your test data in files that you can design and maintain independently of how you execute the tests. With manual testing, you can choose how far you go down the data-driven path. When it comes to testing APIs, your tests are likely to be data-driven whether you intend them to be or not. The question really centres on how effectively you manage the separation between the elements of test data, configuration data, and implementation logic.

One of the main differences between SmartBear Software’s commercial Ready!API software and their open source product SoapUI is the built-in support in Ready!API for data-driven testing. Ready!API includes a number of features that allow you to store and access data, pass the data to a series of SOAP or REST requests, and write the results of the responses into a report. Typically these tests run iteratively over one or more API calls.

The SoapUI community has recognised the value of this style of testing and several online resources have emerged describing how to implement data-driven frameworks in the open source versions of SoapUI. I want to build on this work by describing a framework with the following features:

  • the framework must be independent of the test data
  • the framework must be independent of the API calls
  • the framework must support test data in different formats (CSV text, XML, JSON, Excel)
  • the framework must be maintainable without duplication across potentially hundreds of test cases, test suites, and test projects.

The articles in mysoapui.net offer a number of approaches to scripting that can you can use to implement a data-driven framework in SoapUI. You can select from whichever approach meets your needs.

In preparing these articles, I’ve used SoapUI Open-Source 5.3.0 under Linux and Windows.

About You

I’m going to assume in this article that you are conversant with the following features of SoapUI:

  • Creating and running test cases and test steps
  • Using properties and assertions
  • Basic familiarity with configuring resources and assigning them to test steps
  • Optionally, you may also have an awareness of object-oriented scripting techniques.

Data Formats

I listed as a requirement that the framework will offer support for multiple data formats. Let’s take a look at some of the most common, and identify some of points of comparison between them.

Type Advantages Disadvantages
Text – comma-separated values Compact, easy to edit if the amount of data is small Relies on each row using the columns in the same way
JSON Fairly compact, good support for hierarchical data May not be easy to edit
XML Many tools available for editing Can be verbose
Excel Can format data in a human-readable way Requires spreadsheet software to read and write

These articles will show how to keep the processing of data in these data formats separate from the execution of the test steps in the test cases.

These articles also show how to implement data-driven test cases where the test data is retrieved from a database.

Scripting Techniques

I mentioned that the framework will be maintainable across large numbers of test cases. There are several scripting techniques we can use to achieve this.

Technique Advantages Disadvantages
Procedural scripts stored as a Groovy test step in the SoapUI project file Easily accessible; easy to modify small scripts in single instances. Difficult to maintain when used often
Class-based scripts accessed via the GroovyScriptEngine. The same scripting logic can be accessed from within large numbers of test cases and across different SoapUI projects. Each file can expose only one class. Base classes may need to be duplicated.
Compiled scripts stored in JAR files The same scripting logic can be accessed from within large numbers of test cases Can be used to develop large frameworks in an extensible manner.
Requires knowledge of Java / Groovy compilation techniques.
Requires a re-start of SoapUI when a new JAR file is compiled.

mysoapui.net shows examples of each of these techniques so that you can select the approach that best meets your needs. First, though, we need to create a service definition in order to support our test cases.