PureTest - Scenario Runner

PureTest Scenario Runner
PureTest 5.2 January 2015	http://www.pureload.com support@pureload.com

Documentation Index

Scenario Runner
Test Frameworks Integration
Scenario Runner Command Line Usage
Scenario Runner Ant Task
Scenario Runner Ant Task Reference
- Parameters
- Example

Scenario Runner

Typically a product life cycle includes several releases. This could be bug releases and releases where new functions have been added. Every release includes time consuming and costly re-test activities. The concept of automatic regression test aims to reduce the cost and time for these tests.

PureTest include a command line utility, Scenario Runner, supporting this by executing scenarios in a PureLoad Configuration (PLC) file. Also included is a a Jakarta Ant task to support integration of regression tests into a build process.

Test Frameworks Integration

The Scenario Runner supports generating test reports using a simple XML format that is compatible with JUnit Report XML format. This format can be read by a continuous integration system that understands Ant's JUnit report XML format. This includes test frameworks, such as Jenkins, Hudson and TestNG.

The XML format is produced, when using the -xml option or xml parameter if using Ant (see below).

Scenario Runner Command Line Usage

The Scenario Runner is started from the command prompt as follows:

% install-home/bin/runner <scenario file>

where install-home is the directory where PureTest is installed.

There are a number of options available to control the output from the runner. Starting the runner without arguments shows a help text with the available options:

Usage:
runner [-v   OFF | ERROR | INFO | DEBUG]   (runner verbose level)
       [-log OFF | ERROR | INFO | DEBUG]   (task log level)
       [-se]                               (stop execution within scenario on error)
       [-xml                               (format result in JUnitReport XML format)
       [-xmlpackage <package>              (JUnitReport XML package name)
       [-xmlsuffix <suffix>                (JUnitReport XML test name suffix)
       [-tcdepth <depth>]                  (Test Case depth, default 0 - each scenario a test case)
       [-r <result file>]                  (save log and result to file)
       <scenario file> ...                 (scenario files to execute)

Default is '-v INFO -log ERROR'.

Options:

-v

Controls the verbose level of the runner itself. Default is INFO.

OFF
No information displayed. Only exit code indicates success or failure.
ERROR
Show only error information and final summary per executed scenarios.
INFO
Show execution information and execution time information as well as final summary per executed scenarios.
DEBUG
Show more detailed information about scenarios executed and complete summay for scenarios and tasks executed.

-log: Controls the log level used for all log output from the tasks being executed. Default is ERROR to only display info when tasks fail.
-se: Flag to control the behavior when an error is encountered. If set, the runner will stop executing if any task fails. The default behavior is to continue execution in case of error.
-xml: Format results in JUnit Report XML format.
-xmlpackage <package>: Use defined <package> as test case package name (default is the plc file name) for XML output.
-xmlsuffix <suffix>: Add <suffix> as suffix for genared test case name for XML output.
-tcdepth <depth>: Test Case depth defines at which depth the nodes in the scenario result will be reported as <testcase>. Default is 0 which means that each scenario is reported as a test case. A depth of 2 would mean that each node at depth 2 in the scenario result is reported as test cases.
-r <result-file>: Save results to specified file.

The default verbose and log levels (-v INFO and -log ERROR) will show execution time for each scenario, information on errors if such occurs and a final should summary if scenario(s) failed or executed ok. Set both -v and -log to OFF to run in completely silent mode.

The rest of the arguments are treated as PLC files.

When the runner exits, it will return an exit status code. Exit status 0 means success, 1 means one or more execution errors and 2 means invalid input or failure to load scenario file.

Note: The runner will skip the scenarios in the file that have any kind of distribution associated with them (if defined by using PureLoad).

Scenario Runner Ant Task

The Jakarta Project refers to the build tool Ant as the "make without make's wrinkles". Ant is becoming the de facto standard in the Java and open-source world. In the examples/ant directory provided with PureTest you a simple example of and build file and PLC files.

The following section assumes general Ant knowledge, but should be (together with the examples) enough to get started using Ant and PureTest

Scenario Runner Ant Task Reference

The Scenario Runner Ant task is a simple ant task for invoking the scenario-runner provided with PureTest to execute scenarios defined in PLC file(s).

A set of PLC files to be executed can be defined using the dir attribute in combination with includes ad excludes attributes. If only one PLC file is to be executed use the file attribute.

Parameters

Attribute	Description	Required
dir	Where to find the PC files.	Yes, one of file or dir
file	Name of a single PLC file.	Yes, one of file or dir
includes	Comma- or space-separated list of patterns of files in dir that should be included. All files are included when omitted.	No
excludes	Comma- or space-separated list of patterns of files in dir that should be excluded. No files are excluded when omitted.	No
resultfile	Path to file where to save results. If not specified results are written to stdout.	No
xml	Set to true to generate results in JUnit Report XML format. Default is a simple text format.	No
xmlpackage	Set to test case classname package to be used. Default is PLC file name.	No
xmlsuffix	Add suffix to test case name.	No
tcdepth	Test Case depth, default 0 - each scenario a test case.	No
verbose	Controls the verbose level of the runner itself. Default is INFO.	No
tasklog	Controls the log level used for all log output from the tasks being executed. Default is ERROR.	No
stoponerror	Stop execution of tasks in executing scenario on errors.	No
failonerror	Stop the buildprocess if execution of scenario fails.	No
classpath	The classpath to use to execute scenario runner.	No
classpathref	the classpath to use to execute scenario runner, given as reference to a path defined elsewhere.	No

In addition to specifying path using the dir and file attribute and nested FileSet element can be used.

Example

Set Classpath

Used class path must include the jar-files in the lib directory of the PureTest distribution. The recommended way is to use a path reference as follows:

<property name="pureload.home" location="/home/janne/puretest-4.0"/>


<path id="pureload.path">
    <pathelement location="${pureload.home}/classes"/>
    <fileset dir="${pureload.home}/lib">
      <include name="**/*.jar"/>
    </fileset>
</path>

where the pureload.home property should be set to where your installation is located.

Define Task

To define the scenario-runner task, you can now do:

<taskdef name="scenario-runner" classpathref="pureload.path"
classname="com.pureload.tools.ant.ScenarioRunnerAntTask">
</taskdef>

Execute a PLC file

Now you can define targets to execute PLC files as follows:

<target name="test1">
      <scenario-runner classpathref="pureload.path"
                       file="plc/test.plc"
                       verbose="INFO" tasklog="ERROR"/>
   </target>

   
   <target name="test2">
      <scenario-runner classpathref="pureload.path"
                       dir="plc" includes="**/*.plc"
                       verbose="INFO" tasklog="ERROR"/>
   </target>

    
    <target name="test3">
        <scenario-runner classpathref="pureload.path"
                         verbose="INFO" tasklog="ERROR"
                         xml="true" resultfile="test3-result.xml">
            <fileset dir="plc" includes="**/*.plc"/>
        </scenario-runner>
    </target>

    
    <target name="test4">
        <scenario-runner classpathref="pureload.path"
                         file="plc/test.plc"
                         verbose="INFO" tasklog="ERROR"
                         xml="true" tcdepth="2" resultfile="test4-result.xml">
        </scenario-runner>
    </target>