Testharness documentation

Overview

The test harness provides a system for scripting regression tests on C functions and source modules containing interacting sets of functions. It consists of two parts:

• runtest.c contains all the code needed to carry out unit testing on one of more C function calls as well as the main() function. It should not need modification because all the code that's required to run every test is part of it. Any auxiliary code needed to transform the standard input parameters, which are booleans, ints, chars or strings, into the data types needed by code being tested and to convert test results into types handled by the output display functions, i.e. setboolreply(), setcharreply(), setdoublereply(), seterror(), setfloatreply(), setintreply() or setstringreply(), should be included in the adapter code derived from custom.c.
  runtest() reads test scripts, parses them and then calls the calladapter() function in custom.c. This contains custom code that carries out the test by calling the function being tested and returning test results to runtest() for output.
• custom.c is a C source module that contains two functions:
  • calladapter(), which is customised to call each function being tested and send test results back to testharness for output to the test results file.
  • showhelp() is called to supply help text to the user if the compiled test harness is run with the '-?' command line option. It should name the shell script used to run individual tests and/or regression tests and show the name(s) of the test scripts containing test cases for the functions under test.
• Once custom.c has been extended to call the functions being tested, its compiled and linked with runtest.c and the function(s) to be tested to produce an executable program capable of reading test scripts and producing easily readable test results.

Unit tests are defined within test scripts. These are plain text files which contain a sequence of function test definitions interspersed with comment lines used to describe tests and their expected results. Blank lines may be added for improved readability. Each test definition is a line containing the name of the function to be executed, a list of parameter values and an optional comment at the end of the line, which is typically used to say whether this test is expected to succeed or fail and to show the expected return value, if any.

• Comments start with a '#' and are terminated by the end of the line.
• Test definitions must start with the name of the function being tested. This is followed by up to 10 parameter values separated from the function name and from each other by at least one whitespace character. An optional comment may follow the last parameter.
• Parameters may be unquoted, in which case they must not include whitespace. Quoted parameters may include whitespace. The quotemarks are either a single quote (') or a double quote (") but the endmark must be the same as the opening quotemark. Inside a quoted parameter the 'other' quotemark looses its property of being an endquote, so both "quoted 'string'" and 'another quoted "string"'
  will be interpreted correctly, while
  "quoted "string"" will be treated as two parameters: 'quoted ' and 'string""'.

This is a short example script:

----------------------------------------------------------------
# re01.txt - clean re_error() test.
# ======== - error with simple text description
#
reporterror re01                # OK
re_error "error text 1"         # "re01: error text 1" expected

#
# No errors expected
#
----------------------------------------------------------------

The first test runs the reporterror() function with the string "re01" as the only parameter. The second test runs the re_error() function with "error text 1" as its only parameter.

The output from running a test script is intended to be an easily readable file, with outputs from each function test call, including error reports, immediately following the line that defined the test.

Regression testing is easily managed by a shell script that can:

• Run individual test scripts with test output sent to stdout. Its a good idea to keep these in a separate directory and give the set of scripts similar names, e.g. data/fname99.txt
• Run one or more test scripts and save the output under a related name, e.g data/fname99.result
• Run a regression test: this runs the complete set of tests, preceeding each test by displaying the test script's name. It compares the output of each test script with the stored result file, e.g. by using the diff utility program, and writes differences to stdout. If regression tests pass, all you see are the test script names while, if any tests produce unexpected results, the differences are in the output after the name of the script that contained the failing test.

Test script processing

Every line in a test script written to an output file via stdout, preceded by a line number. Comment lines may appear anywhere and have a hash, #, as the first character. Comments and blank lines are copied to the output but otherwise ignored.

Each non-comment line defines a function call. This takes the form:

function param1 param2 .... # comment documenting expected results

The function name must start at the first character in the line and parameters are separated by one or more whitespace characters. There may be up to 10 parameters on each line. Quoted parameters may contain whitespace. Unquoted parameters must not contain whitespace.

If a parameter contains whitespace it must be 'quoted' by enclosing it in single or double quotes ( 'param text' or "param text" ). Quoted parameters must be terminated with the same single or double quote mark that they started with, but may contain the other type of quote, so "String val '%s'" and 'Char val "%c"' will both be parsed correctly, but "stringvalue' will include all following parameters until either another double quote or the end of the parameter list is found.

The parameter list may be followed by a comment, starting with a hash, which is a good way of documenting the expected results from a function call.

When the text has been parsed into function name plus a list of parameter strings, a parallel list of integer numeric values is set up. Each parameter is first tested to see if it is a boolean value. If its first character is T,t or 1 its numeric value is set to 1 (TRUE) and if its first character is F,f or 0 its numeric value to 0 (FALSE).

If neither comparison matches, the parameter is used as input to atoi() and the result is used as its numeric value.

The calladapter() function, which will be described later, is now called to execute the required function test. It is passed the function name and both string and integer versions of the parameter list, runs the test and returns the test results by calling one or more of the following functions, which each write an output line to stdout immediately after the script line that defined the test:

The number and type of results shown is determined by the way in which you customise calladapter() - see below.

The number of calls to seterror() are counted and output when the test script has been completely processed.

Note:

• calladapter() should normally return zero: returning anything else signals that a fatal error has ocurred and that the script processing should be abandoned immediately along with any other scripts listed on the test program's command line for processing.
• If the function being tested calls exit() script processing ends at that point.

Customisation

This process will normally only require changes to the custom.c source file. Make a copy of custom.c and custom.h before you start, so you still have the original files available building further test harneses

custom.c provides the interface between the test harness functionality in runtest.c and the C functions being tested. It contains two functions:

• show_help(), which provides user information about the tests to be run, such as script names and any external requirements, apart from the test scripts, which are needed to do the tests. It is a good idea for this text to name any shell script used to run regression tests and to give the name(s) used for the set of test scripts used to exercise the function(s).
• calladapter(), which allows individual function tests to be run by mapping the function name and arguments in the test definition to a function call, executes the function call, and writes test results to stdout by calling the setxxxxreply() functions to write results to the test output and seterror() to count test failures as well as writing the error description to the test output.

Tests are driven by code in runtest.c, which uses command line arguments and options to set run conditions and specify which test scripts will be run. Runtest interprets each test script, using the calladapter() function in this module to exercise the functions being tested, and writes the results of each call to stdout immediately after the script line that defined the test.

int calladapter(int debug, char *fn, char s[MAXARGS][MAXLINE], int n[MAXARGS])

This is a code-specific call adapter which must be modified to call the set of functions to be tested.

Entry and exit is reported if debug level is 2 or more. custom.c is an example implementation that contains the code to run one example function, whose prototype is:

int example(char *s int i);

It is used to show a basic setup which will compile and run successfully. The basic if..then..else chain of function calls should be preserved, but any or all if-actions can be converted into code blocks and/or may call additional privately defined functions.

The functions used to return results, report errors, etc, are defined in runtest.c. They are seterror() and the setxxxxreply() functions.

The final item in the condition chain should not be modified. It is there to trap function names in test scripts that don't match any of the functions being tested.

When calladapter() is called by code in runtest.c:

It is the customiser's responsibility to map the contents of the s[][] and n[] arrays to the parameter list of each function being tested. Any additional code that may be needed to provide the correct mapping can be added as required, e.g. converting s[1] to a char or n[2] to a double.

On exit calladapter() should return 0 if the overall call didn't encounter any fatal errors and a non-zero value if an error was encountered that was serious enough to abandon the run part-way through the test script.

void showhelp()

The text in showhelp() is displayed if the test harness is run with the '-?' option. It should say that this is a customised test harness, describe what functions or modules are being tested and identify the (set of) test scripts it uses, e.g. that their names are all of the form data/mytest99.txt

int example(char *s, int n)

This is the example function called by calladapter() as supplied. As you can see it will write its first parameter to stdout and return the second parameter.

Compilation

This process will normally only require changes to the custom.c source file. Make a copy of custom.c and custom.h and adapt the Makefile to suit. Finger trouble aside, if you don't change any function prototypes or headers a new test harness should build and run pretty much straight away.

Here is the Makefile used to compile a demo test program called custom using the files as they stand.

NOTE: compiling the test harness requires my environ library, which you can download from https://www.libelle-systems.com and should be put in /usr/local/lib.

LIB     = -L/usr/local/lib
INC     = -I/usr/local/include
EXE     = /usr/local/bin
MAN     = /usr/local/man/man1
BIN     = custom
MANP    = README.html

CFLAGS  = $(INC) -c
HEADERS = custom.h ../testargparser.h ../testre.h

all: $(BIN) testargparser.o testre.o

clean:
        rm *.o $(BIN)

custom: custom.o runtest.o
        cc custom.o runtest.o -o custom

custom.o: custom.c
        cc custom.c $(CFLAGS)

testargparser.o: testargparser.c ../argparser.h
        cc testargparser.c $(CFLAGS)

runtest.o: runtest.c runtest.h
        cc runtest.c $(CFLAGS)

testre.o: testre.c ../reporterror.o ../reporterror.h
        cc testre.c $(CFLAGS)