Unit Tests

From Daya Bay
Jump to navigation Jump to search

Unit tests are bits of code that test specific aspects of the software in a way that they can be run in concert producing reports. Details on the unit test framework can be found here.

Why write unit tests

They will be run after each commit to make sure nothing breaks and the provide a good source of examples for people trying to learn the code.

How to write unit tests

Location

Tests are written in Python and placed in:

MyPackage/tests/test_someLabel.py

Content

Tests are written as Python functions named like:

def test_something():
    'This tests something about something'
    ...

Initial boiler-plate

A test can fail because of a Python exception or based on matching any patterns in the output. To set up this matching you need the following at the top of your .py file. Let's walk through it:

from dybtest import Matcher, Run, iswritable

checks = { 
         '.*ERROR':1,
         '.*FATAL':2,
         '.*\*\*\* Break \*\*\* segmentation violation':3, 
         '.*IOError':4,
         '^\#\d':None 
         }

Here a set of patterns to check is defined. This is a good start. If your test prints out other notices of failure, you can add patterns that detect the failure as you find them.

Run.parser = Matcher(checks,verbose=False)
Run.opts = { 'maxtime':300 }

That code registered the checks and set a timeout. This timeout can be used to detect infinite loops. Finally, your code may need to write out files as part of the test. If so, you should use the current working directory unless it is not writeable. In that case, use a random directory under /tmp:

if iswritable():
    tmpd = '.'
else:
    from tempfile import mkdtemp
    tmpd = mkdtemp(dir='/tmp')

Now start writing some tests!

Final boiler plate

To allow your test file to run as its own program it is useful to add code at the very end of the file that calls whatever test functions you would like. For example:

if '__main__' == __name__:
    test_check1()
    test_check2()
    ...


Types of test functions

Run a full job test

You can run a full nuwa.py job as a test:

def test_pyloWorld():
    'Test a full job'
    Run("nuwa.py -n 1 DybHelloWorld.PyloWorld")()

An extended test using Python coding

The test_* functions are called in order so you can set up a test file that breaks up actions into parts.

ss = SharedState()
def test_setup():
    'Do any one time setup'
    global ss.init()
    ...

def test_test1():
    'Do first test'
    global ss
    ss.thing1()
    ...

def test_test2():
    'Do second test'
    global ss
    ss.thing2()
    ...

def test_cleanup():
    'Do testing of any cleanup'
    global ss
    ss.final()
    ...


Testing against reference output

In addition to matching against known pathologies like "FATAL" , you can also match against a reference output

def test_refpyloWorld():
    'Test a full job against a reference output'
    Run("nuwa.py -n 1 DybHelloWorld.PyloWorld", reference=True)()


The reference output is saved in a file named after the test, such as "test_refpyloWorld.ref" in the directory from which the test is run. If that directory is not writable a temporary directory is used. To update the reference, simply delete the prior one and rerun the test. Subsequent runs will compare against the reference. If there are changes, the test will fail and a unified diff will be in the error message.

The comparison is made after substituting running dates and memory addresses.

Running tests

With bare Python

You can run the test file directly if you have defined a __main__ section as above:

cd MyPackage/
python tests/test_myPackage.py

With nosetests

The unit test framework provides a program nosetests that gives more options and will let you run the test as it will be done by the auto-build and test system. First, just run a full test:

nosetests tests/test_myPackage.py

Maybe some tests fail, if so you can run specific test functions by naming them:

nosetests tests/test_myPackage.py:test_failing

You can also select which tests to run using nosetest attributes such as

nosetests -A 'minutes < 10'

More Info