filterunit - Unit test a filter program
filterunit [-v] [-c command] [-d diff-options|-r|-W] test-fixture|suite-directory [test]...
filterunit [-v] [-c command] [-d diff-options] [-t directory]
filterunit [-v] [-c command] [-d diff-options] -f log-file|[test-fixture|suite-directory test]...
filterunit -H
Runs one or more unit tests on a program performing a filtering function.
Here are some definitions used later.
A program which processes an input file and produces some output.
filterunit is a framework for unit testing filters. By default input is taken from STDIN and output goes to STDOUT.
A test testing one specific feature of a unit.
For filterunit the unit under test is a filter. A unit test is represented by an input file for this filter. The base name of the input file is used as the name of the unit test.
A certain setup used for running a unit test.
For filterunit a test fixture can be configured by a fixture configuration file. See FIXTURE CONFIGURATION for details. If no fixture configuration file is given the defaults apply.
A number of unit tests sharing the same test fixture.
For filterunit a test suite is represented by a directory containing two more directories. One directory contains input files while the other contains expected output files. The basenames of the files in both directories must match. A test suite directory may also contain a fixture configuration file.
An execution of one or more unit tests.
In filterunit a test run for a unit test is done by feeding the respective input file to the filter program. The working directory is changed to the directory containing the test suite. The actual output is compared to the expected output. The unit test fails if the actual output is different from the expected output.
The first synopsis is useful for running a certain test suite or even certain tests in a test suite. The test fixture configuration file test-fixture defines the test suite to be run. The test suite directory defaults to the directory containing test-fixture but may be overwritten in test-fixture by using suiteDirectory.
If you give any tests then the unit tests with this basename are run. If no test is given all tests in the suite are run.
test-fixture may be - in which case STDIN is taken and the test suite
directory defaults to the current directory.
If you give suite-directory instead of test-fixture then this must refer to a suite directory. In this case the test suite is run without using a fixture configuration file. This is particularly useful to re-run simple tests which are desgined for use with the second synopsis and need no special fixture configuration.
The second synopsis is useful for running all test suites contained in a directory. The rationale of this synopsis is that you can run all your unit tests with a single command.
For the second synopsis the test suite directories are contained in a certain
directory (default: tests). A test fixture configuration file for each test
suite may be given by a file named filterunit.cfg in a test suite directory.
Directories named CVS and directories with a leading . are not considered
as test suite directories.
The third synopsis is useful for re-running tests which previously failed.
If you give log-file you can give a file containing the output of a previous
run of filterunit. The file is searched for the --failed-tests separator
and the remaining lines are considered tab-separated pairs of test fixtures and
tests.
The other option is to copy and paste the output of filterunit directly to the command line of another invocation of filterunit. This way you don't need to rerun an expensive test suite just to catch the output of failed tests. Please note that in this case the file and directory names may not contain spaces because these are interpreted by the shell.
After a test run completed for all failed unit tests some output is generated making it easy to repeat the respective tests. For each failed unit test the name of the fixture configuration file of the respective test suite and the name of the unit test is output on a single line separated by a tab. If no fixture configuration file was given the suite directory is output instead.
Each line of this output can be used directly as arguments using the first
synopsis to repeat the failed unit tests. The whole output is prefixed by a
line containing --failed-tests so it is easy to distinguish from possible
output from the unit tests.
Filter to be tested.
Default: Basename of the current directory for the second synopsis. None for the first synopsis.
Directory containing tests.
May be given only with second synopsis.
Default: tests.
For failing unit tests output differences using diff diff-options.
Default: Output no differences.
Interpret filterunit log file or accept pairs of fixture configuration and test names.
Instead of running a test create the expected output files for the tests which would be run otherwise.
For each test there must be an input file with the correct basename. If multiple expectedExtensions are given the first one is used for constructing the name of the expected output file.
If errorExtension is used the first one will be used for for constructing the name of the expected error output file. Otherwise error output is not recorded.
If exitExtension is used the first one will be used for for constructing the name of the expected exit code file. Otherwise the exit code is not recorded.
Existing files are not overwritten. Instead a warning is issued.
May be given only with the first synopsis and is mutual exclusive with -W/--write-config.
Instead of running any tests write the resulting, expanded configuration to STDOUT. This may be used for debugging of complex configurations.
May be given only with the first synopsis and is mutual exclusive with -r/--record.
Operate verbose.
Generate the man page for this program on standard output.
If an unknown option such as -. is given, a short usage message is generated.
The syntax of test fixture configuration files is defined by the syntax of
the Config::General manpage. Relevant options used are -AllowMultiOptions and
-UseApacheInclude, -IncludeRelative, -IncludeGlob, -SplitPolicy =>
'equalsign'. I.e.
attribute = valueAttributes for which this is valid document this.
Note: This differs from versions prior to V0.6, where multiple values for an attribute have been given by an array notation. This syntax is still supported but deprecated. A warning is given when it is used.
You can include other configuration files with a simple include statement and you may give them relative and may use globbing. I.e.:
include ../*.cfg
works.
You can give multiline options using here-documents like this
attribute = <<endOfValue
First line
Second line
endOfValue
The following attributes are defined.
The command to be used as the filter program.
Default: Value of --command option.
An extension which is used by all files containing expected error output. A leading dot must be specified.
This attribute can be given any number of times to define multiple expected error extensions.
The basenames of all expected error files must be unique.
Unless this attribute is defined with a non-empty extension output on STDERR is not considered.
If this attribute is defined with at least one non-empty extension, STDERR is caught and each basename of an input file must match exactly one basename of an expected error file. The attribute expectedExtension is mandatory in this case.
Default: Unset.
An extension which is used by all files containing expected exit codes. A leading dot must be specified.
This attribute can be given any number of times to define multiple exit extensions.
The basenames of all expected exit code files must be unique.
Unless this attribute is defined with a non-empty extension the exit code of the filter is not considered.
If this attribute is defined with at least one non-empty extension, the exit code of the filter is caught and each basename of an input file must match exactly one basename of an exit code file. The attribute expectedExtension is mandatory in this case.
An exit code file must consist of exactly an integer. Many operating systems allow exit codes only between 0 and 255.
Default: Unset.
The directory containing all files containing expected output. The directory is relative to the test suite directory.
Default: expected.
An extension which is used by all files containing expected output. A leading dot must be specified.
This attribute can be given any number of times to define multiple expected extensions.
The basenames of all expected output files must be unique.
Each basename of an input file must match exactly one basename of an expected file.
Default: Empty string.
Number of input files to use for each unit test. The number of input files for each unit test matching an inputExtension must be equal to this value.
If this value does not equal 1 STDIN is not used. Instead the options
configured must list all input files.
If this value is 0 no input files are used at all. In this case the files
matching an inputExtension are just marker files and their content is not
used.
Default: 1.
The directory containing all input files. The directory is relative to the test suite directory.
Default: input.
An extension which is used by all input files. A leading dot must be specified.
This attribute can be given any number of times to define multiple input extensions.
For each basename there must be exactly inputCount input files or exactly 1
input file determining the basename if inputCount is 0.
Default: Empty string.
An extension which is used by files containing additional options to be used for a unit test. A leading dot must be specified. If this configuration is given an option file for all unit tests defined by input files is required.
This attribute can be given any number of times to define multiple option extensions.
The content of the additional options file is concatenated to the options from the options variable and treated like these. Multiple lines are supported but you have to care for the necessary quoting.
The basenames of all option files must be unique.
Default: Unset.
Fixed options and arguments to give to the filter command. Options are given to the shell without any additional quoting.
Any $< is replaced by all input files. Any $number< is
replaced by the input file number (1-based) where the sequence of
inputExtensions gives the sort order. If such a replacement is given STDIN
is not used and the filter must get the input files from the command line.
Any $> is replaced by the output file. In this case STDOUT is not
considered and the filter must put output data to the file given on the command
line.
This attribute may be given more than once. All values are concatenated using a single space. This may be useful when file inclusion is used.
Options from an additional options file are concatenated to options given by this configuration the same way and after the options given by the configuration file.
Default: Unset.
This option indicates that command does not take an argument naming the wanted output file, but instead writes it's output to a file relative to the directory of the input file. The value of this option gives the path relative to the inputDirectory. The output file is expected to have the same basename as the input file and one of expectedExtension. The output file is cleaned up after use.
Default: Unset.
The directory containing the test suite. If a relative path is given it is relative to the directory containing the test fixture configuration file.
Default: .
The defaults are set before any user supplied configuration is read. Thus the user supplied configuration can reference defaults and may override them.
Everything worked and all unit tests succeeded (green condition).
One or more unit tests failed (red condition).
A problem occured.
A file defining the test fixture for a test suite. Automatically used only with the second synopsis.
The following picture shows an example directory layout.
+-+ my_filter/ # The home directory of my_filter | +-- my_filter* # The executable filter under test | +-+ tests/ # Directory containing test suites | | +-+ simple/ # Directory for the simple test suite | | | +-+ input/ # Directory for the input files | | | | +-- verySimple # Input for a very simple test | | | | +-- quiteSimple # Input for a quite simple test | | | +-+ expected/ # Directory with expected results | | | | +-- verySimple # Expected output for a very simple test | | | | +-- quiteSimple # Expected output for a quite simple test | | +-+ soph/ # Directory for the sophisticated test suite | | | +-- filterunit.cfg # Test fixture configuration file defining | | | | # the extensions used | | | +-+ input/ # Directory for the input files | | | | +-- quiteSoph.in # Input for a quite sophisticated test | | | | +-- verySoph.in # Input for a very sophisticated test | | | +-+ expected/ # Directory with expected results | | | | +-- quiteSoph.out # Expected standard output for a quite | | | | | # sophisticated test | | | | +-- quiteSoph.err # Expected error output for a quite | | | | | # sophisticated test | | | | +-- verySoph.out # Expected standard output for a very | | | | | # sophisticated test | | | | +-- verySoph.err # Expected error output for a very | | | | # sophisticated test
This is the contents of my_filter/tests/soph/filterunit.cfg:
command = ../../my_filter inputExtension = .in expectedExtension = .out errorExtension = .err
With this setup you can do the following things:
In my_filter you can run
filterunit
to run all your tests. If you want to watch the progress add -v.
Use
filterunit my_filter/tests/simple
to run all your simple tests.
In my_filter you can run
filterunit tests/soph/filterunit.cfg verySoph
to run the very sophisticated test.
To (re-)create the expected output files for the quite sophisticated test run
filterunit --record tests/soph/filterunit.cfg quiteSoph
in my_filter. Before this you need to delete any existing expected output files you do no longer want.
Assume some tests fail. To find out where the expected output differs from the actual output use
filterunit --diff=-u
Running all tests may take too much time for quick development-test cycles. Log the failures to a file fails.log by
filterunit > fails.log
Now you fixed some bugs. To rerun only the formerly failed tests use
filterunit --failed-tests fails.log
Non-standard Perl packages needed:
Stefan Merten <smerten@oekonux.de>
This program is licensed under the terms of the GPL. See http://www.gnu.org/licenses/gpl.txt