filterunit - Unit test a filter program
filterunit common-options [-d diff-options|-r|-W|-L] test-fixture|suite-directory [test]...
filterunit common-options [-d diff-options] [-L] [-t directory]
filterunit common-options [-d diff-options] -f log-file|[test-fixture|suite-directory test]...
filterunit -H
common-options: [-v] [-c command] [-o tag]... [-s tag]...
Runs one or more unit tests on a program performing a filtering function.
Here are some definitions used later.
A program which processes an input file and produces some output.
filterunit is a framework for unit testing filters. By default input is taken from STDIN and output goes to STDOUT.
A test testing one specific feature of a unit.
For filterunit the unit under test is a filter. A unit test is represented by an input file for this filter. The base name of the input file is used as the name of the unit test.
A certain setup used for running a unit test.
For filterunit a test fixture can be configured by a fixture configuration file. See "FIXTURE CONFIGURATION" for details. If no fixture configuration file is given the defaults apply.
A number of unit tests sharing the same test fixture.
For filterunit a test suite is represented by a directory containing two more directories. One directory contains input files while the other contains expected output files. The basenames of the files in both directories must match. A test suite directory may also contain a fixture configuration file.
An execution of one or more unit tests.
In filterunit a test run for a unit test is done by feeding the respective input file to the filter program. The working directory is changed to the directory containing the test suite. The actual output is compared to the expected output. The unit test fails if the actual output is different from the expected output. See "EXECUTION" for details on how the command line is constructed.
The first synopsis is useful for running a certain test suite or even certain tests in a test suite. The test fixture configuration file test-fixture defines the test suite to be run. The test suite directory defaults to the directory containing test-fixture but may be overwritten in test-fixture by using suiteDirectory.
If you give any tests then the unit tests with this basename are run. If no test is given all tests in the suite are run.
test-fixture may be -
in which case STDIN is taken and the test suite directory defaults to the current directory.
If you give suite-directory instead of test-fixture then this must refer to a suite directory. In this case the test suite is run without using a fixture configuration file. This is particularly useful to re-run simple tests which are desgined for use with the second synopsis and need no special fixture configuration.
The second synopsis is useful for running all test suites contained in a directory. The rationale of this synopsis is that you can run all your unit tests with a single command.
For the second synopsis the test suite directories are contained in a certain directory (default: tests). A test fixture configuration file for each test suite may be given by a file named filterunit.cfg in a test suite directory. Directories named CVS and directories with a leading .
are not considered as test suite directories.
The third synopsis is useful for re-running tests which previously failed.
If you give log-file you can give a file containing the output of a previous run of filterunit. The file is searched for the --failed-tests
separator and the remaining lines are considered tab-separated pairs of test fixtures and tests.
The other option is to copy and paste the output of filterunit directly to the command line of another invocation of filterunit. This way you don't need to rerun an expensive test suite just to catch the output of failed tests. Please note that in this case the file and directory names may not contain spaces because these are interpreted by the shell.
After a test run completed for all failed unit tests some output is generated making it easy to repeat the respective tests. For each failed unit test the name of the fixture configuration file of the respective test suite and the name of the unit test is output on a single line separated by a tab. If no fixture configuration file was given the suite directory is output instead.
Each line of this output can be used directly as arguments using the first synopsis to repeat the failed unit tests. The whole output is prefixed by a line containing --failed-tests
so it is easy to distinguish from possible output from the unit tests.
Filter to be tested.
Default: Basename of the current directory for the second synopsis. None for the first synopsis.
For failing unit tests output differences using diff diff-options.
Please note that with parallel execution of unit tests output of multiple diffs may intermix.
Default: Output no differences.
Interpret filterunit log file or accept pairs of fixture configuration and test names.
Run only tests tagged with tag.
May be given more than once. Giving an undefined tag results in a warning.
See "TAGS" for further details.
Default: None.
Run at most count unit tests in parallel.
0
means to use all available processors based on the number of available cores and machine load. This functionality is available only when Sys::CPU and Sys:CpuLoad are available.
1
means to run exactly one unit test at each time.
Default: 0
Instead of running a test create the expected output files for the tests which would be run otherwise.
For each test there must be an input file with the correct basename. If multiple expectedExtensions are given the first one is used for constructing the name of the expected output file.
If errorExtension is used the first one will be used for for constructing the name of the expected error output file. Otherwise error output is not recorded.
If exitExtension is used the first one will be used for for constructing the name of the expected exit code file. Otherwise the exit code is not recorded.
Existing files are not overwritten. Instead a warning is issued.
May be given only with the first synopsis and is mutual exclusive with -W/--write-config and -L/--list-tags.
Skip tests tagged with tag.
May be given more than once. Giving an undefined tag results in a warning.
See "TAGS" for further details.
Default: None.
Directory containing tests.
May be given only with second synopsis.
Default: tests.
Operate verbose.
Please note that with parallel execution of unit tests verbose output and error output of multiple tests may intermix.
=
value=
valueSet configuration attribute attribute to value. attribute may be a user defined attribute. value may contain interpolatable parts. An attribute defined by this option supersedes any value set in the fixture configuration file.
May be given more than once. If attribute is defined more than once the last value is used.
See "Interpolation" for further details.
Default: None.
Instead of running any tests write the configured tags to STDOUT.
In first synopsis each defined tag is listed on a new line followed by the tests tagged by this tag. If one or more tests are given only the tags for these tests are output and only those tests are listed.
In second synopsis each defined tag is listed on a new line followed by a colon and the suite directories defining this tag. This way you can get a quick overview over the tags used in a complex test setup.
See "TAGS" for further details.
May be given with the first and the second synopsis and is mutual exclusive with -r/--record and -W/--write-config.
Instead of running any tests write the resulting, expanded configuration to STDOUT. This may be used for debugging of complex configurations. Interpolation is not done here (see "Interpolation").
May be given only with the first synopsis and is mutual exclusive with -r/--record and -L/--list-tags.
Generate the man page for this program on standard output.
If an unknown option such as -. is given, a short usage message is generated.
The syntax of configuration files is defined by the syntax of Config::General. Relevant options used are -AllowMultiOptions
and -UseApacheInclude
, -IncludeRelative
, -IncludeGlob
, -SplitPolicy => 'equalsign'
. I.e.
attribute = value
Attributes for which this is valid document this.
You can include other configuration files with a simple include statement and you may give them relative and may use globbing. I.e.:
include ../*.cfg
works.
You can give multiline options using here-documents like this
attribute = <<endOfValue
First line
Second line
endOfValue
Some attributes accept blocks. They can be given like this
<attribute>
entry1 = value1
entry2 = value2
</attribute>
In addition there is an interpolation feature implemented by filterunit. (Note: The interpolation of Config::General does not work properly :-( .) Interpolation is done when an attribute is read for use.
Interpolation takes place for every interpolatable part. An interpolatable part looks like this
$<attribute>
or like this
$<attribute[7]>
Interpolation simply replaces the interpolatable part with the value of attribute in the same configuration. attribute is a alphanumeric string including underscore. If like in the second example an index is given attribute must be a list attribute and the value at (0-based) index is interpolated. An index must be a number.
A value is interpolated as long as there are interpolatable parts.
If an interpolation directly or indirectly refers to itself this is an error. If a referenced attribute is a list attribute and there is no index given the list values are joined with a space and interpolated. If a referenced attribute is indexed it must be a list attribute. If the index is out of range this is an error. If a referenced attribute is a block attribute this is an error. If attribute is not defined in the same configuration this is an error.
The syntax can be quoted by doubling the leading $
. I.e.
$$<attribute>
will be replaced by $<attribute>
which will not be interpolated further.
If you really need it it's possible to construct interpolations. I.e.
bar = foo
name = bar
thing = $<$<name>>
results in thing being interpolated into foo
.
Please note that the interpolation syntax is chosen to not collide with the various shell syntaxes for interpolation.
A fixture configuration is a configuration file. See "CONFIGURATIONS" for details on the syntax.
The following attributes are defined.
An extension which is used by files containing additional arguments to be used for a unit test. A leading dot must be specified. If this configuration is given an argument file for all unit tests defined by input files is required.
This attribute can be given any number of times to define multiple argument extensions.
The basenames of all argument files must be unique.
See "EXECUTION" for further details.
Default: Unset.
Fixed arguments to give to the filter command. Arguments are given to the shell without any additional quoting.
This attribute may be given more than once.
See "EXECUTION" for further details.
Default: Unset.
The command to be used as the filter program.
Default: Value of --command option.
An extension which is used by all files containing expected error output. A leading dot must be specified.
This attribute can be given any number of times to define multiple expected error extensions.
The basenames of all expected error files must be unique.
Unless this attribute is defined with a non-empty extension output on STDERR is not considered. Please note that with parallel execution of unit tests in this case error output of multiple tests may intermix.
If this attribute is defined with at least one non-empty extension, STDERR is caught and each basename of an input file must match exactly one basename of an expected error file.
Default: Unset.
A command which is used to post process the error output. It receives error output on STDIN and writes the post processed result to STDOUT. This may be used to canonicalize error output.
This attribute can be given any number of times to define multiple post filter commands. They are applied in the sequence given in the fixture configuration.
The strings given are subject to "Interpolation".
Default: Unset.
An extension which is used by all files containing expected exit codes. A leading dot must be specified.
This attribute can be given any number of times to define multiple exit extensions.
The basenames of all expected exit code files must be unique.
Unless this attribute is defined with a non-empty extension the exit code of the filter is not considered.
If this attribute is defined with at least one non-empty extension, the exit code of the filter is caught and each basename of an input file must match exactly one basename of an exit code file.
An exit code file must consist of exactly an integer. Many operating systems allow exit codes only between 0 and 255.
Default: Unset.
The directory containing all files containing expected output. The directory is relative to the test suite directory.
Default: expected.
An extension which is used by all files containing expected output. A leading dot must be specified.
This attribute can be given any number of times to define multiple expected extensions.
The basenames of all expected output files must be unique.
Each basename of an input file must match exactly one basename of an expected file.
If at least one of errorExtension, exitExtension or expectedTreeExtension is given this attribute is mandatory.
Default: Empty string.
A command which is used to post process the expected output. It receives expected output on STDIN and writes the post processed result to STDOUT. This may be used to canonicalize expected output.
This attribute can be given any number of times to define multiple post filter commands. They are applied in the sequence given in the fixture configuration.
The strings given are subject to "Interpolation".
Default: Unset.
An extension which is used by all directories containing an expected tree generated by the filter command. A leading dot must be specified.
This attribute can be given any number of times to define multiple expected tree extensions.
The basenames of all expected trees must be unique.
Each basename of an input file must match exactly one basename of an expected tree.
See "EXECUTION" for further details.
Default: Unset.
Number of input files to use for each unit test. The number of input files for each unit test matching an inputExtension must be equal to this value.
If this value does not equal 1
STDIN is not used. Instead the options and arguments configured must list all input files.
If this value is 0
no input files are used at all. If inputExtension is given the matching files are just marker files giving the basenames of the tests and their content is not used. If no inputExtension is given then there need to be at least one of optionExtension, argumentExtension or inputTreeExtension for determining the basename of the tests.
Default: 1
.
The directory containing all input files. The directory is relative to the test suite directory.
Default: input.
An extension which is used by all input files. A leading dot must be specified.
This attribute can be given any number of times to define multiple input extensions.
For each basename there must be exactly inputCount input files. See inputCount for further information.
If any of optionExtension, argumentExtension or inputTreeExtension is given and inputCount does not equal 0
this attribute is mandatory.
Default: Unset if inputCount equals 0
, empty string otherwise.
An extension which is used by all directories containing an input tree to be used by the filter command. A leading dot must be specified.
This attribute can be given any number of times to define multiple input tree extensions.
The basenames of all input trees must be unique.
See "EXECUTION" for further details.
Default: Unset.
An extension which is used by all files in an expected tree containing meta information. A leading dot must be specified.
This attribute can be given any number of times to define multiple meta extensions.
See "META FILES" for further details.
Default: .meta
.
An extension which is used by files containing additional options to be used for a unit test. A leading dot must be specified. If this configuration is given an option file for all unit tests defined by input files is required.
This attribute can be given any number of times to define multiple option extensions.
The basenames of all option files must be unique.
See "EXECUTION" for further details.
Default: Unset.
Fixed options to give to the filter command.
This attribute may be given more than once.
See "EXECUTION" for further details.
Default: Unset.
Number of parallel unit tests to run for this fixture. 0
means to use the effective value of the --parallel option.
Default: 0
A directory to prepend before environment variable PATH. This can be used for mock commands used in the tested command. Also useful for complicated filters and setUp / tearDown commands.
Relative paths are taken to be relative to suiteDirectory.
This attribute may be given more than once. The last value given ends up as the first entry in PATH.
Default:
../..
This is useful if the command under test resides two directories above the fixture directory.
bin
This is useful as a default for putting mock or other commands.
This option indicates that command does not take an argument naming the wanted output file, but instead writes it's output to a file relative to the directory of the input file. The value of this option gives the path relative to the inputDirectory. The output file is expected to have the same basename as the input file and one of expectedExtension. The output file is cleaned up after use.
Default: Unset.
Use of this option is deprecated. Use inputTreeExtension / expectedTreeExtension instead.
A command which is run after the execution environment has been set up but before the test is run. Output of this command is not treated specially.
This attribute can be given any number of times to define multiple setup commands. They are applied in the sequence given in the fixture configuration.
The strings given are subject to "Interpolation".
Default: Unset.
The directory containing the test suite. If a relative path is given it is relative to the fixtureDirectory.
Default: .
A block containing configurations associating a test with tags. Each configuration associates the test named as the keyword with the tags given as a whitespace separated list of values.
You may give attribute blocks of this type any number of times. If you give multiple configurations for one test these are concatenated.
See "TAGS" for further details.
Default: Unset
A block containing configurations associating a tag with tests. Each configuration associates the tag named as the keyword with the tests given as a whitespace separated list of values.
You may give attribute blocks of this type any number of times. If you give multiple configurations for one tag these are concatenated.
See "TAGS" for further details.
Default: Unset
A command which is run after the comparison is done but before the execution environment is destroyed. Output of this command is not treated specially.
This attribute can be given any number of times to define multiple tear down commands. They are applied in the sequence given in the fixture configuration.
The strings given are subject to "Interpolation".
Default: Unset.
The defaults are set before any user supplied configuration is read. Thus the user supplied configuration can reference defaults and may override them.
The following attributes are set automatically. They may not be set in a fixture configuration but they are useful for interpolation.
The directory of the fixture configuration. This is the directory where the explicit fixture configuration file is located, the current directory if STDIN is used for the fixture configuration or the directory where an implicit fixture configuration is assumed. This is always an absolute path.
Tests can be associated with tags. The association is done using the configuration attributes tagWith and/or tagTest. tagWith associates tests with tags while tagTest associates tags with tests. This way you can use the direction of association most convinient to you.
Both configuration attributes are blocks in the sense of "BLOCKS" in Config::General. You may use the configuration attributes any number of times and you may mix them arbitrarily. All configuration attributes are combined internally and you may check the result by using the option -L/--list-tags.
Here is an example. The configuration
<tagWith>
tag1 = test1
tag1 = test2
tag2 = test1 test3
</tagWith>
<tagTest>
test1 = tag3
test2 = tag1 tag2
test4 = tag1
</tagWith>
<tagWith>
tagA = testA
tagB = testB
tag1 = test4
</tagWith>
results in these tags
tag1 test1 test2 test4
tag2 test1 test2 test3
tag3 test1
tagA testA
tagB testB
You may use tags for skipping certain tests from a bigger suite or to limit the tests run to certain tests. Use -s/--skip=skip-tag for skipping all tests tagged with skip-tag. Use -o/--only=only-tag to execute only tests tagged with only-tag. If you use both options first the tests tagged with an only-tag are selected and the tests with a skip-tag are skipped. Unless you use one of these options tagging is ignored. If you give tests on the command line then these are the tests the options operate on.
Useful tags include
Tag tests which run for a long time with long. This way you can use -s/--skip=long for the casual test during development.
Tag tests which you are currently working on with inProgress. This way you can quickly check your progress by using -o/--only=inProgress.
Tag tests which are known to not run for some reason with off and run all tests with -s/--skip=off.
For each execution of a unit test a command line is constructed and executed by giving a whitespace seperated string to Perl's exec. Perl's exec executes the command line using sh -c when there are shell meta-characters or directly otherwise.
The command line is constructed like this:
command [options]... [optionExtension]... [arguments]... [argumentExtension]...
For command the respective configuration attribute is used. options and arguments are the respective configuration attribute, too. For optionExtension and argumentExtension the content of the respective option and argument files, respectively, are included. All values are concatenated using a single space and no quoting is done.
Multiple lines are supported in option and argument files but you have to care for the necessary quoting.
After the command has been executed expected output and error output is processed by expectedFilter and errorFiter, respectively.
The complete command line as well as the post filters and the setup and tear down commands are subject to interpolation (see "Interpolation"). In particular this means that interpolatable parts in argument and option files are considered.
Normally the command is executed in suiteDirectory. If expectedTreeExtension and/or inputTreeExtension is given the command is executed in a temporary directory. If inputTreeExtension is given the matching input tree is copied to this temporary directory, otherwise this input directory is empty. The command may create files and/or modify this tree. Redirection of STDIN, STDOUT and STDERR is not affected by this but using relativeOutput may have unexpected results. workingDirectory is always set to the working directory.
After setting up the execution environment the command setUpCommand from the configuration is executed. After running the test and after the comparison but before destroying the execution environment the command tearDownCommand is run.
The signals INT and TERM are caught by filterunit. They are propagated to the running tests, interrupt execution of the current suite and prevent further processing of tests. The results so far are perserved, however. This makes is possible that you hit CTRL-C in the middle of an execution and still get the results so far.
Here is the complete execution sequence:
All execution environments created for a fixture are destroyed after the complete fixture ran.
The following automatic attributes are defined for each execution in addition to the configuration attributes from the fixture configuration.
A list of inputCount names of input files for the current execution. These are always entries in inputDirectory. They are given as absolute paths if workingDirectory is not suiteDirectory. The rationale is that you can always simply use inputFile. The sort order is determined by the sequence of inputExtensions.
If inputFile is used STDIN is not used and the filter must get the input files from the command line.
$<
is a shortcut notation for $<inputFile>
and $
index<
is a shortcut notation for $<inputFile[
index - 1]>
where index is a 1-based number. The shortcut notation may not be followed by an alphanumeric character.
The output file for the current execution. This is an absolute path to a temporary file.
If outputFile is used STDOUT is not considered and the filter must put output data to the file given on the command line.
$>
is a shortcut notation for $<outputFile>
unless followed by an alphanumeric character.
The name of the test which runs currently.
The working directory for the current execution. This is an absolute path to a temporary directory or to suiteDirectory.
Meta files are a way to describe expected meta features about an entry in a directory of an expected tree. All features of an entry besides the contents of a file are meta features in this sense.
A meta file describes the features of the entry with the same name with metaExtension stripped. The meta file itself is ignored when comparing an expected tree with an actual result tree.
A meta file is a configuration file. See "CONFIGURATIONS" for details.
The following attributes are defined independent of the type of the file system object.
How to compare the content of the expected file system object with the actual file system object.
binary
Like default
for type file
but a failing comparison does not create a diff output. Valid only for type file
.
default
Comparison is done in the default way. For the various types this means:
file
Content is compared byte by byte. A failing comparison may create a diff output.
directory
All entries are compared recursively.
link
The target of the link is compared.
Same as ignore
.
ignore
Content is not compared.
Default: default
if the file system object does exist in the expected tree or ignore
otherwise.
The state of the timestamp of the file system object. The following values are supported:
touched
The file system object has been modified during the test.
untouched
The file system object has not been modified during the test.
Default: Unset
The type of the file system object. The following types are supported:
file
A normal file.
directory
A directory.
link
A symbolic link.
pipe
A pipe.
socket
A socket.
block
A block special file.
character
A character special file.
ignore
This object must be ignored. I.e. it may or may not exist.
deleted
This object may not exist.
This attribute is mandatory.
Each type of file system object may have own attributes. These are described in the following sections.
The size of the file in bytes.
Default: Unset.
The number of non-trivial entries. I.e. without .
and ..
.
Default: Unset.
The target of the symbolic link.
Default: Unset.
The type of the target of the symbolic link. May be one of the values for type or unknown
for a symbolic link pointing to a non-existant target.
Default: Unset.
Everything worked and all unit tests succeeded (green condition).
One or more unit tests failed (red condition).
A problem occured.
A test has been killed.
A file defining the test fixture for a test suite. Automatically used only with the second synopsis.
The following picture shows an example directory layout.
+-+ my_filter/ # The home directory of my_filter
| +-- my_filter* # The executable filter under test
| +-+ tests/ # Directory containing test suites
| | +-+ simple/ # Directory for the simple test suite
| | | +-+ input/ # Directory for the input files
| | | | +-- verySimple # Input for a very simple test
| | | | +-- quiteSimple # Input for a quite simple test
| | | +-+ expected/ # Directory with expected results
| | | | +-- verySimple # Expected output for a very simple test
| | | | +-- quiteSimple # Expected output for a quite simple test
| | +-+ soph/ # Directory for the sophisticated test suite
| | | +-- filterunit.cfg # Test fixture configuration file defining
| | | | # the extensions used
| | | +-+ input/ # Directory for the input files
| | | | +-- quiteSoph.in # Input for a quite sophisticated test
| | | | +-- verySoph.in # Input for a very sophisticated test
| | | +-+ expected/ # Directory with expected results
| | | | +-- quiteSoph.out # Expected standard output for a quite
| | | | | # sophisticated test
| | | | +-- quiteSoph.err # Expected error output for a quite
| | | | | # sophisticated test
| | | | +-- verySoph.out # Expected standard output for a very
| | | | | # sophisticated test
| | | | +-- verySoph.err # Expected error output for a very
| | | | # sophisticated test
This is the contents of my_filter/tests/soph/filterunit.cfg:
command = ../../my_filter
inputExtension = .in
expectedExtension = .out
errorExtension = .err
With this setup you can do the following things:
In my_filter you can run
filterunit
to run all your tests. If you want to watch the progress add -v.
Use
filterunit my_filter/tests/simple
to run all your simple tests.
In my_filter you can run
filterunit tests/soph/filterunit.cfg verySoph
to run the very sophisticated test.
To (re-)create the expected output files for the quite sophisticated test run
filterunit --record tests/soph/filterunit.cfg quiteSoph
in my_filter. Before this you need to delete any existing expected output files you do no longer want.
Assume some tests fail. To find out where the expected output differs from the actual output use
filterunit --diff=-u
Running all tests may take too much time for quick development-test cycles. Log the failures to a file fails.log by
filterunit > fails.log
Now you fixed some bugs. To rerun only the formerly failed tests use
filterunit --failed-tests fails.log
Non-standard Perl packages needed:
Stefan Merten (fileunit at merten-home de)
This program is licensed under the terms of the GPL. See http://www.gnu.org/licenses/gpl.txt
See http://www.merten-home.de/FreeSoftware/filterunit/
Hey! The above document had some coding errors, which are explained below:
Expected text after =item, not a number
Expected text after =item, not a number
Expected text after =item, not a number