The pc_select script addresses the problem of selecting an appropriate subset of tests to run in order to test a limited change made to the code. Most of the time, it is too time consuming to run the entire test suite after each change, so the typical practice is for the user to update the master source code without running any of the test suite. Then, as part of an overnight build process, the full test suite is run and reported.
There are problems with this practice, for example:
If the update introduces severe problems, the rest of the development group may be "stuck" while they wait for a repair.
If several developers have made updates, it may not be clear which code changes caused things to break.
If several developers have introduced bugs, one set of problems may mask the other, causing a series of delays between the initial update of the code and the eventual discovery of problems introduced.
When problems are eventually discovered, the developer may already be involved in other changes, making it inconvenient or difficult to stop and fix those problems.
The pc_select script gives developers a way to run just that subset of the test suite which is necessary to exercise the code changes they have just made. It can greatly reduce the testing time required, making it much more feasible to do the appropriate testing before the code update.
Even though some test harnesses support the selection of an arbitrary subset of the tests to run, they still do not address the issue of determining just which subset of the tests is the appropriate subset to exercise a given set of changes. The pc_select script addresses this problem. It assumes that:
Your test harness supports specification of an arbitrary subset of tests to run.
There is a set of .pcv files available, one per testcase, that might need running; and further that you have some way to determine from the name of a .pcv file which testcase corresponds to it.
You can identify which source files were modified.
There is some way to determine for each source file the full set of diff output identifying the changes made since the coverage data for each test was collected.
When pc_select runs, it searches all the coverage files, and identifies each coverage file that exercises any function that was modified. The resulting list of coverage files is output. These can then be converted into whatever form you need to run just the corresponding tests from the test suite.
%
<changed files> | pc_select [-diff=<rule>] \
[-canonicalize=<rule>] <file>.pcv ...
The <changed files> list piped to pc_select via stdin should consist of filenames, one per line, identifying each file modified since the coverage data was collected. For example, if you have been making the changes for the last 3 days, you can use a find command to produce the correct list, and then pipe the list into the pc_select command:
% find * -type f -mtime -3 -print | pc_select
The list of <.pcv> files on the command line should be the full list of coverage data corresponding to the test suite. For example,
% /usr/builds/951211/tests/*.pcv
The -diff <rule> option is used to specify exactly how to obtain the diff output for each file. It is a generic Perl expression that is applied (via eval) to each file on stdin in turn to obtain the diff output for each file. When the expression is evaluated, the variable $file contains the name as it was found on stdin, and for convenience the variable $_ is also set to the same value.
Since the shell and Perl use some of the same metacharacters, it is very important to apply appropriate quotation marks and escapes to make sure the correct program parses the value.
The default value for the -diff <rule> is
-diff='`rcsdiff $file`'
which says to use rcsdiff on each file to look up the changes in the RCS data. Another example value is
-diff='`diff $file.original $file`'
which says to look for a file foo.original from which the new version of foo is descended, then to use the standard diff program to calculate the changes.
While the script can detect syntax errors in the specification, it cannot detect various other problems, such as reversing the order of the two files and argument errors to the command which cause it to produce no output.
The -canonicalize <rule> is used in the case where the fullpaths in the .pcv files do not match those of the files which were being modified. For example, suppose that the name of the original file.c when it was built for the overnight full test run was:
/usr/builds/951211/src/file.c
while the fullpath in your development area is now:
/usr/home/pat/work/file.c
In this case, the pc_select script looks for coverage information for the file under the latter name, but does not find it because all the information in the .pcv file is under the former name.
The -canonicalize <rule> is a generic Perl expression that is applied (via eval) to each file in turn in order to determine what name to use to locate coverage for it in the .pcv files. When the expression is evaluated, the variable $file contains the name as it was found on stdin, and for convenience the variable $_ is also set to the same value.
The default value of the -canonicalize <rule> is:
-canonicalize='chop(local($here)=`pwd`); "$here/$file"'
This converts the relative pathname to an absolute pathname, using pwd to calculate the fullpath. For the example shown here, if the filename is just the basename of the file, you can use:
-canonicalize='"/usr/builds/951211/src/$file"'
While the script can detect syntax errors in this specification, it cannot detect problems such as incorrect use of automounter prefixes or specification of a fixed value instead of a variable value.