You can use PureCoverage to collect coverage data and distribute reports automatically.
To build your applications with PureCoverage, you can add just one word, purecov, in front of the link line in your makefile:
a.out: hello_world.c
purecov cc –g hello_world.c
You can also create separate targets for instrumented and non-instrumented executables by adding a few lines to the makefile. The sample makefile hello_world.Makefile.simple, located in the <purecovhome>/example directory, illustrates how to do this:
#
Sample makefile template
# May
also be
used in instrumenting Hello World with
PureCoverage
# Use make -f hello_world.Makefile.simple a.out.pure
a.out: hello_world.c
cc
-g -o a.out hello_world.c
a.out.pure: hello_world.c
purecov cc -g -o a.out.pure
hello_world.c
The following example discusses how to incorporate PureCoverage into a more complex nightly build procedure and into a test harness.
The programs prog1 and prog2 share some code segments. They are recompiled from scratch each night and then a test suite is run to exercise each program multiple times. Results from the tests are collected and distributed automatically.
The section of the makefile responsible for recompiling and linking your application, before you start using PureCoverage, looks like this:
all: prog1 prog2
prog1:
prog1.o shared.a
cc -o prog1 prog1.o shared.a
prog2:
prog2.o shared.a
cc -o prog2 prog2.o shared.a
This section is used both for automated nightly builds and for routine daytime builds. You can change it so that the nightly builds run in PureCoverage mode while the daytime builds remain unchanged. For example:
all: prog1 prog2
prog1:
prog1.o shared.a
$(PURECOV) cc -o prog1
prog1.o shared.a
prog2:
prog2.o shared.a
$(PURECOV) cc -o prog2
prog2.o shared.a
Change the nightly compile command line from:
% make all
to:
% make PURECOV=purecov all
Do not change the command line for the daytime builds.
PureCoverage now collects coverage data for each test as the applications run each night.
To notify developers automatically when their code is not being tested sufficiently, incorporate the Low coverage mail report in your makefile.
When the programs are instrumented with PureCoverage, the coverage data is automatically accumulated for each execution of each program. PureCoverage generates the files prog1.pcv and prog2.pcv. You do not need to change the test harness to incorporate the basic data collection from PureCoverage.
However, there are several reasons to consider changing the harness:
To control whether the coverage data is collected for each test separately or for the entire test suite
To discard coverage data from failed tests
To send coverage data reports automatically when the test suite is finished
By default, PureCoverage accumulates coverage data for the various executions of each program in the files prog1.pcv and prog2.pcv. These files reside in the same directory as the programs prog1 and prog2. As each session completes, PureCoverage adds coverage data for that session to the application's default .pcv file. At the end of the test harness, the files contain the cumulative coverage data for the entire test suite.
PureCoverage .pcv files are often in existence at the beginning of a test suite, as for example when they are left over from the previous night's run. If this is the case, PureCoverage combines the existing data and the new data in the .pcv file.
To collect separate data for each test rather than accumulating the data over multiple program runs, specify the option -counts-file=<filename> in the environment variable PURECOVOPTIONS when running the application.
csh
% setenv PURECOVOPTIONS -counts-file=prog1.test1.pcv; \
prog1 <
test1.input
sh
$ PURECOVOPTIONS=-counts-file=prog1.test1.pcv \
prog1 <
test1.input
You can also use the PureCoverage API to specify the name of the output file. This is particularly useful when you want to get information about particular sections of a program.
Since coverage data is collected on a program-by-program basis, the coverage data for the library shared.a is split between the two programs. To combine the coverage data for the programs, you can use the -merge option. For example:
You can use the Viewer to examine both.pcv. It shows the combined data for the shared portion of the code. To open the Viewer, use the command:
% purecov -view both.pcv
You do not have to use the -merge option if you plan to examine the data in the Viewer. You get the same results with the command purecov -view prog1.pcv prog2.pcv. If, however, you have a large number of files and plan to run multiple Viewer sessions, using the -merge option is more efficient.
You can use the -merge option to combine coverage data for separate test cases that were stored separately. For example:
% purecov -merge=tests.pcv test1.pcv test2.pcv test3.pcv
Suppose that the compilation fails and that your test suite runs tonight's version of prog1 but last night's version of prog2, so that some of the shared code is from tonight and some is from last night. PureCoverage tracks the version of each program internally, using an object-code checksum scheme to detect whether the code has been changed.
Before combining sets of data for an object file, PureCoverage looks at the checksums of each version of the object file that is associated with a set of data. If the checksums match, PureCoverage treats the sets of data as generated from the same version and combines them. Otherwise, it uses the newer data and discards the older data. This way, when you examine the results for shared code, you can be sure you are seeing only valid data.
PureCoverage merges or discards data on an object-file basis. If you change one part of an application, and then recompile and make additional runs, PureCoverage discards data only for the files containing changed code and continues to accumulate data for the rest of the application.
If PureCoverage discards data when you merge .pcv files, it prints a warning:
PureCoverage 4.4 Solaris 2, Copyright 1994-1999 Rational Software Corp.
PureCoverage licensed to Rational Software Corp.
Warning: some data ignored because they were from older versions.
Check the Runs column in the Viewer for each file to note the number of runs for which the data has been collected. If the Runs value differs from file to file, either data for multiple applications sharing some code has been merged together, or data has been discarded for earlier runs of a changed file included in the application.
You can display columns in the Viewer that show you the dates of the oldest and newest data for each directory and file. (Select View > Select Columns . . . to display the Column Selection dialog.) This is often helpful when the Runs figures differ from file to file.
You can use the -force-merge option to force PureCoverage to accumulate data even when it appears to be from different versions of the program. This is useful if you build into your program timestamps or other data that varies from one compilation to the next. In these cases, you know that the instructions and line-numbers are unchanged between compilations, and that you can safely merge the results.
When PureCoverage computes checksums, it ignores the timestamps that the compiler automatically inserts into object files.
Unless you are absolutely certain that the two versions were both built from precisely the same source, you should avoid using the -force-merge option. Otherwise, you can end up with meaningless data.
PureCoverage cannot tell if a run of your program succeeded or failed. By default, PureCoverage accumulates the data for each run.
To discard data from failed test runs, you can:
Discard data using the test harness
Discard data from within the test program
When a given test finishes and your test harness determines that the test has failed, the test harness can delete the appropriate .pcv file. This prevents PureCoverage from accumulating data from the failed tests. This is useful if you are keeping separate data for each test case.
To discard data using the test harness, you can change your test script from this:
#!/bin/sh
if test_program
then
echo "test passed!"
else
echo "test failed!"
fi
to this:
#!/bin/sh
if test_program
then
echo "test $name passed!"
mv test_program.pcv $name.pcv
else
echo "test $name failed!"
echo "cleaning up coverage data...."
rm test_program.pcv
fi
At the end of the test suite, you can view the coverage data for a single test, test1:
% purecov -view test1.pcv
or you can see the combined results of the passing tests:
% purecov -view *.pcv
You can use the -run-at-exit option to provide an alternate method of preventing the accumulation of data from failed tests. This method has several advantages: It requires no change in the test harness, and the exit script runs only with the version of the test program that has been instrumented by PureCoverage.
If you do not collect separate data for each test, you can modify the programs to use the PureCoverage API so that data for the failed tests is discarded.
For example, you can use purecov_disable_save() to disable saving data when a test program detects a failure condition. Or you can call purecov_set_filename("pass.pcv") right before a successful exit to arrange that data for passing tests ends up in pass.pcv while all other data ends up in the default file.
You can even use PureCoverage API functions to discard coverage data for only the failed portion of your program, and retain the data for the rest of the run.