Rational TestMate Release Notes

Release 4.4.0

Thank you for choosing Rational TestMate!

Please read this document to learn about known product issues, important tips, and other TestMate product information. The Release Notes are also available online in HTML format from the Help > Manuals command. The online Release Notes are guaranteed to be the latest version of this document. If you find discrepancies between the hard-copy and online versions of the Release Note, please assume that the online document contains the most current information.

---

Objectives

These release notes provide information critical to installing and using Rational TestMate, including supported platforms and known issues with this release.

---

Scope - Product Definition

This document applies to Release 4.4.0 of Rational TestMate. These release notes describe:

• Hardware/Software Information
• What's New in TestMate 4.4.0
• Known Limitations

---

Hardware/Software Information

This section provides basic information on the platforms supported and the hardware and software requirements for installing and running Rational Apex and its add-ins.

Supported Platforms

This document uses the following terms to designated the various platforms on which Rational Apex 4.4.0 runs.

Table 1 Supported Platforms

Term	Platform
Linux	IBM xSeries or compatible Intel x86 workstations, Pentium III or later, running one of the following: Red Hat Enterprise Linux 3 (RHEL 3) SuSE Linux Enterprise Linux 9 (SLES 9) SuSE 9.1
Solaris	Sun SPARC workstations running Solaris 2.6, Solaris 7, Solaris 8, or Solaris 9.
AIX	IBM RS/6000 or pSeries workstations running AIX 4.3.3
Alpha, OSF/1, Tru64 UNIX	Digital Alpha workstations running release 4.0d, 4.0.e, 4.0f, and 5.1(formerly known as OSF/1)
HP-UX or HP	HP PA workstations running release 11.0 and 11.11 of HP-UX (with and without threads)
IRIX	SGI workstations running IRIX 6.5 N32

This release of TestMate requires Apex release 4.4.x. Refer to the Rational Apex Installation Guide for further information about hardware and software configurations.

---

Documentation

Rational Software Corporation provides the following documentation with this release.

The following documents are available in printed form as well as HTML format:

• Rational Apex Installation Guide
• Rational TestMate Release Note
• Rational TestMate User's Guide
• Rational TestMate Customization and Configuration Guide
• Rational TestMate Command Reference

These documents can be accessed through the Help > Manuals command from any directory viewer or the List of Documents entry in the button bar at the top of each HTML document.

Postscript versions of these manuals are also provided and can be printed using the Help > Print Manuals command.

If updates have not been issued, you may note discrepancies between the printed documentation and the online documentation. In these cases, the online documentation is the most up-to-date and should be followed.

The following are provided in on-line form only:

• Rational TestMate Online Reference
• "man" pages for Rational TestMate command-line interface commands.

---

Upgrading from Previous Releases

Once installation is complete, you may need to upgrade existing views containing TestMate artifacts, such as .tc files. This section describes how to upgrade views created by earlier releases of TestMate.

Upgrading From All Previous Versions

New API Views

There are new versions of the TestMate API view:

$APEX_BASE/ada/testmate.ss/<arch>.ada83.4.2.0.rel
$APEX_BASE/ada/testmate.ss/<arch>.ada95.4.2.0.rel
$APEX_BASE/c++/testmate.ss/<arch>.4.2.0.rel

User views which are upgraded to Apex 4.2.0 must import the corresponding TestMate 4.2.0 API. This can be done by using the apex import command or (preferably) by using the TestMate > Setup dialog box.

The new API views (which are Summit views) are used for software development in both Rational subsystems (new in Apex 4.0.0) and Summit views.

New Customization Keys

There are new versions of the TestMate customization keys. Typically, all views are setup to reference the new keys indirectly via the TestMate.sw file switch TM_CUSTOMIZATION_KEY. In the event that a site specific key is used or this switch is hardcoded to a value other than the default value, the appropriate modifications must be made to the TestMate.sw file to either use the new keys directory provided in this release, or to update the site specific keys referenced by the TestMate.sw file to reflect any modifications made to the new keys.

The new keys directories are:

$APEX_BASE/{c++,ada}/keys/tm_standard.4.2.0

For TestMate Ada Cross for Rational Exec or Tornado, the keys directory is:

$APEX_BASE/{c++,ada}/keys/tm_cross.4.2.0

For TestMate Ada Cross for LynxOS, the keys directory is:

$APEX_BASE/{c++,ada}/keys/tm_unix.4.2.0

For TestMate Ada Cross via Apex RCI, the keys directory is:

$APEX_BASE/{c++,ada}/keys/tm_rci.4.2.0

Testing View Imports

Existing TestMate-created testing views will not automatically be updated to import the new testmate.ss API view. Users should delete the testing views.

Note: Testing views are created for execution of certain types of data-driven test lists and for instrumented coverage. TestMate creates these transient views as needed, and they may be deleted by the user without affecting original source code or tests themselves. These views are normally located in the directory <subsystem name>.ss/TestMate.

Upgrading from TestMate 2.4.1 or 2.4.6

Upgrading TestMate Artifacts

TestMate 4.2.0 is upwardly compatible with all TestMate versions 2.4.6 and later. TestMate artifacts created using the earlier versions can be directly used by TestMate 4.0.0 without being upgraded. However, in general, test cases and test lists which have been created or modified using TestMate 3.x or 4.x may not be used again by TestMate 2.4.6.

There is no requirement to explicitly upgrade any of the TestMate artifacts (test cases, test lists, etc.) for 4.2.0.

With TestMate 3.0.0, a summary field was added to test cases and test lists. The tmupgrade command (described below) can be used to initialize this field to an initial value based upon the first line of the description field. In addition, the switches FT_HIDDEN, FTGEN, STGEN, LIVE_UNITS, and TEMPLATE_PLACEHOLDER_SUB are no longer used by TestMate and will be deleted from any TestMate.sw files by the tmupgrade command. It is recommended that tmupgrade only be used if it is desirable to initialize the summary field or these switches have been defined in the TestMate switch file (warnings will be issued by TestMate commands if these switches are encountered; leaving them defined will have no other affect besides issuing these warnings).

The tmupgrade command can be run either directly from the command line in an Apex xterm, or indirectly via a dialog interface accessible from the TestMate > Misc > Upgrade to new release... menu.

Dialog Option	tmupgrade switch	Description
Delete Obsolete files	+/-delete	Deletes TestMate files which become obsolete in this release and cannot be upgraded. Default: TRUE
Display changes only...	+/- effort_only	Only indicate what would be done; does not change/delete anything. Default: FALSE
Update switch model	+/- update_switches	Updates the TestMate.sw file to be compatible with this release. Default: TRUE
Initialize summary field...	+/-update_summary	Sets the summary field of the test case/list to contain the first line of the description field. Default: TRUE

The objects operated on can be views, directories, or any TestMate specific object. In the case of a directory or view, all objects contained within it will be processed, recursively processing any subdirectories.

Objects which are under Apex CMVC control will only be modified if the proper TestMate CMVC switches are set in the view in which the objects reside, or if the global session switch is set, or if they are currently checked out. See the TestMate switch information for AUTO_CHECKIN and AUTO_CHECKOUT.

Only test cases, test lists, and TestMate.sw files will be examined, and, if necessary, modified. Other files will be ignored.

---

What's New in TestMate 4.4.0

Linux-Hosted Development

TestMate 4.4.0 adds support for development on Linux host systems, in addition to development on Solaris.

With respect to embedded development, TestMate supports the same host and target machines that are supported by the Apex Embedded layered products.

---

What's New in TestMate 4.2.0

Numerous improvements have been made in instrumented coverage to handle source release variants, though some of these changes will benefit all users. TestMate's instrumented coverage feature now handles units that are imported by the testmate.ss subsystem itself, and units with library unit pragmas such as pragma Pure. Also, the user can now create subprogram tests for units arbitrarily deep in the Ada closure of the test list view.

The mechanism for manipulating instrumented library contexts has been completely reworked to maintain the integrity of instrumented code. The instrumented "state" is now saved in TestMate-created libraries (both views and Rational subsystems) so that TestMate can better determine when it can re-use existing instrumented code.

Many changes have been made to eliminate or reduce unneeded processing in TestMate. Processing of the Ada closure has been minimized to cases where it is really needed. In addition, ASIS library opens are now conditionally cached when possible. The results of some internal lookups are also cached. In some scenarios, these changes dramatically decrease the time it takes to run tests, especially for nested test lists of Ada test cases.

---

What's New in TestMate 4.0.0

The most obvious change in TestMate 4.0.0 is support for Apex's integration with ClearCase. Refer to subsequent sections for changes introduced in TestMate 3.0 and 3.2.0.

ClearCase Support

TestMate 4.0.0 offers full support for both Summit views and ClearCase style Rational subsystems.

Note that Apex 4.0.0 introduces the term "library context". A library context can take either of two forms:

• Summit view (.wrk or .rel)
• Rational subsystem (.rss)

TestMate adopts the same terminology. In general, any TestMate operation that applies to a summit view will apply equally to a Rational subsystem. TestMate also supports a new keyword, <library_context> (alias name <lc>) which resolves to the library context. For compatibility, existing keyword <view> will resolve to the library context, even for a Rational subsystem.

Instrumentation Improvements

• Code instrumentation is more robust, for both code de-encapsulation (used in data-driven tests and ada subprogram stubs) and instrumented coverage.
• Improved handling of complex import closures, especially mutual imports.
• Problems with decision coverage have been fixed.
• Instrumented coverage now handles Ada95 protected types
• Fewer dependencies on the runtime in instrumented coverage harness.

Smart Output Comparison

A "smart" output comparison script, tmeval_partial_compare, was introduced in TestMate 3.2. This script has been completely rewritten to provide diagnostics to locate differences between the files. In addition, a new test case prototype is provided to use this feature.

Improved TestMate Setup

TestMate setup has been made simpler to use and less error-prone, especially for use with Apex Embedded:

• TestMate > Setup dialog no longer requires separate TestMate models.
• Apex Embedded users no longer need to use the tmbuild command.

---

Known Limitations

Program Test Cases and RCI Environments

Program test cases are used to drive executables. In the case of RCI, these executables may be text or binary files that need to be downloaded to a target processor.

TestMate may not locate these executables properly in some RCI environments. In order to make program test cases execute properly for RCI targets, make sure you do the following:

For each test driver in the RCI directory, create a symbolic link from the target test driver to one of the locations on your test driver path (as specified in your TEST_DRIVER_LOC switch). The name of the symbolic link _must_ match the name of the file containing the Ada main procedure, minus the ".2.ada" suffix.

For example, if you have an Apex RCI to VADScross environment, and have a test driver named foo.vox, whose main subprogram is located in <view>/Tests/drivers/foo.2.ada, you must create a symbolic link from the file foo.vox to <view>/Tests/drivers/foo:

% ln -s <target_rci_directory>/foo.vox <view>/Tests/drivers/foo

(when executing the "ln" commands, you should substitute the correct values for the placeholders shown above).

When locating the Ada source for a test driver, TestMate first adds ".2.ada" to the end of the test driver and starts traversing the Ada library from that point (in other words, it assumes that the test driver executable file is located in the same location as the corresponding .2.ada file, and that they have the same simple name).

By creating a symbolic link from the executable to the place that TestMate expects to find the file (and with the correct name), you essentially work around a limitation that TestMate would otherwise have with running program test cases for RCI targets.

Test Result Group

Coverage Summary of Test Result Group Coverage

The test result group display does not summarize the coverage totals across all test results in the display. To obtain a summary, select all the test results in the image and use TestMate > Test Evaluation > Coverage Summary to get a coverage summary.

Graphical User Interface

Match/Filter

The CONTROLS operators only work for textual comparisons. Numeric comparisons such as filters for all "coverage < 80%" will not work.

Skeleton test cases

The test case signature and pattern information for a test list is kept in a special test case associated with every test list. This special test case has the string "_skeleton.tc" as part of its file name. You should never delete this test case nor attempt to manually insert this test case into a test list.

Missing test list elements

If elements (test cases or test lists) referenced by the test list (test list group) have been removed, relocated, or had their permissions changed, you may not be able to visit the test list. Depending on which element is missing, one of the following situations may arise:

• The test editor appears, but some of the elements are missing. In this case, an error message in the message output window indicates which element(s) were not found. If the test list is subsequently modified and saved, the reference to the missing items will be removed from the test list. If you want to keep the reference (because you may be able to remedy the situation by restoring the elements), exit the Test Editor without making any changes to the test list. The test list will still have the references to the troublesome items. Once the problem has been fixed, you should be able to visit the test list once again.
• Error messages indicate that the skeleton test case is missing and the test list open failed. In this case, you must copy one of the other test cases in the list to the name of the skeleton in the error message. Usually, this name is the base name of the test list (without the ".tl") with "_skeleton.tc" appended. You can dump the test list using /bin/cat or more to determine the other test cases in the list. Once you've restored the missing skeleton test case, you should be able to visit the test list.

Test List Editor and Test Case Cell Values

Prior to TestMate 3.0 the test case driver, evaluation action, epilog, precondition and prolog fields had three fields each. These were the Result field, Procedure field, and Parameters field. These have been collapsed into a single field which is parsed internally to derive the result, procedure and parameters fields. This section describes the language constraints associated with these cells.

There are three varieties of syntax which can appear in one of these test case fields. The syntax is described below using quasi-regular expressions. In the expression to follow a set of bracket enclosing an element means the element is optional. A \Q*' symbol following an element means the element can appear 0 or more times.

Program Test Cases

The format for program test cases is:

[<result> =] [<procedure> [<arg> [<arg>]*]]

<result> is optional, arguments can only appear if a valid <procedure> value is present, and the null string is accepted.

Ada Test Cases

The format for an ada test case is:

[<result> :=] [<procedure> [ ( <arg> [,<arg>]* ) ]]

<result> is optional, arguments can appear only if a procedure appears and must be enclosed in parens, and the null string is accepted. Further the values for <result> and <procedure> simple names.The values for <result> and <procedure> may be simple identifiers, or identifiers with selectors, for example, r = p.f(x).

C and C++ Test Cases

The format for a C or C++ test case is:

[<result> =] [<procedure> ( [ <arg>, [<arg>]* ] ) ]]

Parameters can only appear if a procedure appears and must be enclosed in parenthesis. The parenthesis must appear even in the absence of arguments.

This differs from the ada syntax above. The values for <result> and <procedure> may be simple identifiers, or identifiers with selectors, for example. r := p.f(x). The scope resolution operator is also available for <procedure> values, for example, r := c::p.f(x).

Data-Driven Testing - C/C++

TestMate does not currently support data-driven test lists for C or C++ units. Unit testing may still be performed using C and C++ test cases by writing your own driver.

Data-Driven Testing - Ada Test Cases

This section describes limitations of the data-driven testing features of TestMate. If these limitations apply to an Ada unit that you wish to test, consider using a non-data-driven Ada test case instead, and write an Ada driver manually. For more information see the chapter on "Ada Test Cases" in Using TestMate.

Note: Some limitations relate to TestMate's ability to de-encapsulate private types in data-driven test cases. Normally, TestMate allows you to enter literal values of any parameter in the test editor, regardless of whether or not the parameter's type is directly visible (any literal value that is entered is interpreted as if the literal were located immediately following the full type declaration). However, in some instances TestMate cannot generate code to de-encapsulate literals. In these instances you can still enter values in the cell that follow normal Ada visibility rules, for example, you can enter a functional call that returns a value of the private type.

Task Types

TestMate will either fail to generate a compilable script, or it will fail to generate the script for subprograms which have task type. (including function return types).

This restriction also applies to stubs with task type parameters or function return types.

Unconstrained Output Parameters

Unconstrained output parameters are not supported due to a current limitation in creating an appropriate constrained output target object. An example would be an out parameter of string type.

Anonymous Arrays

Global variables of an anonymous type are not yet supported (for example Xarr : array(1..10) of integer;). The user is encouraged to create a declared type and use that instead of the anonymous type:

type Xarr_Type is array(1..10) of integer;
Xarr:Xarr_Type;

Tasks

This release does not support data-driven tests of tasks. All other test cases support testing of tasks and task types (for example, an Ada test case may call a custom driver that calls task entries).

This restriction also applies to stubs -- automated stubs cannot be tasks.

Hidden Variables & Subprograms in Generics or Subunits

Automatic pattern generation for hidden global variables and testing of hidden subprograms is not supported for generic instantiations or units under test that have subunits.

Stubs and Generics

Stub values for generic instantiation subprograms are not yet supported. The user can workaround this limitation by declaring the generic instantiation unit and generic unit as live.

De-Encapsulation in Generics

A private type in a generic specification cannot be de-encapsulated. Note that generic formals used in instantiations can be private types with proper de-encapsulation.

Composites with Surrogate Components

If a composite type (record or array) includes a component which would require a surrogate routine (e.g., for stubbing or de-encapsulation), then the composite type requires a de-encapsulation. This de-encapsulation is not yet supported. Note that de-encapsulation is also required if the user tests a floating point component using a "range" test.

Unconstrained Private Records

A private type with non-default initialized discriminants cannot yet be handled with a de-encapsulation. A warning will be issued and the user should use external values of this type as parameters (rather than having TestMate create a surrogate based on the user-entered literal value).

Data-Driven Ada Tests - Test Data Generation

Limitations on Parameter Types

Test data generation supports subprograms with parameters of most Ada types, or types derived from them. However, the following types are not yet supported:

• arrays (except strings)
• task types
• incomplete types

Limitations on Certain Program Units

TestMate cannot currently generate test data for the following program units:

• tasks
• generic program units (generic instantiations are supported)

Coverage Analysis

Trap Based Coverage Analyzer Decision Limitation with Tasking

A window exists in which decision coverage can be missed due to two tasks executing the same object code decision at the "same time". If each task takes a different decision target, it is possible that one of the targets will not be recorded, resulting in no coverage for that decision when in fact there was coverage.

C/C++ Statement and Decision Coverage

This is currently not supported.

Ada Source Line Coverage

This is currently not supported. Statement and decision coverage provide more information than line information, and should be used instead.

In-lined Subprograms

Trap based coverage analysis is not supported for inlined subprograms for both Ada and C/C++.

C++ Templates

Coverage for templates is currently not supported.

Ada Generics

In order to collect trap based coverage on a compilation unit which contains, or is itself, a replicated generic, one must specify the following as units under test:

• The generic unit
• At least one instantiation of that unit

Optimized Code

Under some conditions of optimized code, source level decision coverage using the trap based coverage analyzer may indicate that a decision has been optimized away if the explicit target of the decision has been optimized away.

Trap Based Incorrect "Optimized Away" Message

Elaboration segments/decisions for generics are incorrectly annotated as having no object code. The source span for these elaboration segments covers the entire spec or body. If one displays the Ada source coverage based upon the summary totals for a unit which has 0% coverage for the elaboration code, the entire unit will be highlighted as uncovered. The coverage totals for each unique subprogram are correct, and as a workaround, one can display the Ada source coverage on a subprogram by subprogram basis (just not the entire unit at once).

Under some conditions, a return statement which starts a segment will be highlighted as optimized away, when in fact it has not. Under this condition, there is no way of knowing whether or not it was executed. There are two simple conditions in which this is known to occur - returning the value passed and returning a literal. For example:

function Foo (A_Value : Boolean) return Boolean is
begin            
  return A_Value;
end Foo;
function Foo return Boolean is
begin 
  return True;
end Foo;

A work around is to preceed the return statement with another statement which causes code to be generated.

Multiple UNIX Processes

Coverage analysis is not supported outside the normal Ada tasking implementation; i.e. if a user program (Ada, C, or C++) performs a UNIX fork thereby creating another UNIX process, coverage analysis on the new process is not supported and may result in erroneous coverage results for the originating process.

C/C++ Threaded programs

Coverage for C/C++ programs which use threading is not currently supported.

C/C++ Test Cases

CM/Timestamp for C/C++ Units

CM information is not collected for C/C++ units specified as units under test.

Test Case #Includes

In the C/C++ test case - Includes section, use of the notation "#include <myfile.h>" will result in a warning message during script generation:

!!! Unknown Placeholder <myfile.h> ignored

This warning can be ignored.

MCDC Coverage Analysis

MCDC_Analysis Primitive and Test Case Generation

Use of the "Test Case Generation" option with the MCDC Analyzer primitive may cause premature termination of this operation.

Coverage File Report Format Error

When generating a coverage file report that includes a CASE statement with a Boolean expression the display for this decision points will include an error message. Subsequent metrics information will be off by the condition totals for these expressions.

Pair Table Display for large Boolean Expressions

The maximum number of conditions supported for any single boolean expression is 24 for this release. Expressions with more then 24 subconditions are rejected with an appropriate error message.

Strong MCDC

The default algorithm utilized to generate boolean pair tables does not recognize coupled subconditions. Coupled subconditions are subconditions that are directly dependent upon each other (such as two subconditions that are the same boolean variable). The result of this is that when coupled subconditions are present, impossible to achieve pairs will be displayed and complete coverage will not be attainable without manual evaluation.

One solution is to restructure the code to remove (or test for) coupled conditions.

Overloading Package Standard

Overloading package standard may hinder the instrumentation of source level constructs that require type conversion. For instance, instrumentation of an expression of a type derived from boolean will attempt to convert the expression to type "standard.boolean" which may fail at compilation time if the overloaded standard package does not properly remap type boolean to the expected predefined type.

Nested Generic Instantiations

Visible objects in nested generic instantiations (more then one level deep) may type is based on a generic formal whose actual type is based on standard.boolean fail to instrument properly. The type used in type conversions is the formal type name instead of the actual (based on boolean) type name.

Floating Point Type Attributes used as CASE Choices

Floating point attributes used as CASE statement choices are not currently statically evaluated by MCDC processing. For instance:

case My_Obj is       
 when My_Float'Mantissa | My_Float'emax =>                
  null;        
 when others =>                
  null;
end case;

In the above example the \QMantissa and \QEmax attributes will be rejected with an appropriate error message and processing will continue.

Categorization Pragmas

MCDC instrumentation will fail to produce a compilable program if a unit with a pragma pure declaration is not instrumented and it "withs" a unit with the declaration that is instrumented. The same limitation applies to pragma shared-passive.

C/C++

Exceptions

Drivers which terminate with an Unhandled exception (C++) are not correctly processed by the Test Execution Server. A workaround is to change the default value of the FAILED_WITH_EXCEPTION status mapping. This should be set to the value 32774. See "User-defined Results" in the TestMate Customization and Configuration Guide for more information.