TOC PREV NEXT INDEX DOC LIST MASTER INDEX


TestMate MCDC Tool Qualification

Many safety-critical standards require that certain software tools be qualified for use, especially if they automate some required activity. This is to ensure that the tool not only performs its stated function, but that it also can automate the activity with accuracy at least as good as a person doing it manually.

RTCA/Doc DO-178B

RTCA/Doc DO-178B, "Software Considerations in Airborne Systems and Equipment Certification" is a regulatory standard used by the U.S. civil aviation industry (as well as those in other countries) for ensuring that software used in commercial/civil aircraft is safe. In the U.S., the Federal Aviation Administration (FAA) is responsible for ensuring that the aviation industry follows the standards described in DO-178B.

DO-178B Tool Qualification

Section 12.2 of RTCA/Doc DO-178B describes the requirements for achieving tool qualification. Furthermore, this section identifies two distinct types of software tools: Software Development Tools and Software Verification Tools.

Based on the definitions in this section of DO-178B, TestMate is a considered a Software Verification Tool.

Section 12.2 continues by describing the process and data required for tool qualification.

Rational's Involvement with Tool Qualification

There are no provisions within DO-178B that describe how a tool vendor should be involved with tool qualification. In fact, DO-178B indicates that tool qualification is the sole responsibility of the specific project using the tool. Once a software tool has been qualified for use on a particular project, it does not guarantee that it will be qualified on another project. However, experience has shown that the level of effort to achieve tool qualification is much simpler if a tool has been qualified before.

The data provided in this package is meant to serve as examples of documentation that can be prepared by a particular customer and submitted for tool qualification consideration. In fact, this data has been successfully used by other customers to achieve tool qualification. Rational provides no guarantees or warranties that this data is sufficient for achieving tool qualification for your project. We encourage you to work closely with your regulatory agency, as early in your development as possible, to determine what is required by your agency for tool qualification.

Rational grants you permission to use, modify, and submit this documentation for your tool qualification submission. From a certification standpoint, you are the sole responsible party for the contents of this documentation. If changes are required by your regulatory agency, you are responsible for making them. However, we also encourage you to provide feedback to us so that we can better improve these documents for other tool qualification efforts. Submit this feedback through Rational's normal support channel support@rational.com.

Overview of Tool Qualification

This section describes the steps that are typically taken to achieve verification tool qualification for a project being certified to RTCA/DO-178B. This process could be different if you are certifying your software to a different standard.

The following sections identify the steps commonly used for DO-178B.

Identify the development/verification tools that you plan to use

DO-178B indicates that this should be done at the beginning of your project as part of the Software Planning Process. These tools include compilers, configuration management tools, visual modeling tools, and test tools.

Identify which of these tools will be required to be qualified as a development or verification tool.

DO-178B indicates that a tool needs to be qualified when processes of DO-178B are "eliminated, reduced, or automated by the use of a software tool without its output being verified" according to DO-178B.

TestMate MCDC meets these criteria. In addition, it performs verification activities and therefore needs to be qualified as a verification tool. The service history of TestMate shows that other features of TestMate (such as test generation, test management) do not require tool qualification because they do not meet this test.

Inform your regulatory agency that you will be using a tool that requires qualification.

This is typically done at the start of the project and is identified in your project's Plan for Software Aspects of Certification (PSAC) document.

Depending on the complexity regarding the tool qualification, it may be possible to include a tool qualification plan within the PSAC document. If not, a separate Tool Qualification Plan should be prepared and submitted to your regulatory agency.

A sample Tool Qualification Plan for TestMate MCDC is provided by Rational. You may find that it contains useful information when preparing your own Tool Qualification Plan. This is provided in electronic form and is described later in this chapter.

Your regulatory agency approves your planned usage of the verification tool when they approve your PSAC.

Prepare a Tool Operational Requirements (TOR) Document

DO-178B indicates that qualification of software verification tools be achieved by "demonstration that the tool complies with its Tool Operational Requirements under normal operating conditions." (DO-178B Sect 12.2.2). This implies that there is a TOR for the verification tool.

Rational provides a TOR for TestMate MCDC. This is provided in electronic form and is described in more detail later in this chapter.

Verify that the tool fulfills the requirements in the TOR

As mentioned in the previous step, tool qualification occurs when you demonstrate that the tool complies with its TOR under normal operational conditions. This implies that a test suite needs to be run. Furthermore, since the behavior of TestMate MCDC can be affected by the operational environment (e.g. target hardware that is being used, type of cross compiler, etc.), the regulatory agency often requires for the tests to be run in the actual target environment. This means that the test results that Rational collects when it runs the test suite may not be applicable to your project and that you may have to rerun the qualification tests using your actual target environment.

Rational has extensively tested TestMate MCDC against its TOR. This test suite is provided as part of the Rational Apex Embedded MARK certification package. Instructions for running this test suite are provided later in this chapter. This test suite is designed to be run on a variety of target platforms (including native and cross-targets).

Submit Test Results

DO-178B does not specifically require for detailed test plans or test results be provided to the regulatory agency. However, it is often more convenient to create a Test Plan and Test Results to document that you have executed the qualification test suite.

Rational provides electronic forms of these documents that you can use.

DO-178B states that the verification tool is considered qualified when it accepts the project's "Software Accomplishment Summary". Therefore, you may want to include the tool test plan and test results as supplemental data to the Software Accomplishment Summary.

Testmate MCDC Qualification history

Even though tool qualification history plays no official role in the tool qualification process (at least for RTCA/Doc DO-178B), it can be important to some customers.

This section describes the relevant aspects of TestMate's tool qualification history. However, it is by no means inclusive of every project that has successfully qualified TestMate. In many cases, customers qualify TestMate without the assistance of Rational. Unless they explicitly tell us about their qualification efforts, we have no way to track their progress. Furthermore, of the customers that have told us about their tool qualification efforts, for privacy reasons, we are unable to provide their names here. If you require a reference, please contact your local Rational representative.

Rational TestMate Product History

Rational TestMate was first released in 1994. Its predecessor was a test tool developed for Rational's R1000 Ada development environment. This initial release of TestMate included test management and simple coverage analysis capabilities.

Since 1994, there have been approximately 1 new release of TestMate each year. Each of these releases introduced new product functionality.

In 1995 and 1996, Rational teamed with up with a major airplane manufacture to incorporate MCDC (Modified Condition/Decision Coverage Analysis) into TestMate. The culmination of this effort was the MCDC add-on to TestMate known as "TestMate MCDC". TestMate MCDC was released in 1996.

Working together, Rational and this customer generated the qualification documentation and the test suite needed to qualify TestMate MCDC as a DO-178B verification tool. These artifacts included: a test suite to test the operational requirements of TestMate MCDC, Tool Operational Requirements, Tool Qualification Plan, Software Configuration Index, and Test Results. In late 1996, TestMate MCDC was qualified as a DO-178B Level A verification tool for the first time.

After the introduction of TestMate MCDC in 1996, Rational enhanced TestMate to also perform DO-178B-defined statement and decision coverage analysis. In mid 1997, a commercial avionics project successfully qualified these features as verification tools (for DO-178B Level B & Level C software).

Since 1996, other customers have also qualified TestMate MCDC as a verification tool using the preliminary set of documentation that Rational generated for the first tool qualification. These customers typically are developers of commercial or military avionics systems and are required to adhere to DO-178B.

Tool Qualification Issues

According to DO-178B, verification tool qualification involves testing the verification tool to ensure that it meets its tool operational requirements. This implies that the following artifacts are required to satisfy this process:

Tool Qualification Plan Tool Operational Requirements Test suite Tool Test Results According to DO-178B, Verification Tool Qualification is required for any tool that automates a required verification activity without subsequent human inspection that the tool operated correctly. Coverage analysis (such as MCDC) is a required verification activity for Level A software. Since TestMate MCDC automates the collection of MCDC, it is often necessary to qualify it as a verification tool. Other features of TestMate do not automate required verification activities as described in DO-178B and are therefore not required to be qualified.

Tool qualification is ultimately the responsibility of the developer of the application being certified, the tool end user. Unfortunately, there are no clear standards or expectations as to the level of effort expected from a tool vendor during the tool qualification process. At one extreme, tool qualification can occur completely without the support or assistance of a tool vendor and can be completely performed by the end user. At the other extreme, the tool vendor can perform all of the required activities.

Rational's position is one found in the middle of these two extremes. Rational recognizes that the end user is the ultimate party responsible for tool qualification. In addition, Rational recognizes that it has access to data that could make tool qualification simpler for the end user. As such, Rational offers a set of documentation that an end user may find useful during the tool qualification process. However, Rational makes no claims that this documentation, alone, will ensure a successful tool qualification by the end user. This documentation should be viewed by the end user as sample documentation that they can start with when generating the data that their certification authority requires for tool qualification.

Using Qualification Data provided for testmate MCDC

Rational provides a number of electronic documents that will help you achieve tool qualification for TestMate MCDC. These documents are written using the Rational SoDA for UNIX product (which is based on the Adobe FrameMaker publishing product). Some of these documents can be used as-is - that is you do not need to modify them (unless you want to add your company name, etc). Others need to be generated using Rational SoDA as they pull information in from TestMate and Apex.

The qualification test suite is provided as a series of TestMate test lists and test cases. You will need to run this in your actual development environment using your actual target hardware.

This section provides you additional details about the documentation provided.

Locating and Installing the Qualification Data

The TestMate MCDC tool qualification data and test suite is located with the Rational Apex Embedded MARK product. This is provided with Rational Apex Embedded, but requires a special license key to install (purchased separately from Rational Apex Embedded). Once you have installed the MARK product you will have access to a tar file that contains the TestMate MCDC qualification data.

Preparation

Before proceeding, ensure that you have created a Rational subsystem that will be used to store the qualification data. For this discussion, we will call it tm_mcdc_qual.ss. From within this subsystem, create a working Apex view using the model for your particular target hardware (or a native model if you want to run the tests on the host workstation). For the remainder of this section, the path to this working view will be referred to as:

You should also ensure that TestMate Cross has been properly set-up for your particular target environment. If not, refer to the TestMate Cross Development documentation.

Installation of Qualification Data

You are now ready to copy the TestMate MCDC qualification data from the MARK installation area to the view that you created in the previous step. The tar file is located at:

<NEED THE PATH NAME TO WHERE THE TM MCDC QUAL TAR FILE IS>/mcdc_qualification_test.tar

Copy this file to $QTS_DIR

From $QTS_DIR, execute the following UNIX command:

This will extract the TestMate MCDC qualification data into your working view.

Run the Installation Script

After following the previous step, a new subdirectory will be created in your working view. This subdirectory is called 'install'. Execute the following command to set-up your current view to run the TestMate MCDC qualification tests:

This will set multiple TestMate switches and will ensure that all Ada packages are coded in your working view.

After running this test setup script, you may want to verify that the sample programs in the ./install subdirectory can be compiled, linked, and executed on the target. To verify that your installation is correct, perform the following:

1 . Choose one of the 'hello' world programs that compiles (There are several versions because the underlying I/O packages are not available for all targets)

2 . Make sure that the program links and executes successfully.

3 . Link the tms_test_drive.2.ada, and make sure it links and executes successfully.

4 . Run the TestMate test located in $QTS_DIR/Tests/lists/in_try.tl . It should run successfully and produce a result of PASS.

5 . Finally, run the TestMate test located in /Tests/lists/in_try.tl with coverage enabled. It should run successfully, and produce coverage. Any coverage value of better than 0% is acceptable.

Directory Structure of the Qualification Test Suite (QTS)

Your view $QTS_DIR should now contain the following subdirectories:

Tool Qualification Plan

DO-178B specifies that a Tool Qualification Plan is used to describe the tool qualification process. While this document is not technically required for verification tool qualification, it is useful to specify the tool qualification process somewhere. According to DO-178B (Section 12.2.3.1), the Tool Qualification Plan should contain:

You can locate the Tool Qualification Plan in the following location:

This directory contains several files in it. The main file, qualplan.book, is a Frame 'book' file that contains a reference to the other files in this directory that comprise this document.

Open the qualplan.book file with SoDA. This book (as well as most of the other books) generally have the following structure:

This Tool Qualification Plan is a template that you need to modify for your particular project. It contains various instructions throughout the document on what type of information you should provide. This document does not utilize any type of SoDA domain so therefore it does not need to be generated.

TestMate Operational Requirements (TOR) Document

DO-178B specifies that a Tool Operational Requirements is required for both software development tools and software verification tools. This document describes how the tool is supposed to operate. According to DO-178B (Section 12.2.3.2), the Tool Operational Requirements should contain:

You can locate the Tool Qualification Plan in the following location:

This directory contains several files in it. The main file, swrd.book, is a Frame 'book' file that contains a reference to the other files in this directory that comprise this document.

Open the swrd.book file with SoDA. This book (as well as most of the other books) generally have the following structure:

This TOR does not require any modification. It can be printed as is and used as the TestMate Tool Operational Requirements Document.

Test Plan and Procedures

DO-178B does not require a Test Plan and Procedures Document for verification tool qualification. However, one has been provided to explain how the TestMate MCDC qualification test suite works.

You can locate the Test Plan and Procedures in the following location:

This directory contains several files in it. The main file, test_plan.book, is a Frame 'book' file that contains a reference to the other files in this directory that comprise this document.

Open the test_plan.book file with SoDA. This book (as well as most of the other books) generally have the following structure:

This book is a SoDA generated document. That is, besides containing static text this document must be generated using SoDA. When doing this, SoDA will extract information from TestMate and Apex to construct the document.

This document primarily extracts information from the test lists and test cases.

It also contains information on how to actually run the TQS.

Test Results

DO-178B does not require a Test Results Document for verification tool qualification. However, one has been provided as a convenient way to document the results of running the TQS.

You can locate the Test Results Document in the following location:

This directory contains several files in it. The main file, test_results.book, is a Frame 'book' file that contains a reference to the other files in this directory that comprise this document.

Open the test_plan.book file with SoDA. This book (as well as most of the other books) generally have the following structure:

This book is a SoDA generated document. That is, besides containing static text this document must be generated using SoDA. When doing this, SoDA will extract information from TestMate and Apex to construct the document.

This document primarily extracts information from the test results files generated by TestMate.

If you modify the TQS (such as adding, removing, or modifying TestMate test lists and test cases), you should regenerate this document so that it is up to date.

You should always regenerate this document when you re-run the TQS in order to have the most recent test results stored within this document.

Rational Software Corporation 
http://www.rational.com
support@rational.com
techpubs@rational.com
Copyright © 1993-2001, Rational Software Corporation. All rights reserved.
TOC PREV NEXT INDEX DOC LIST MASTER INDEX TECHNOTES APEX TIPS