Service Data Object 2.1.1

Java™ API (JSR-235)

Test Appeals Process

1 Purpose

The JSR-235 TCK is currently the property of IBM. IBM will collect the test challenges and provide a resolution for addressed and validated challenges with the next applicable maintenance release of the TCK. If a test is determined to be invalid in function or if its basis in the specification is suspect, any licensee of the JSR-235 may challenge the test. Each test validity issue must be covered by a separate test challenge. Test validity or invalidity will be determined based on its technical correctness.

2 Test Challenge Reasons

The following reasons will be considered for corrective action to the TCK by IBM:

1. Test has bugs (i.e., program logic errors)
2. Specification item covered by the test is ambiguous
3. Test does not match specification

Every challenge that is submitted should indicate at least one of the above “Test Challenge Reasons”. Challenges based upon issues unrelated to technical correctness as defined by the specification will normally be rejected.

Test challenges must be made in writing to IBM via the JSR-235 comments mailing list (jsr-235-comments@jcp.org) and include all relevant information as described in the Test Challenge form below. The process used to determine the validity or invalidity of a test (or related group of tests) is described in JSR-235 TCK Test Appeals Step 3.

All tests found to be invalid will either be placed on the Exclude List for that version of the JSR-235 TCK or have an alternate test made available as follows:

1. Tests that are placed on the Exclude List will be placed on the Exclude List within a reasonable time after the determination of test validity. The current Exclude List will be available to all JSR-235 TCK licensees.
2. The Spec Lead has the option of creating alternative tests to address any challenge. Alternative tests (and criteria for their use) will be made available within a reasonable time, either in a maintenance release or in a patch made available from the Tech Lead (goodson@uk.ibm.com).
   
   
   

3 JSR-235 TCK Test Appeals Steps

1. JSR-235 licensee writes a test challenge to IBM contesting the validity of one or a related set of JSR-235 tests.

A detailed justification for why each test should be invalidated must be included with the challenge as described by the Test Challenge form below. The email address for test challenges is jsr-235-comments@jcp.org.

2. IBM evaluates the challenge.

If the appeal is incomplete or unclear, it is returned to the submitting licensee for correction. If all is in order, IBM will check with the test developers to review the purpose and validity of the test before writing a response. IBM will attempt to complete the response within a reasonable time. If the challenge is similar to a previously rejected test challenge (i.e., same test and justification), IBM will send the previous response to the licensee.

A decision of test validity or invalidity is normally made within a reasonable time counting of receipt of the challenge. All decisions will be documented with an explanation of why test validity was maintained or rejected.

3. The licensee is informed of the decision and proceeds accordingly.

If the test challenge is approved and one or more tests are invalidated, IBM places the tests on the Exclude List for that version of the test(s) (effectively removing the test(s) from the Test Suite). All tests placed on the Exclude List will have a bug report written to document the decision and is made available to all licensees.  

4. If the test challenge is rejected, the licensee may choose to escalate the decision to the Executive Committee (EC), however, it is expected that the licensee would continue to work with IBM to resolve the issue and only involve the EC as a last resort.

4 Test Challenge Form

• Test Challenger Name and Company

• Specification Name(s) and Version(s)
• Test Suite Name and Version
• Exclude List Version
• Test Name
• Test Challenge Reason(s) (see above)
• Complaint (argument for why test is invalid)

5 Test Challenge Response Form

• Test Defender Name, email contact and Company
• Test Defender Role in Defense (e.g., test developer, Maintenance Lead, etc.)
• Specification Name(s) and Version(s)
• Test Suite Name and Version
• Test Name
• Defense (argument for why test is valid). Can be iterative.
• Implications of test invalidity, e.g., other affected tests and test framework code, creation or exposure of ambiguities in the specification (due to unspecified requirements, invalidation of the reference implementation, creation of serious holes in test suite
• Alternatives (e.g., are alternate tests appropriate?)
• Passing an alternative test is deemed equivalent to passing the original test