MustGather information for Certificate problems

The documentation required to diagnose problems with Certificates includes

  1. Diagnostic steps to take before reproducing

    Known issues to check first

  2. Known IHS problems at startup or request processing issues to check first

  3. Other known issues


    Unable to start Ikeyman

    GSKit v5 Ikeyman doesn't start in Windows 2003

    On Windows 2003, Ikeyman requires setting of the "Application Compatibility" flag:
    1. Right click in the Ikeyman icon and select "Properties"
    2. On the properties dialog select the "Compatibility" tab
    3. In the "Compatibility mode" section of this tab tick the "Run this program 
    in compatibility mode for:" check box.
    4. Select "Windows 95" from the resultant list box.
    5. Press "OK" on the "Properties" dialog box
    6. Run Ikeyman as normal
    

    Ikeyman fails to load / crashes on PPC Linux

    Solution: Ensure a 32-bit JRE is being used

    Can't receive certificate in Ikeyman: All the signer certificates must exist in the key database

    To receive a certificate, your KeyFile must be able to validate the new certificate all the way up to a trusted root in the signer certificate section of the KDB.

    If your certificate was issued by a certificate authority that is not among the default trusted certificate authorities automatically included in new KDB files by ikeyman, you must add the certificate of the issuer (the signer) to your KDB before receiving your certificate

    Adding the signer certificate

    1. Download the Signer Certificate from your certificate authority
    2. Open your KDB, click "Signer Certificate", then "add".
    3. Enter a label and click "OK"
    4. Open the signer cert you just created, and check that the "Set the certificate as a trusted root" is checked

    If ikeyman still issues the same error message when you try to receive your certificate, use the following procedure to verify that the signer certificate is the same as the one that actually signed your certificate

    openssl x509 -text -in certificate_from_certificateauthority.crt|grep Issuer:
    openssl x509 -text -in signer_certificate.cer |grep Subject:
    If the output of the two commands isn't the same distinguished name, you are either using the wrong signer certificate or your certificate is signed using an intermediate certificate.

    Intermediate Certificates

    Some certificate authorities issue certificates that are signed by an intermediate issuer, and not one of the default trusted root CA certificates that are pre-loaded into your KDB. Your certificate authority should provide any intermediate certificates required to build the trust chain and you must add them to your KDB before receiving your signed certificate.

    Solution: To add the correct intermediate certificate(s), download all intermediate certificates from your certificate authority. Perform the steps outlined here for each certificate starting from the root CA and ending with the signer certificate that issued your certificate.

    Subject/Authority key identifier mismatch (Microsoft CA)

    If the certificate you are trying to receive or add contains an Authority Key Identifier (AKI), the issuer of this certificate in your KDB must have a Subject Key Identifier (SKI) with the same value.

    The AKI/SKI can be an arbitrary binary value, or a combination of the issuers DN and Serial Number.

    Both intermediate and end-entity certificates may contain an AKI.

    openssl x509 -in intermediate.crt -text|grep -C1 "X509v3 Authority Key Identifier:" && openssl x509 -in root.crt -text|grep -C1 "X509v3 Subject Key Identifier:"
     
                    X509v3 Authority Key Identifier:
                    keyid:7B:58:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
    
                    X509v3 Subject Key Identifier:
                    keyid:4A:D3:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
      
    

    Solution: Contact certificate authority for the proper version (possible Serial Number bump) of the issuer certificate with the matching SKI I

    Error importing pkcs7 (.p7b) file

    PKCS7 files are not supported by Ikeyman

    Solution: Ask your certificate authority for a pkcs12 formatted file or individual certificates, or unpack the PKCS7 files manually with openssl

    Ikeyman: An error occurred while inserting keys to the database

    Solution: This can occur when importing from a PKCS12 or CMS key file, onto a CMS Cryptographic Token. It is resolved by upgrading GSKit to at least 7.0.3.27 and removing gskikm.jar.

    If upgrading and removing gskikm.jar does not resolve the issue, adding the private key and signer certificates in separate steps can sometimes workaround the issue:

    1. Verify your unrestricted JCE policy files are installed
    2. Make a copy of the PKCS12 file, privkey.p12, and open the copy in ikeyman.
    3. Select "Signer Certificates" and "extract" each signer certificate necessary for your personal certificate into a file.
    4. Remove each signer certificate from the PKCS12 file.
    5. Close the PKCS12 file.
    6. Open the CMS Cryptographic Token in ikeyman and choose the applicable secondary CMS key database.
    7. Select "Signer Certificates" and "add" each extracted signer certificate.
    8. Select "Personal Certificates" and "import" from your privkey.p12

    Ikeyman: Certificate Request Not Found

    Certificates can only be received into the KDB where the request was originally created. Attempting to duplicate the parameters of a previously submitted certificate request in a new KDB will not allow you to receive the certificate in the new KDB..

    Solution: Receive certificate into the KDB that made original request or resubmit certificate request.

    Ikeyman widgets perform unexpectedly

    It has been reported that ikeyman sometimes performs erratically when it is displayed on the PC X11 Server distributed by Exceed. The behavior is not seen when used with XFree86/Xorg X11 servers running natively or under Cygwin, and is also not seen when being run in a VNC server running on the IHS system.

    Reported Symptom: When selecting a country name from the selection box, the select box may be reset to default (US)

    Solution: Display ikeyman on a different X11 server, or contact X11 server vendor.

    Ikeyman: Wrong version reported in "About" dialogue

    A file provided by the IBM JRE, gskikm.jar, can conflict with or mask GSKit maintenance applied alongside IHS. One clear sign of this problem is that the version reported in ikeyman differs from the version of GSKit installed on the system.

    Solution: Ensure the JVM being used to run Ikeyman does not have a file named gskikm.jar under $JAVA_HOME/jre/lib/ext/. If this file exists, change the file extension or move it into a different directory.

  4. Ikeyman: Can't create CMS key database, error loading native CMS library

    New private key only has 1023 bits instead of 1024

    A defect in older levels of Java causes ikeyman to create new certificates with a 1023 bit private key instead of a 1024 private bit key.

    Solution: For levels prior to IHS 6.0, the system JRE (based on JAVA_HOME environment variable) is used for key management. Upgrade to the latest service release of IBM Java 1.4.2 or 1.5

    Alternatively, install IHS 6.1 on any supported OS and use that system for key management.

    Error received during import/receive: "Error processing X509 certificate"

    This error is typically seen in Ikeyman when one or more X509 extensions present in the certificate are malformed. Some common errors follow.

    If your certificate is part of an existing key database, you can extract the certificate with the following command. Otherwise, you can analyze your certificate directly.

    gsk7cmd -cert -extract -label label -db key.kdb -pw abc123 -format ascii -target cert.arm

    Causes:

    Issuer does not exist in secondary key database while using cryptographic accelerator

    When cryptographic hardware is in use, iKeyman will sometimes report "Error processing X509 certificate" instead of "All the signer certificates must exist in the key database" when the issuer of a certificate you're receiving is not present in your secondary key file.

    Verify the issuer of the certificate you're trying to receive exists in the "Signer Certificates" section in iKeyman

    Unexpected security popup in some browsers

    If you're using an intermediate certificate (your server certificate isn't signed by one of a browser's default trusted root certificate authorities but is signed by an issuer who is in turn signed by a trusted CA), web browsers can improperly cache expired copies of the intermediate certificates. This can occur if in the past the user has clicked 'Trust site permanently' on a site that uses the same intermediate certificate authority.

    Solution: Remove the expired cached copy of the intermediate certificate from the browser's SSL configuration.

    Client Authentication is enabled but browser doesn't prompt for certificate

    During the SSL handshake, the web server informs the browser of what Certificate Authorities it trusts to assist the browser in selecting the correct client certificate. If the browser detects that none of the user's client certificates would be validated given the list of certificate authorities sent by the server, the browser will NOT prompt the user for a certificate.

    If a subset of the user's client certificates can be validated by the servers list of certificate authorities, the browser will display that partial list of certificates to the user.

    Solution: The issuer of the client certificates must be added as a trusted Certificate Authority in the servers KeyFile.

    Ikeyman: The specified database has been corrupted (Strong Encryption).

    Often PKCS12 files (or other Key Databases) use strong encryption that is not available in the default JCE policy files provided by java.

    To test if the cryptography level in your PKCS12 file exceeds the JCE defaults, use the keytool command supplied in your JRE:

    keytool -list -v -keystore /tmp/your.p12 -storetype pkcs12 -storepass password

            ... Unsupported keysize or algorithm parameters
    
    Solution: Install the appropriate JCE policy files for your JRE:
    Java 5 on all platforms, or Java 1.4.2 on AIX, Linux, Windows IBM unrestricted JCE policy files

    Java 1.4.2 on HPUX, Solaris: Sun unrestricted JCE policy files

    Continue on for another common cause of this error message

    Ikeyman: The specified database has been corrupted

    Outside of PCKS12 keystores, the most common cause of this message is using an incorrect password. If IHS can service SSL requests (even if clients terminate the handshake due to an expired individual certificate) with the KDB, that means the KDB is not corrupt and the password being provided does not matched the stashed password.

    gsk7cmd reports: "Exception in thread "main" java.lang.NoClassDefFoundError: com/ibm/asn1/ASN1Exception"

    Solution: Rename $JAVA_HOME/lib/ext/gskikm.jar to $JAVA_HOME/lib/ext/gskikm.bak as described here

    erroneously reported "A duplicate certificate already exists in the database"

    If you cannot add a CA certificate using ikeyman, and you're sure it's not actually a duplicate, try adding the CA with gsk7capicmd to check if a different error code is issued.

    gsk7capicmd -cert -add -db key.kdb -pw YOURPASSWORD -file your_issuer.arm -label myca

    If the error reported by gsk7capicmd is GSKKM_ERR_VALIDATION_KEY_SIGNATURE, and you're using GSKit 7.0.4.1 or higher, see DER encoding error below.

    Certificate validation failures with GSKit 7.0.4.1 and higher

    When validating a certificate, GSKit versions 7.0.4.1 and higher first converts the certificate to a BER encoding, the most detailed encoding of the certificate.

    Another encoding, DER, is the more commonly used as a file format for certificates. It's also the mandatory format used as input to the certificate signing process. GSkit converts the BER into DER (it may or may not have started as DER) before checking that the alleged issuer really signed this certificate.

    The DER encoding dictates that default values must not be present in the DER-encoded representation. If a certificate uses an illegally constructed DER, or if the certificate authority has signed the BER-encoded form instead of the DER-encoded form, GSKit signature validation will fail because the certificate authority has not created a proper signature for this particular certificate.

    Most X509 certificate extensions have a "Criticality" field that defaults to off. The presence of this field with a value of FALSE, when the default value is FALSE, triggers the signature error discussed above.

    $ dumpasn1 /tmp/pmr/certs/keysig-7c-vs-7d/3.der |grep -B1 -i false
     623    3:           OBJECT IDENTIFIER authorityKeyIdentifier (2 5 29 35)
     628    1:           BOOLEAN FALSE
    --
     659    3:           OBJECT IDENTIFIER subjectKeyIdentifier (2 5 29 14)
     664    1:           BOOLEAN FALSE
    --
     710    3:           OBJECT IDENTIFIER certificatePolicies (2 5 29 32)
     715    1:           BOOLEAN FALSE
    --
     960    3:           OBJECT IDENTIFIER cRLDistributionPoints (2 5 29 31)
     965    1:           BOOLEAN FALSE
    

    Solution

    Contact your certificate authority and provide them the info above to re-issue your certificate

    Errors on z/Linux using cryptographic accelerator

    Java crash in ikeyman on z/Linux when PKCS11 token is being manipulated

    or

    Error Message: "Cryptographic token initialization failed. Cryptographic token support will not be available."

    Causes

    During configuration using the pkcsconf tool, some error messages may not have been noticed and the "user PIN" may not have been set after being initialized. The PIN may also have been locked due to too many failing password attempts.

    If ikeyman crashes due to this issue, the top of the javacore backtrace will look like this:

    com.ibm.gsk.ikeyman.basic.CryptographicToken.c_BuildKeyLabelList(Native Method)

    Solution

    The user PIN must be changed/set (pkcsconf -p) after being initialized (pkcsconf -u). If the user PIN is locked due too many incorrect passwords, the token must be re-initialized (pkcsconf -I)

    See the following redbook: Using Cryptographic Adapters for Web Servers with Linux on IBM System z9 and zSeries

    Detecting an improperly initialized cryptographic token

    Note: all examples assume the first token is being used, so each pkcsconf invocation contains "-c 0".

    Run the following command in a terminal:

    $ pkcsconf -t -c 0| grep Flags

    If the output matches any of the following patterns, your PKCS11 token is not usable by IHS because it has not been configured properly.

    • The 'n' digits below are placeholders that show the column of interest (5th/6th digit counting from the right).
    • No leading digits need to be present when a user or security officer PIN error is indicated by the 5th or 6th digit; more significant digits may be present.
    outputexplanationaction (see list below)
    Flags: 0xnCnnnn USER_PIN_LOCKED and USER_PIN_TO_BE_CHANGED 3,4
    Flags: 0xn8nnnn USER_PIN_TO_BE_CHANGED4
    Flags: 0xn4nnnn USER_PIN_LOCKED3,4
    Flags: 0xCnnnnn SO_PIN_LOCKED and SO_PIN_TO_BE_CHANGED1,2,3,4
    Flags: 0x8nnnnn SO_PIN_TO_BE_CHANGED 2
    Flags: 0x4nnnnn SO_PIN_LOCKED 1,2,3,4

    For more information about the Flag values see this file

    Solutions (make sure these complete succesfully)

    1. To correct SO_PIN_LOCKED, contact a system administrator and have them run: (Note: This may destroy corresponding personal certificates)
      pkcsconf -I -c 0
    2. To correct SO_PIN_TO_BE_CHANGED, change (set) the Security Officer PIN:
      pkcsconf -P -c 0
    3. To correct USER_PIN_LOCKED, contact a system administrator or initialize the user PIN (may destroy corresponding personal certificates):
      pkcsconf -u -c 0
    4. To correct USER_PIN_TO_BE_CHANGED, change (set) the user PIN:
      pkcsconf -p -c 0
      Note: Solution 2 uses a upper case 'P' and solution 4 uses an lower case 'p'.

    An example of a properly configured token is:

    Flags: 0x44D

    Examples of improperly configured token is:

    • Flags: 0x8044D
    • Flags: 0xC044D
    • Flags: 0xA044D

    Cryptographic Token runtime errors

    • SSL0154E: Initialization error, A PKCS#11 token is not present for the slot.

      Verify the SSLServerCert directive starts with the same "Label" as displayed by

      pkcsconf -t

    • [error] SSL0155E: Initialization error, The password/pin to access the PKCS#11 token is invalid.

      The file pointed to by the SSLStashFile directive, created with the <ihsinst>/bin/sslstash utility, does not contain the same password as the user PIN associated with the token indicated in SSLServerCert directive.

      Solution: Open your cryptographic token in ikeyman and verify the token name and label for your personal cert match your SSLServerCert directive in httpd.conf. Use the <ihsinst>/bin/sslstash utility to re-stash the User PIN to the file referenced by your SSLStashFile directive.

      You can query the cryptographic token on the command line, substituting your token label and password, with the command below. To be usable with IHS, you should have at least one result that has a private key (displayed with a dash in the first column)

      # LD_PRELOAD=/usr/lib/libcrypto.so.0.9.7 gsk7capicmd -cert -list -crypto /usr/lib/pkcs11/PKCS11_API.so -tokenlabel YOURTOKENLABEL -pw YOURUSERPIN
      Certificates found:
      * default, - has private key, ! trusted
              myca
      -       mylabel
      

      If no personal certificates exist, they must be created via ikeyman before IHS can use the crypto card.

    Cannot import certificate encrypted with a password that differs from keystore password

    Some non-GSKit certificate management utilities allow you to create private keys with one password and store them in a keystore with a different password. In GSKit utilities, it is assumed the private key and keystore password are the same.

    If you try to import a personal certificate of this type, GSKit will report that the private key is corrupted or unsupported, because it tries to decrypt it with the keystore password.

    The only known workaround is to use whatever native tool created the keystore and change the passwords.

    Cannot renew Verisign certificate

    When issuing a certificate, Verisign may add custom text to the Distinguished Name of the requested certificate in several Organizational Unit (OU) fields. When you renew this certificate via ikeyman, the Certificate Signing Request (CSR) will include the additional OU fields originally added by Verisign.

    Verisign may reject the CSR because it already explicitly contains the Verisign OU.

    Solution: Create a new certificate signing request instead of clicking "renew" in Ikeyman.

    Failure validating certificate issued by GPKI certificate authority

    GPKI is an SSL certificate standard published by the government of Japan that deviates from the two standards supported by IHS and the Tivoli Global Security Kit (GSKit). One such deviation that does not pass validation is an issuer chain with both a critical "Certificate Policies" (or any other RFC3280-specific) extension and a non-critical "Basic Constraints" extension

    The presence of an RFC3280-specific critical extension (e.g. Certificate Policies) anywhere in the validation chain forces GSKit to validate the certificate using RFC3280/PKIX rules, however PKIX rules state that issuer certificates MUST set the "Basic Constraints" extension as critical. These certificates fail validation. IHS 6.1.0.9 and later can support this specific deviation in otherwise RFC3280-compliant GPKI issuer certificates with the directive SSLAllowNonCriticalBasicConstraints.

    To determine if SSLAllowNonCriticalBasicConstraints is required for a specific server or client certificate, inspect the fields in each Certificate Authority (including intermediates) and look for BOTH of the following in the validation chain:

    Example Configuration:

    <VirtualHost *:443>
    SSLEnable
    SSLAllowNonCriticalBasicConstraints on
    </VirtualHost>
    

    Adding certificates before they're valid

    If an end-entity or issuer certificate is created with its beginning validity date in the future, it cannot be added to a KeyFile via ikeyman.

    Hangs or delays using key management tools and VMware

    On systems running inside of VMware, ikeyman and related tools can encounter a shortage of random data and appear to hang after many certificate operations.

    Solution: Perform complicated key management tasks on a native platform or retry with the following VMware configuration option set to 'false'

    monitor_control.virtual_rdtsc
  5. Gathering documentation for IHS support when problem is not a known issue

    Supplemental data to collect for cryptographic token related errors

    If a z/Linux crypto accelerator is being used, collect the following additional doc:

    Ikeyman problems

    1. Configure environment for Ikeyman trace
    2. Restart Ikeyman
    3. Recreate the problem and record the exact steps you take as well as the difference in expected/observed results into a text file (steps.txt). Append the KDB password to the list of steps.
    4. Send the following to IBM support:
      • ikm*.log in working directory.
      • KDB file in use along with accompanying .sth/.crl/.rdb files.
      • steps.txt, which contains the steps and expected/observed results as well as the KDB password.
      • httpd.conf
      • Any intermediate certificate provided by Certificate Authority
      • Details of cryptographic token configuration described above, when appropriate.

    IHS startup or request processing problems

    1. Set LogLevel to Debug and SSLTracein httpd.conf.
    2. Configure environment for GSKit trace
    3. Restart IBM HTTP Server.
    4. Start an iptrace that will show the interaction between the web browser and web server. This can be taken from the IHS server, the client machine, or another machine on the network. (Possible tools include sniffer, Network Monitor, or Ethereal.)
    5. Recreate the problem from the browser.
    6. Save a screen capture of the browser window.
    7. Send the following to IBM support:
      • web server error log, access log, and httpd.conf
      • Output from system and web server information, from ihsdiag collector tool
      • GSKit trace file
      • If a client certificate is in use, please include it along with any necessary CAs
      • IP trace file, unformatted
      • KDB file in use along with accompanying .sth/.crl/.rdb files as well as KDB password
      • Detail on time of request/handshake and IP address of client
      • Description of client and server trust chain with Certificate Authority in use
      • Details of cryptographic token configuration described above, when appropriate.