ProblemDetermination: Java Secure Socket Extension and SSL problems
 Technote (troubleshooting)
 
Problem(Abstract)
This document should is designed to assist with Problem Determination in the analysis of Java™ Secure Socket Extension (JSSE) and Secure Socket Layer (SSL) problems for IBM® WebSphere® Application Server V5.0 and V5.1. Covering SSL handshake steps basic configuration files and reading a JSSE trace.
 
Resolving the problem
Problem determination documents provide in-depth review and investigation into the information collected by JSSE MustGather
SSL handshake steps
  1. Client sends server version information and ciphers.

  2. Server sends client version information, ciphers, and its certificate (keystore on server).

  3. Client authenticates server using the server’s certificate (truststore on client).

  4. Client generates premaster secret and sends it to the server
    1. If client authentication is enabled, the client sends its certificate at this point along with the premaster secret (keystore on client).

    2. The server authenticates the client using its certificate (truststore on server).

  5. The server and client follow steps to create a master secret using the premaster secret sent by the client.

  6. The master secret is used to create session keys (symmetric) to be used to verify the data being between the client and server.

  7. Client and Server send message created by session key to and indicate that the handshake is complete.


WebSphere Dummy SSL key files
The following files are located in the directory install_root/etc:
  • DummyServerKeyFile.jks
    Contains the private key for the Application Server.

  • DummyServerTrustFile.jks
    Contains signer certificates for EJB, HTTPS, or LDAP servers communicating with the Application Server.

  • DummyClientKeyFile.jks
    Contains a private key for commands or stand-alone clients that use sas.client.props.

  • DummyClientTrustFile.jks
    Contains the public key of the Application Server.

  • plugin-key.kdb
    Contains the public key of the Application Server and private key for the plug-in.

  • cacerts
    Applications making outbound JSSE calls to HTTPS servers will use the
    install_root/java/jre/lib/security/cacerts file as the keystore/truststore.

Technotes that describe how to replace the Dummy file and cacerts.


WebSphere Client Property Files
  • ORB Clients use SSL keys defined in install_root/properties/sas.client.props

    Examples: dumpNameSpace

  • SOAP clients use SSL keys defined in install_root/properties/soap.client.props

    Examples: WSAdmin

The location of the sas.client.props/soap.client.props is set at runtime by setupCmdLine:

SET CLIENTSAS=
-Dcom.ibm.CORBA.ConfigURL=file:/%WAS_HOME%/properties/sas.client.props
SET CLIENTSOAP=
-Dcom.ibm.SOAP.ConfigURL=file:/%WAS_HOME%/properties/soap.client.props


Reading JSSE trace
The majority of client problems with certificates and SSL is "Step 3: Client authenticates server using the server’s certificate (truststore on client)". This is where the "Alert: fatal, unknown certificate" or "expired certificate" errors happen. Below outlines important information to reviewing in the trace to debug these problems. This example is wsadmin (java client) uses SOAP protocol to talk with the dmgr.


Java Client trace
The following files are used for the Java client:
  • Trust and key files loaded

    TrustManagerFactoryImpl: trustStore is : C:/WebSphere51/DeploymentManager/etc/DummyClientTrustFile.jks
    TrustManagerFactoryImpl: trustStore type is : jks
    ...
    KeyManagerFactoryImpl: keyStore is : C:/WebSphere51/DeploymentManager/etc/DummyClientKeyFile.jks
    KeyManagerFactoryImpl: keyStore type is : jks

  • Server Certificate transferred to be verified by Client (trustStore)

    SSLSocket: using raw socket <port=18889,localport=1505>          JSSE Start point, note the port matches the server port (SOAP)
    ...
    >> serverCertificate.                                               Actual Certificate used needed in trustStore
    JSSEContext:  confirmPeerCertificate[Socket[addr=localhost/127.0.0.1,port=18889,localport=1505]]
    X509TrustManagerImpl: checkServerTrusted
    X509TrustManagerImpl: Certificate [
    [
      Version: V3
      Subject: CN=jserver, OU=SWG, O=IBM, C=US
      Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4

      Key:  IBMJCE RSA Public Key:
    modulus:
    9278462158268216672835140907586697128358482398420257066366912
    0518370819221770454928416386278888362102796841959012400833673
    1183973950955566673479462018528554376816840646568455236994259
    0671394251276774813634851728131127148802235766140248068823684
    3830996347540486698724905002161313848064418923142829862028699
    247

    public exponent:
    65537

      Validity: [From: Wed Jul 30 16:39:20 EDT 2003,
                   To: Wed Oct 13 16:39:20 EDT 2021]         <--  date and time when this expires
      Issuer: CN=jserver, OU=SWG, O=IBM, C=US
      SerialNumber: [1059597560]                  <-- Decimal serial number that correlates to the trustStore certificate in Hexadecimal

    ]
      Algorithm: [MD5withRSA]
      Signature:
    0000: 10 f0 91 c0 45 01 c7 3e  be f4 0e 7f 02 83 c1 2e  ....E...........
    0010: 26 08 9e ba 30 2d f6 3e  c3 7c 49 c8 30 a2 7b e1  ....0.....I.0...
    0020: a7 78 b9 e2 ac bf b5 1d  74 41 6c d3 89 61 04 c2  .x......tAl..a..
    0030: a7 33 74 d0 28 2a d8 cf  cb 53 ab 54 79 e6 50 de  .3t......S.Ty.P.
    0040: 0c 22 76 2a 85 51 cb a5  8b 66 7e f3 07 7a d2 af  ..v..Q...f...z..
    0050: 2f cb aa 9d 27 59 2f 83  d6 ad c0 c7 50 57 62 c5  .....Y......PWb.
    0060: 7f b4 cf 35 c7 26 50 b0  fd fc b5 7d 87 70 15 a0  ...5..P......p..
    0070: 16 ec 16 e9 19 02 ba 05  d1 9b ce fb fe 2f ac 89  ................
    ]
    ...
    JSSEContext: handleConnection[Socket[addr=localhost/127.0.0.1,port=18889,localport=1505]]  
    >> handleData <com.ibm.sslite.p@22976c47>
    >> handleAlert <com.ibm.sslite.p@22976c47>
    Alert: warning, close notify    <-- Client response to processed certificate
    << sendAlert.
    Alert: warning, close notify
    SSLConnection: terminate - RESUMABLE <com.ibm.sslite.p@22976c47>
                                            JSSE End point


WebSphere Application Server trace
Note to follow the same thread in the trace when debugging.
  • Dmgr trace

    [5/10/05 11:30:18:746 EDT] 6cc6ec3f SystemOut     O SSLSocket: using raw socket <port=1505,localport=18889>                           JSSE Start point, note the port matches the client
    ...
    [5/10/05 11:30:18:746 EDT] 6cc6ec3f SystemOut     O << sendCertificate.            <-- Server Certificate was sent the client  (SSL step 2)
    ...
    [5/10/05 11:30:19:337 EDT] 6cc6ec3f SystemOut     O Alert: warning, close notify    <-- Client response to processed certificate
    [5/10/05 11:30:19:357 EDT] 6cc6ec3f SystemOut     O SSLInputRecordLayer: close <com.ibm.sslite.q@448aac27>
    [5/10/05 11:30:19:357 EDT] 6cc6ec3f SystemOut     O SSLOutputRecordLayer: close <com.ibm.sslite.q@448aac27>
    [5/10/05 11:30:19:357 EDT] 6cc6ec3f SystemOut     O SSLSocket: closing raw socket <port=1505,localport=18889>
    [5/10/05 11:30:19:377 EDT] 6cc6ec3f SystemOut     O SSLInputRecordLayer: terminate <com.ibm.sslite.q@448aac27>
    [5/10/05 11:30:19:377 EDT] 6cc6ec3f SystemOut     O SSLConnection: terminate - RESUMABLE <com.ibm.sslite.q@448aac27>
                                            JSSE End point
 
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Java Security (JSSE/JCE)
Operating system(s): AIX
Software version: 5.1
Software edition:
Reference #: 1208858
IBM Group: Software Group
Modified date: Jun 9, 2005