GSK_ERROR in the WebSphere Application Server plug-in log
 Technote (troubleshooting)
 
Problem(Abstract)
The WebSphere® Application Server HTTP server plug-in log records the following error when trying to open an SSL connection to the backend server:

ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414)

Note: This error might vary slightly, depending on the certificate presented and version of GSKit being used.
 
Cause
Set the plugin-cfg.xml Loglevel to Trace and restart the Web server. Then try an HTTPS request to recreate the problem.

The following is an example of what would be seen in the plug-in trace showing the GSK_ERROR:

TRACE: ws_common: websphereFindTransport: Setting the transport(case 1): AppserverHostName on port 9443
TRACE: ws_common: websphereExecute: Executing the transaction with the app server
TRACE: ws_common: websphereGetStream: Getting the stream to the app server
TRACE: ws_transport: transportStreamDequeue: Checking for existing stream from the queue
TRACE: lib_stream: openStream:
Opening the stream
TRACE: lib_stream: openStream:
 Stream is SSL
ERROR: lib_stream: openStream:
Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414)  **
TRACE: lib_stream: destroyStream: Destroying the stream
ERROR: ws_common: websphereGetStream: Could not open stream
ERROR: ws_common: websphereExecute: Failed to create the stream

The preceding log shows that the plug-in used the application server HTTPS transport port of 9443. Then, upon opening the stream or connection by way of SSL, the plug-in was presented with a certificate that could not be authenticated by the key file of the plug-in. The following are two possible causes:
  1. There was not an equivalent signer within the plug-in key file to authenticate the personal certificate sent by the application server.

  2. The personal certificate sent by the application server has expired.

Note: The exact error message can be different based on WebSphere Application Server version.
 
Resolving the problem
First, you must determine what personal certificate the application server is sending to the plug-in key file. The easiest way to view this is to bypass the plug-in and make the HTTPS request directly to the application server. Based on the preceding example, the request would be sent to the 9443 port of the application server.
  1. Use a browser to connect to the following URL:

    https://AppserverHostName:9443/SomeURI

    Note: You must have AppserverHostName:9443 configured as a virtual host alias in order to access the embedded port directly from a browser.

  2. When prompted with the security alert, click View Certificate.


    This resulting screen provides the validity dates for the certificate. In this example, the certificate is still valid. It also shows that the certificate is not trusted. Since both Issued to and Issued by are the same name, this must be a Self-Signed Certificate.



Based on the information gathered to this point, a trusted relationship between the plug-in and application server key files needs to be created. The following will provide the necessary steps to create this trusted relationship:
  1. Locate the WebSphere Keystore that holds the personal certificate that is being sent to the plug-in for authentication. Go to Locating WebSphere Keystore below for detailed instructions and the example location. According to our example (below) /etc/DummyServerKeyFile.jks is the file you need to use.

  2. Open this /etc/DummyServerKeyFile.jks keystore file with the Appserver/bin/ikeyman tool. The default password to open this file is WebAS (case sensitive).

  3. Select Personal Certificates within the Key database content window.

  4. Within that listing, you will see the "test" certificate. This is the self-signed certificate that was issued and signed by jserver. Click Extract Certificate.

  5. A new popup will appear asking for a Certificate file name and Location. Fill both in making sure that the Data Type is Base64-encoded ASCII data and the file extension is .arm.



The next step is to add the newly extracted ARM file into the plug-in keystore as a signer:
  1. Use the following instructions labeled Locating Plugin Keystore to determine which file needs to have the signer added.

  2. Use the following command to open the keystore for the plug-in:

    GSKit_install_root/bin gsk#ikm

  3. Select Signer Certificates from the Key database content list box.

  4. Click Add.

  5. Make sure the Data type field is set to Base64-encoded ASCII data.

  6. Click Browse and locate the previously created ARM file.

  7. When prompted, type in a Label name for the new signer. It can be anything you want.


  8. Once the cert.arm file has been added, you will have to restart the Web server and try the https connection again.

    Note: If this still results in failure, you will need to capture a plug-in and GSKit trace from the Web server. Instructions are below.




Locating the Plug-in Keystore
You can locate the keydatabase in the following location within the plugin-cfg.xml file:

<Transport Hostname="appserver.com" Port="9443" Protocol="https">
  <Property Name="keyring" Value="/opt/WebSphere/AppServer/etc/plugin-key.kdb"/>
  <Property Name="stashfile" Value="/opt/WebSphere/AppServer/etc/plugin-key.sth"/>
</Transport>



Locating the WebSphere Keystore
Note from this point forward that the ${USER_INSTALL_ROOT} will be the WebSphere Application Server home directory which can be confirmed from the variables.xml file. The following examples might be different per installation location. The following is just from the test system used.
  • For releases of WebSphere Application Server V6.0
    1. You will have to find this by the nodes serverindex.xml file, for example:

      /opt/was60base/profiles/default/config/cells/
      lab3Node01Cell/nodes/lab3Node01/serverindex.xml

      Locate the following end point:

      <specialEndpoints xmi:id="NamedEndPoint_1119918320416" endPointName="WC_defaulthost_secure">
        <endPoint xmi:id="EndPoint_1119918320416" host="*"
      port="9443"/>
      </specialEndpoints>

    2. Now find the endPointName="WC_defaulthost_secure" string from previous in the server.xml under the "transportChannels": (for example within)

      /opt/was60base/profiles/default/config/cells/lab3Node01Cell/nodes/lab3Node01/servers/server1/server.xml

      <transportChannels xmi:type="channelservice.channels:TCPInboundChannel" xmi:id="TCPInboundChannel_1119918321703" name="TCP_4" endPointName="WC_defaulthost_secure" maxOpenConnections="20000" inactivityTimeout="60" threadPool="ThreadPool_1119918321701"/>


    3. You will then need to take the TCPInboundChannel_1119918321703 set on the Channel with that endPointName="WC_defaulthost_secure"

    4. Again within the same server.xml find the TCPInboundChannel_1119918321703 "Chain";

      <chains xmi:id="Chain_1119918321709" name="WCInboundDefaultSecure" enable="true" transportChannels="TCPInboundChannel_1119918321703 SSLInboundChannel_1119918321702 HTTPInboundChannel_1119918321706 WebContainerInboundChannel_1119918321707"/>


    5. *IF you do NOT see a "Chain" with the "SSLInboundChannel_" set, keep looking. The "Chain" without this is simply an unsecure connection chain.

    6. Within the same server.xml take the SSLInboundChannel_1119918321702 and find the "transportChannels" that contains this. Now you have the sslConfigAlias="lab3Node01/DefaultSSLSettings" which will allow you to find the sslConfigAlias="lab3Node01/DefaultSSLSettings" for that port.

      <transportChannels xmi:type="channelservice.channels:SSLInboundChannel" xmi:id="SSLInboundChannel_1119918321702" name="SSL_2" discriminationWeight="1" sslConfigAlias="lab3Node01/DefaultSSLSettings"/>



    7. Now you have the sslConfigAlias="lab3Node01/DefaultSSLSettings string to look in the security.xml to nail down the true Java key store used; /opt/was60base/profiles/default/config/cells/lab3Node01Cell/security.xml

      <repertoire xmi:id="SSLConfig_1" alias="lab3Node01/DefaultSSLSettings">
      <setting xmi:id="SecureSocketLayer_1"
       keyFileName="${USER_INSTALL_ROOT}/etc/DummyServerKeyFile.jks" keyFilePassword="{xor}CDo9Hgw=" keyFileFormat="JKS" trustFileName="${USER_INSTALL_ROOT}/etc/DummyServerTrustFile.jks" trustFilePassword="{xor}CDo9Hgw=" trustFileFormat="JKS" clientAuthentication="false" securityLevel="HIGH" enableCryptoHardwareSupport="false">


  • For releases of WebSphere Application Server V5.1 and V5.0
    1. Open the server.xml file found in the following location:

      /opt/WebSphere/AppServer/config/cells/lab3/nodes/lab3/servers/server1/server.xml

    2. Look for the port in questions transports xmi:type line like below;

      <transports xmi:type="applicationserver.webcontainer:HTTPTransport" xmi:id="HTTPTransport_2" sslEnabled="true" sslConfig="lab3/DefaultSSLSettings">
      <address xmi:id="EndPoint_2" host=""
       port="9443"/>

    3. Use the lab3/DefaultSSLSettings information associated to the sslConfig line. Take the sslConfig= information, here is the node name lab3/DefaultSSLSettings and look for this name in the following file:

      /opt/WebSphere/AppServer/config/cells/lab3/security.xml

      You will find the application server SSL repertoire:

      <repertoire xmi:id="SSLConfig_1" alias="lab3/DefaultSSLSettings">
      <setting xmi:id="SecureSocketLayer_1" keyFileName="${USER_INSTALL_ROOT}
      /etc/DummyServerKeyFile.jks" keyFilePassword="{xor}CDo9Hgw=" keyFileFormat="JKS" trustFileName="${USER_INSTALL_ROOT}/etc/DummyServerTrustFile.jks" trustFilePassword="{xor}CDo9Hgw=" trustFileFormat="JKS" clientAuthentication="false" securityLevel="HIGH" enableCryptoHardwareSupport="false">

      Within this key file you will find the "jserver" certificate that was presented to the plug-in for authentication.


GSKit Trace Instructions
Go to the following URL for GSKIT trace instructions. MustGather: IBM HTTP Server SSL handshake and configuration problems. When you have collected the GSKit trace per the instructions outlined in the MustGather, send in all required documents from those instructions. Also include the server.xml and security.xml files. Use WebSphere support Open service request to contact IBM WebSphere Support.
 
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers WebSphere Application Server - Express Plugin AIX 6.0, 5.1, 5.0
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Plug-in
Operating system(s): AIX
Software version: 6.0
Software edition:
Reference #: 1215867
IBM Group: Software Group
Modified date: Aug 31, 2005