GSK_ERROR in the WebSphere Application Server plug-in log
 Technote (FAQ)
 
Problem
The WebSphere® Application Server HTTP server plug-in log records the following error when trying to open an SSL connection to the backend server:

ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414)

Note: This error might vary slightly, depending on the certificate presented and version of GSKit being used.
 
Cause
Set the plugin-cfg.xml Loglevel to Trace and restart the Web server. Then try an HTTPS request to recreate the problem.

The following is an example of what would be seen in the plug-in trace showing the GSK_ERROR:

TRACE: ws_common: websphereFindTransport: Setting the transport(case 1): AppserverHostName on port 9443
TRACE: ws_common: websphereExecute: Executing the transaction with the app server
TRACE: ws_common: websphereGetStream: Getting the stream to the app server
TRACE: ws_transport: transportStreamDequeue: Checking for existing stream from the queue
TRACE: lib_stream: openStream:
Opening the stream
TRACE: lib_stream: openStream:
 Stream is SSL
ERROR: lib_stream: openStream:
Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414)  **
TRACE: lib_stream: destroyStream: Destroying the stream
ERROR: ws_common: websphereGetStream: Could not open stream
ERROR: ws_common: websphereExecute: Failed to create the stream

The preceding log shows that the plug-in used the application server HTTPS transport port of 9443. Then, upon opening the stream or connection by way of SSL, the plug-in was presented with a certificate that could not be authenticated by the key file of the plug-in. The following are two possible causes:
  1. There was not an equivalent signer within the plug-in key file to authenticate the personal sent by the application server.

  2. The personal certificate sent by the application server has expired.

Note: The exact error message can be different based on WebSphere Application Server version.
 
Solution
First, you must determine what personal certificate the application server is sending to the plug-in key file. The easiest way to view this is to bypass the plug-in and make the HTTPS request directly to the application server. Based on the preceding example, the request would be sent to the 9443 port of the application server.
  1. Use a browser to connect to the following URL:

    https://AppserverHostName:9443/SomeURI

    Note: You must have AppserverHostName:9443 configured as a virtual host alias in order to access the embedded port directly from a browser.

  2. When prompted with the security alert, click View Certificate.


    This resulting screen provides the validity dates for the certificate. In this example, the certificate is still valid. It also shows that the certificate is not trusted. Since both Issued to and Issued by are the same name, this must be a Self-Signed Certificate.



Based on the information gathered to this point, a trusted relationship between the plug-in and application server key files needs to be created. The following will provide the necessary steps to create this trusted relationship:
  1. Locate the WebSphere Keystore that holds the personal certificate that is being sent to the plug-in for authentication. Go to Locating WebSphere Keystore below for detailed instructions and the example location. According to our example (below) /etc/DummyServerKeyFile.jks is the file you need to use.

  2. Open this /etc/DummyServerKeyFile.jks keystore file with the Appserver/bin/ikeyman tool.

  3. Select Personal Certificates within the Key database content window.

  4. Within that listing, you will see the "test" certificate. This is the self-signed certificate that was issued and signed by jserver. Click Extract Certificate.

  5. A new popup will appear asking for a Certificate file name and Location. Fill both in making sure that the Data Type is Base64-encoded ASCII data and the file extension is .arm.



The next step is to add the newly extracted ARM file into the plug-in keystore as a signer:
  1. Use the following instructions labeled Locating Plugin Keystore to determine which file needs to have the signer added.

  2. Use the following command to open the keystore for the plug-in:

    GSKit_install_root/bin gsk#ikm

  3. Select Signer Certificates from the Key database content list box.

  4. Click Add.

  5. Make sure the Data type field is set to Base64-encoded ASCII data.

  6. Click Browse and locate the previously created ARM file.

  7. When prompted, type in a Label name for the new signer. It can be anything you want.


  8. Once the cert.arm file has been added, you will have to restart the Web server and try the https connection again.

    Note: If this still results in failure, you will need to capture a plug-in and GSKit trace from the Web server. Instructions are below.




Locating Plugin Keystore
You can locate the keydatabase in the following location within the plugin-cfg.xml file:

<Transport Hostname="appserver.com" Port="9443" Protocol="https">
 <Property Name="keyring" Value="/opt/WebSphere/AppServer/etc/plugin-key.kdb"/>
 <Property Name="stashfile" Value="/opt/WebSphere/AppServer/etc/plugin-key.sth"/>
</Transport>



Locating WebSphere Keystore
Note from this point forward that the ${USER_INSTALL_ROOT} will be the WebSphere Appserver home directory which can be confirmed from the variables.xml. The examples below maybe different per installation location, the following is just from the test system used.
  • For releases of WebSphere Application Server V5.0 and V5.1
    1. Open the server.xml file found in the following location:

      /opt/WebSphere/AppServer/config/cells/lab3/nodes/lab3/servers/server1/server.xml

    2. Look for the port in questions transports xmi:type line like below;

      <transports xmi:type="applicationserver.webcontainer:HTTPTransport" xmi:id="HTTPTransport_2" sslEnabled="true" sslConfig="lab3/DefaultSSLSettings">
      <address xmi:id="EndPoint_2" host="" port="9443"/>

    3. Use the "lab3/DefaultSSLSettings" information associated to the sslConfig line, Take the sslConfig= info, here is the node name lab3/DefaultSSLSettings
      and look for this name in the
      /opt/WebSphere/AppServer/config/cells/lab3/security.xml
      you will find the app server ssl repertoire;
      <repertoire xmi:id="SSLConfig_1" alias="lab3/DefaultSSLSettings">
      <setting xmi:id="SecureSocketLayer_1" keyFileName="${USER_INSTALL_ROOT}/etc/DummyServerKeyFile.jks" keyFilePassword="{xor}CDo9Hgw=" keyFileFormat="JKS" trustFileName="${USER_INSTALL_ROOT}/etc/DummyServerTrustFile.jks" trustFilePassword="{xor}CDo9Hgw=" trustFileFormat="JKS" clientAuthentication="false" securityLevel="HIGH" enableCryptoHardwareSupport="false">
      and within this key file you will find the "jserver" certificate that was presented to the plugin for authentication.

  • For releases of WebSphere Application Server V6.0
    1. You will have to find this by the nodes serverindex.xml: (for example)

      /opt/was60base/profiles/default/config/cells/lab3Node01Cell/nodes/lab3Node01/serverindex.xml

      <specialEndpoints xmi:id="NamedEndPoint_1119918320416" endPointName="WC_defaulthost_secure">
      <endPoint xmi:id="EndPoint_1119918320416" host="*" port="9443"/>
      </specialEndpoints>

    2. Now find the endPointName="WC_defaulthost_secure" string from above In the server.xml under the "transportChannels": (for example within)

      /opt/was60base/profiles/default/config/cells/lab3Node01Cell/nodes/lab3Node01/servers/server1/server.xml

      <transportChannels xmi:type="channelservice.channels:TCPInboundChannel" xmi:id="TCPInboundChannel_1119918321703" name="TCP_4" endPointName="WC_defaulthost_secure" maxOpenConnections="20000" inactivityTimeout="60" threadPool="ThreadPool_1119918321701"/>

    3. You will then need to take the TCPInboundChannel_1119918321703 set on the Channel with that endPointName="WC_defaulthost_secure"

    4. Again within the same server.xml find the TCPInboundChannel_1119918321703 "Chain";

      <chains xmi:id="Chain_1119918321709" name="WCInboundDefaultSecure" enable="true" transportChannels="TCPInboundChannel_1119918321703 SSLInboundChannel_1119918321702 HTTPInboundChannel_1119918321706 WebContainerInboundChannel_1119918321707"/>

    5. *IF you do NOT see a "Chain" with the "SSLInboundChannel_" set then keep looking, the "Chain" without this is simply an unsecure connection chain.

    6. Within the same server.xml take the SSLInboundChannel_1119918321702 and find the "transportChannels" that contains this and now we have the sslConfigAlias="lab3Node01/DefaultSSLSettings" which will allow us to find the sslConfigAlias="lab3Node01/DefaultSSLSettings" for that port.

      <transportChannels xmi:type="channelservice.channels:SSLInboundChannel" xmi:id="SSLInboundChannel_1119918321702" name="SSL_2" discriminationWeight="1" sslConfigAlias="lab3Node01/DefaultSSLSettings"/>

    7. Now we have the sslConfigAlias="lab3Node01/DefaultSSLSettings string to look in the security.xml to nail down the true Java key store used; /opt/was60base/profiles/default/config/cells/lab3Node01Cell/security.xml

      <repertoire xmi:id="SSLConfig_1" alias="lab3Node01/DefaultSSLSettings">
      <setting xmi:id="SecureSocketLayer_1" keyFileName="${USER_INSTALL_ROOT}/etc/DummyServerKeyFile.jks" keyFilePassword="{xor}CDo9Hgw=" keyFileFormat="JKS" trustFileName="${USER_INSTALL_ROOT}/etc/DummyServerTrustFile.jks" trustFilePassword="{xor}CDo9Hgw=" trustFileFormat="JKS" clientAuthentication="false" securityLevel="HIGH" enableCryptoHardwareSupport="false">


GSKIT TRACE INSTRUCTIONS
Go to the following URL for GSKIT trace instructions.

http://www.ibm.com/support/docview.wss?rs=177&uid=swg21141302
When you have collected the GSKit trace per the instructions outlined in the URL above, please send in all required docs from those instructions. Please also include the Server.xml and Security.xml files. Then call in to the 800-IBM-SERV number to open a new PMR.
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Plug-in
Operating system(s): AIX
Software version: 4.0
Software edition:
Reference #: 1215867
IBM Group: Software Group
Modified date: Aug 31, 2005