Troubleshooting tips
Tips for troubleshooting Liberty.
To help you identify and resolve problems, the product has a unified logging component. See Logging and Trace. You can also use theIBM® Support Assistant Data Collector (ISADC) command tool in the ${wlp.install.dir}/bin directory to quickly collect diagnostic files, such as log files, configuration files or to run traces.
If you receive an exception message, information about the message is available in Messages.
The Java™ API documentation for each Liberty API is detailed in the Programming interfaces (Javadoc) section of the online IBM Knowledge Center, and is also available as a separate .zip file in one of the javadoc subdirectories of the ${wlp.install.dir}/dev directory.

For details of the main known restrictions that apply when using Liberty,
see Runtime environment known issues and restrictions.
- Troubleshooting JDKs
- Troubleshooting security
- Troubleshooting LDAP
- Troubleshooting SSL
- Troubleshooting CORBA/IIOP
- Troubleshooting logging and tracing
Applying fix packs and interim fixes to an archive install
- Troubleshooting performance
- Troubleshooting collectives
- Troubleshooting SAML
- Troubleshooting REST API Discovery
Check that your JDKs are at a supported level
If you experience problems that are not readily explained, check that the JDKs you are using are compliant with Java Version 7 or later, and are at a current service level. See Minimum supported Java levels.
Troubleshooting security
This section describes some common security problems and solutions you can choose.
- SESN0008E: A user authenticated as anonymous has attempted to access a session owned by user:LdapRegistry/cn=steven,o=myCompany,c=US.
- This error happens when an unauthenticated user tries to access a session created by an authenticated user. Session security that is enabled by default prevents unauthorized access of the sessions. Only the user who created a session can access it. See session security for more information.
This error can happen when you use a JSP (login.jsp for example) for your form-login and the SSO token sent by the browser is expired. Because the SSO token is expired, the user is prompted to log in again using the login.jsp page configured for the form-login. The jsp page, by default, tries to get a session. This session was originally created by the user whose token is expired. Because the token is expired and the user is not authenticated, no credentials are established when accessing this session, which results in this error.
To avoid this error, restart the browser that starts a new session, or configure the login.jsp file to not create the session by default. If you choose to update the login.jsp file, set <%@ page session="false" %>.
- CWWKS9104A: Authorization failed for user {0} while invoking {1} on {2}. The user is not granted access to any of the required roles: {3}.
- This error occurs when you do not have authorization to the roles required by the application. Make sure that the user or the group it belongs to is mapped to at least one of the roles that are mentioned in the error message. A user-to-role mapping defined in the ibm-application-bnd.xmi/xml file takes precedence over a mapping defined in the server.xml file. Check both resources to ensure that the correct mapping is defined. See Configuring authorization for applications in Liberty.
- CWWKS9104A: Authorization failed for user {0}.
- This error can occur if you specify both an application and webApplication for the same context root. If a conflict happens the latest configuration that is defined is ignored and causes an unexpected error, such as CWWKS9104A.
- CWWKZ0013E: It is not possible to start two applications called {0} followed by unexpected security behavior and error messages such as CWWKS9104A.
- This error occurs when you specify your application in both the server.xml
by using the application element and in the dropins folder. As
a result, the application is attempted to be installed twice and the security configuration in the
server.xml file might or might not take effect. To fix this, you must remove
your application from the dropins folder and copy it to another directory. If
you have to leave it in the dropins folder, you must disable the application
monitoring by using the following code in your server.xml
file:
<applicationMonitor dropinsEnabled="false"/>
- An unauthenticated user was able to access my servlet or JSP.
- A user with a principal of UNAUTHENTICATED (or the unauthenticated SAF user
on z/OS®) is created when authentication fails or when your
servlet or JSP is unprotected. An unauthenticated user can access your servlet or JSP if you do not
define any security constraints or if you map the EVERYONE special subject to
the role required by your application. Review the user-to-role mappings in the
ibm-application-bnd.xmi/xml and server.xml files. Take one
of the following options:
- If your servlet or JSP is unprotected, add security constraints to the deployment descriptor of your application. See Authentication.
- If you do not want any unauthenticated user to access your application, remove the EVERYONE special subject from the mapping for that role. See Configuring authorization for applications in Liberty.
- If a certain user cannot be authorized to your servlet or JSP, review who is mapped to that role by examining the ibm-application-bnd.xmi/xml and server.xml files. You can map a role to a user, group, or special subject. If your role is mapped to the EVERYONE special subject, any user is granted access. If your role is mapped to the ALL_AUTHENTICATED special subject, any authenticated user is granted access. Remove these special subjects if you want to further limit who can access your servlet or JSP. Finally, check what group the user belongs to. Although the user might not have explicit access, the group might be mapped to the role. In this case, remove the user from the authorized group or remove the group from role mapping. See Configuring authorization for applications in Liberty.
- Single sign-on (SSO) does not work.
- If SSO does not work, make sure that the different Liberty servers that use the same LTPA keys, password, and ssoCookieName attribute of webAppSecurity element have the same Universal Time (UTC) and share the same user registry. Also, if the token expires or if the cookie is sent from a web browser after changing the user registry in a manner that is inconsistent, such as modifying the realm or removing the user the cookie represents, the SSO fails and the user is prompted to enter the credential information again. See Customizing SSO configuration using LTPA cookies in Liberty.
- Debugging security public APIs.
- WSSecurityHelper and RegistryHelper are loaded even if security is not enabled, for example, if a security feature - appSecurity-1.0,appSecurity-2.0 or zosSecurity-1.0 - is not specified. If security is not enabled, then WSSecurityHelper.isServerSecurityEnabled() and WSSecurityHelper.isGlobalSecurityEnabled() methods both return false, and RegistryHelper.getUserRegistry() method returns null.
Other security public API classes might not be loaded if security is not enabled. If you try to access these classes and call a method on one of these classes, you might get a java.lang.NoClassDefFoundError exception.
To avoid getting java.lang.NoClassDefFoundError exceptions, you must first test to see whether security is enabled by calling the WSSecurityHelper.isServerSecurityEnabled() or WSSecurityHelper.isGlobalSecurityEnabled() class, and then call other security public API classes only when security is enabled. See Security public APIs for examples of this coding technique.
- Note: This behavior is different from the WebSphere Application Server traditional. In WebSphere Application Server traditional, all classes are always loaded regardless of whether security is enabled or not.
- Cannot authenticate users with Unicode characters
- In order to authenticate users whose names contain Unicode characters, you must set the
character encoding type used by the Liberty server to UTF-8 by adding the following jvm option to
the server start command:
-Dclient.encoding.override=UTF-8
You must also specify the charset and pageEncoding in your login page. Here is an example for specifying these parameters on a login JSP page:<%@page contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
- java.lang.annotation.AnnotationFormatError: java.lang.IllegalArgumentException:Wrong type at constant pool index at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:87)
- This error can occur when an OpenID Connect or OAuth provider uses a Database Store for client
registration with some JDK 7 versions.
To avoid this problem, upgrade to JDK version 7.1.
Troubleshooting LDAP
This section describes some common LDAP problems and solutions you can choose.
- FFDC1015I: An FFDC Incident has been created: javax.naming.ServiceUnavailableException: myldapserver.mycompany.com:636; socket closed com.ibm.ws.security.registry.ldap.internal.LdapRegistry 298
- This message in messages.log indicates that the configured LDAP server cannot be reached. Check your LDAP server to verify that it is running and that its IP address can be accessed from the Liberty server.
- The javax.naming.CommunicationException: simple bind failed: myldapserver.mycompany.com:636 [Root exception is javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.g: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target]
- If you enable SSL on your LDAP server without copying the signer of the LDAP server into the truststore referenced in the LDAPSSLSettings element, a connection with the LDAP server fails with an SSL handshake error. Make sure that you copy the signer of the LDAP server into your truststore.
- The javax.naming.CommunicationException: myldapserver.mycompany.com:389 [Root exception is java.net.BindException: Address already in use: connect]
- This message might appear in the FFDC logs and indicates that the usable sockets on the local server are exhausted, which can result in a failure when connecting to your LDAP server. Make sure that the socket is not used and try again.
- CWWKS1100A: Authentication did not succeed for user ID xxxxx. An invalid user ID or password was specified
- This FFDC exception might happen on the server even though the user mentioned in the message is a valid user on the LDAP server under heavy load. With the LDAP configuration, you can add the reuseConnection=false property or disable it by using the developer tools. To fix the problem, set this property to the default value of true.
Troubleshooting SSL
This section describes some common SSL problems and solutions you can choose.
- CWWKS9105E: Could not determine the SSL port for automatic redirection.
- This error occurs when you try to access an application that redirects to an SSL port and the SSL port is not available. The port might not be available because of a missing SSL configuration or some error in the SSL configuration definition. Check the SSL configuration in the server.xml file to make sure that it exists and is correct. See Securing communications with Liberty.
- FFDC1015I: An FFDC Incident has been created: "java.lang.IllegalArgumentException: Unknown, incomplete configuration: missing id field com.ibm.ws.config.internal.cm.ManagedServiceFactoryTracker aSyncReadNupdate. Exception thrown while trying to read configuration and update ManagedServiceFactory. Exception = java.lang.IllegalArgumentException: Unknown, incomplete configuration: missing id field" at ffdc_12.04.18_16.09.42.0.log
- This error occurs when a keystore element exists in the configuration without an ID field. If you use a minimal SSL configuration, set the ID field to defaultKeyStore.
- You might get an exception if using a LDAP user registry with sslEnabled and a sslRef value is not specified.
- For example, a configuration has sslEnabled set to true but there is not a
sslRef value, as shown in the following
example:
<ltldapRegistry id="ldap" realm="SampleLdapIDSRealm" host="ccwin12.austin.ibm.com" port="636" ignoreCase="true" baseDN="o=ibm,c=us" bindDN="cn=root" bindPassword="rootpwd" ldapType="IBM Tivoli Directory Server" idsFilters="ibm_dir_server" sslEnabled="true" searchTimeout="8m" />
You must enter a sslRef value. If you are using a minimal SSL configuration that is similar to the following:
the sslRef field must be set to defaultSSLConfig.<ltkeyStore id="defaultKeyStore" location="key.jks" password="mypassword" />
If a custom SSL configuration is configured, the name of that configuration must be placed in the sslRef field.
- If you use a JDK from the WebSphere Application Server, you might see the following error if SSL is enabled on your Liberty Server.
java.net.SocketException: java.lang.ClassNotFoundException: Cannot find the specified class com.ibm.websphere.ssl.protocol.SSLSocketFactory at javax.net.ssl.DefaultSSLSocketFactory.a(SSLSocketFactory.java:11) at javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:6) at com.ibm.net.ssl.www2.protocol.https.c.afterConnect(c.java:161) at com.ibm.net.ssl.www2.protocol.https.d.connect(d.java:36) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1184) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:390) at com.ibm.net.ssl.www2.protocol.https.b.getResponseCode(b.java:75) at com.ibm.ws.jmx.connector.client.rest.internal.RESTMBeanServerConnection.loadJMXServerInfo(RESTMBeanServerConnection.java:142) at com.ibm.ws.jmx.connector.client.rest.internal.RESTMBeanServerConnection.<init>(RESTMBeanServerConnection.java:114) at com.ibm.ws.jmx.connector.client.rest.internal.Connector.connect(Connector.java:315) at com.ibm.ws.jmx.connector.client.rest.internal.Connector.connect(Connector.java:103)
This error occurs because the WebSphere Application Server SSL socket factories are not supported by Liberty. You can get past this problem by taking the following steps:- Create a text file, such as my.java.security with the following two empty
entries
ssl.SocketFactory.provider= ssl.ServerSocketFactory.provider=
- Create a jvm.options file for your Liberty server
- Add the following property to your jvm.options file, that includes the full
path to your text file that you just created
-Djava.security.properties=fullPathTo/my.java.security
- If you want to make this more reusable, you can put the my.java.security
file in your server directory, and then you will be able to use a relative path like this:
-Djava.security.properties=./my.java.security
For more information, see Customizing the Liberty environment.
- Create a text file, such as my.java.security with the following two empty
entries
Troubleshooting CORBA/IIOP
This section describes some common CORBA problems and solutions you can choose.
- If you use a JDK from the WebSphere Application Server, you might see the following error if your application uses CORBA/IIOP communications.
15:21:58.096 com.ibm.rmi.pi.InterceptorManager runPreInit:178 Init Process ORBRas [default] java.lang.ClassNotFoundException: com.ibm.ISecurityLocalObjectBaseL13Impl.CSIClientRI at com.ibm.CORBA.iiop.UtilDelegateImpl.loadClass(UtilDelegateImpl.java:685) at javax.rmi.CORBA.Util.loadClass(Util.java:246) at com.ibm.rmi.pi.InterceptorManager.runPreInit(InterceptorManager.java:172) at com.ibm.rmi.corba.ORB.initializeInterceptors(ORB.java:664) at com.ibm.CORBA.iiop.ORB.initializeInterceptors(ORB.java:1084) at com.ibm.rmi.corba.ORB.orbParameters(ORB.java:1359) at com.ibm.rmi.corba.ORB.set_parameters(ORB.java:1271) at com.ibm.CORBA.iiop.ORB.set_parameters(ORB.java:1694) at org.omg.CORBA.ORB.init(ORB.java:371) ...
This error occurs because the WebSphere Application Server Object Request Broker (ORB) interceptors are not supported by Liberty. You can resolve this problem by editing the orb.properties file from the JDK to not use these interceptors. This file is usually found in the WebSphere <JAVA_HOME>/jre/lib directory, though it might have been overridden with a copy in the user's <USER_HOME> directory. The following example shows the lines in the orb.properties file that must be commented out:
# WS Interceptors #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.Transaction.JTS.TxInterceptorInitializer #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ejs.ras.RasContextSupport #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ISecurityLocalObjectBaseL13Impl.ClientRIWrapper #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.activity.remote.cos.ActivityServiceClientInterceptor #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ISecurityLocalObjectBaseL13Impl.CSIClientRI #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.debug.olt.ivbtrjrt.OLT_RI #org.omg.PortableInterceptor.ORBInitializerClass.com.ibm.ws.wlm.client.WLMClientInitializer # WS ORB & Plugins properties #com.ibm.ws.orb.transport.ConnectionInterceptorName=com.ibm.ISecurityLocalObjectBaseL13Impl.SecurityConnectionInterceptor
Troubleshooting logging and tracing
This section describes some common problems with logging and tracing.
- The java.util.logging -- Java logging programming interface.
- Liberty does not support using the logging.properties file to configure java.util.logging. Use java code, for example in a deployed application or user feature, to create and add java.util.logging handlers, filters, or formatters.
- Since the Liberty server manages the java.util.logging logger levels in accordance with the traceSpecification attribute of the logging configuration element, avoid using the Logger.setLevel method.


Applying fix packs and interim fixes to an archive install
If you installed your Liberty runtime environment from an archive file, rather than by using Installation Manager, you must take special measures when you apply service updates. For more information, see Applying a fix pack to a Liberty Java archive installation and Applying an interim fix to a Liberty archive installation.
Troubleshooting performance
This section describes some common performance problems and solutions you can choose.
- High CPU usage by your application monitor.
This error can occur if your application monitor has many files under the apps/ directory and is polling too frequently.
To avoid this problem there are a number of things you can change.
- Increase the value of the pollingRate attribute in the applicationMonitor element.
- Update the server.xml to include an applicationMonitor
element with an updateTrigger that is not polled.
server.xml <applicationMonitor updateTrigger="mbean" />
- Reduce the number of files under the apps/ directory.
For more information about the applicationMonitor element, see Controlling dynamic updates.
Troubleshooting collectives
This section describes a problem with collectives and the solution you must apply.
- java.lang.IllegalArgumentException: CWWKX0219E: The MBean 'WebSphere:feature=collectiveController,type=CollectiveRepository,name=CollectiveRepository' does not have an operation by the name 'retrieveMemberRootCertificate'
- A server with feature collectiveMember-1.0, a member,
cannot register with a server with feature collectiveController-1.0, a controller, that is at a lower level of Liberty. For example,
a member at this beta level of Liberty cannot register itself with a controller at V8.5.5.2
Liberty.
With default logging, this problem is reported as a First Failure Data Capture (FFDC) in the FFDC logs of the member.
To fix the problem, you must move the controller to the same or higher level of Liberty as the member.
Troubleshooting SAML
This section describes a problem with SAML and the solution you must apply.
- java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 0
- This exception can occur when attempting multiple logins through an unsolicited Service Provider (SP) initiated request without removing the Identity Provider token (IdP).
- To avoid this, add <httpSession invalidateOnUnauthorizedSessionRequestException="true" /> in the relevant unsolicited server.xml file.
- java.lang.IllegalStateException: CWWKS0010E: While getting the caller principal, the caller subject was found to have more than one principal of type WSPrincipal. Only one WSPrincipal can exist in the subject. The names of the WSPrincipals are: {0}
- This exception can occur if a SAML user has previously logged directly into an on-premises user registry. To avoid this problem, a SAML user must not directly login to an on-premises user registry.
Troubleshooting REST API Discovery
If the IBM REST API Discovery Explorer "Try it out" selection is invoked remotely and fails with a response code equal to 0, ensure that the fully qualified hostname or the IP address is returned to the IBM REST API Discovery Explorer in the Curl and Request URL associated with the GET, PUT, POST or DELETE operation. If the fully qualified hostname or IP address is not in the Curl and Request URL, complete the following actions:
- Add the variable element in server.xml and specify the fully qualified
hostname. Here is the example of adding the variable element that specifies the fully qualified
hostname in the server.xml
file:
<httpEndpoint host="*" httpPort="9080" httpsPort="9443" id="defaultHttpEndpoint"/> <variable name="defaultHostName" value="developer.rtp.raleigh.ibm.com"/>
Specify the full computer name for your operating system as the fully qualified host name.
- Verify that the fully qualified hostname is returned in the Curl and Request URL associated with the GET, PUT, POST or DELETE operations in the IBM REST API Discovery Explorer. For more information, see Setting the default host name of a Liberty server and read about configuring your network in the IBM InfoSphere Information Server, Version 11.3.1 product documentation.