Fix (APAR): WAS_Security_10-30-2003_4.0.7-4.0.6-4.0.5_AE_cumulative_Fix Status: Fix Release: 4.0.7,4.0.6,4.0.5 Operating System: All Supersedes Fixes: WAS_Security_01-06-2003_4.0.5-4.0.4_AE_cumulative_eFix, WAS_Security_03-17-2003_4.0.5-4.0.4-4.0.3_AE_cumulative_Fix, WAS_Security_05-16-2003_4.0.5-4.0.4_AE_cumulative_Fix.jar, WAS_Security_06-17-2003_4.0.6-4.0.5-4.0.4_AE_cumulative_Fix and WAS_Security_08-15-2003_4.0.6-4.0.5-4.0.4_AE_cumulative_Fix as well as any fixes for APARs listed below. CMVC Defect: See APAR list below Byte size of APAR: 1321412 Date: 2003-10-30 Abstract: Security cumulative fix 10/30/2003. Description/symptom of problem: See APAR list below. Directions to apply fix: 1) Create temporary "efix" directory to store the jar file: AIX: /tmp/WebSphere/efix Solaris/Linux: /tmp/WebSphere/efix Windows: c:\temp\WebSphere\efix 2) Copy jar file to the directory 3) Shutdown WebSphere 4) Run the jar file with the following command answering questions/prompts as they appear: java -jar 5) Restart WebSphere 6) The temp directory may be removed but the jar file should be saved. Do not remove any files created and stored in the /WebSphere/AppServer/efix/ directories. These files are required if an efix is to be removed. Directions to remove fix: NOTE: EFIXES MUST BE REMOVED IN THE ORDER THEY WERE APPLIED. DO NOT REMOVE AN EFIX UNLESS ALL EFIXES APPLIED AFTER IT HAVE FIRST BEEN REMOVED. YOU MAY REAPPLY ANY REMOVED EFIX. Example: If your system has efix1, efix2, and efix3 applied in that order and efix2 is to be removed, efix3 must be removed first, efix2 removed, and efix3 re-applied. 1) Change directory to the efix location (/WebSphere/AppServer/efix/). 2) Shutdown WebSphere 3) Run the backup jar file with the following command: java -jar 4) Restart WebSphere Directions to re-apply fix: Follow the instructions for applying an efix. If the backup files still exist (from the previous efix application), you will be prompted to overwrite. Answer "yes" at the overwrite prompts. Additional Information: ------------------------------------------------------------------ If the following excption is encountered after applying the fixes, apply the latest JSSE fix. java.net.SocketException: Socket closed at java.net.PlainSocketImpl.socketSetOption(Native Method) at java.net.PlainSocketImpl.setOption(PlainSocketImpl.java:210) at java.net.Socket.setKeepAlive(Socket.java:619) at .... The following properties must be set for PQ56218 to be effective: In the admin.config file: com.ibm.CORBA.ListenerPort=2000 com.ibm.CORBA.SSLPort=2001 com.ibm.CORBA.LSDPort=9000 com.ibm.CORBA.LSDSSLPort=9001 In each of the Application servers system properties: com.ibm.CORBA.ListenerPort=3000 com.ibm.CORBA.SSLPort=3001 Note, each port number must be unique and must not be in use by any other service on the system. To enable PQ63020, set the following property in the admin.config. com.ibm.ejs.security.customregistry.useSecurityName=true Included APARS: APAR: PQ69036 Version: 400 Abstract: APPLYING 11/19 CUMULATIVE SECURITY EFIX ON WAS 4.0.4 CAUSES SECJ0129A AUTHORIZATION FAILURE THAT DIDN'T OCCUR BEFORE EFIX Error Description: Environnment: WebSphere Application Server (WAS) 4.0.4 AE LDAP server . Description: After applying the 11/19/2002 cumulative security eFix, customer starts getting the following exception in the stdout file that didn't occur before the cumulative security efix was applied: . 12/5/02 10:29:02:301 CST 6e93ebd8 WebCollaborat A SECJ0129A: Authorization failed for while invoking POST on default_host:, Authorization failed, Not granted any of the required roles: Local Fix: Remove the cumulative security eFix Logically Dependent Apars: Users Affected: WebSphere Application Server users who have enabled security and use LDAP for the user registry. Problem Description: Authorization failure (403) received after security cache time out is exceeded. Recommendation: Problem Summary: After the cache timeout is exceeded, authorization failures (403) could occur. The reason is while creating new credentials, the group type was not properly appended to the group name which cuased the authorization code to fail in finding the proper group name in security roles. Problem Conclusion: The group type is now properly appended to the group name. A fix for this APAR will be contained in any security cumulative eFix dated after the closure date of this APAR. Test Comments: Circumvention: Temporary Fix: A test fix was provided. Comments: APAR: PQ69643 Version: 400 Abstract: AUTHORIZATION (403) FAILURES FORWARDING FROM AN UNPROTECTED SERVLET OR JSP TO A PROTECTED ONE. Error Description: Authorization (403) failures when forwarding from an unprotected servlet or JSP to a protected one. This issue can also be seen if the contect root is not protected but the default page is protected as a forward is implicit in this scenario. Local Fix: Protect the initially requested page. Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have enabled sec urity and are using RequestDispatcher.forward() to forward from an unprotected servlet or JSP to a prot ected one. Problem Description: Authorization failure (403) is received. Recommendation: Problem Summary: An authorization failure is received when using RequestDispatcher.forward() to forward from an unprotected servlet or JSP to a protected one. Problem Conclusion: The servlet 2.3 specification, section 12.2, specifies that the security model does not apply when using a RequestDispatcher. Therefore, the recommended resolution to this issue is to protect the URI which is invoking RequestDispatcher.forward(). This prepares the application for migration to WebSphere 5.X. If this is not possible then setting the following property on each application server will yield a challenge when forwarding from an unprotected URI to a protected one. com.ibm.ws.security.RequestDispatcherChallenge=true Code implementing this property will be contained in any security cumulative eFix dated after the closure date of this APAR as well as the cumulative eFix dated 01-06-2003. Internal defect number 155475. Test Comments: Circumvention: Protect the URI which is invoking RequestDispatcher.forward(). Temporary Fix: Comments: APAR: PQ72357 Version: 400 Abstract: 403 FORBIDDEN ERRORS 20% OF THE TIME Error Description: . When accessing the /wps/portal page, 403 (forbidden) errors are sometimes returned. In a 62 hour test run, the 403 error occurred aproximately 20% of the time. Local Fix: None Logically Dependent Apars: Users Affected: WebSphere Application Server users who have enabled security and have secured at least one URI. Problem Description: Authorization fails without being challenged. Recommendation: Problem Summary: When multiple concurrent requests are received by the Web container for protected URIs, authorization failures may occur without any challenge. Problem Conclusion: A timing issue existed with the use of an instance variable. The variable was changed to a method variable which insures that two simultaneous threads do not update the variable concurrently. Test Comments: Circumvention: Temporary Fix: send testing fix. Comments: APAR: PQ74897 Version: 400 Abstract: Intermittent authentication & authorization failures concurrent logins on 4.02. CNTR0019E findByPrivilegeAttributeId Error Description: Users encounter intermittent authorization failures under load. Caused by multithreading defect in WebSphere 4.0x using LDAP registry, fixed in cumulative efix for 4.05, but backport to WebSphere 4.02 is required by customer. Local Fix: Use admin.config setting to disable caching Logically Dependent Apars: Users Affected: WebSphere Application Server users who have enabled security and configured an LDAP user registry. Problem Description: Authentication or authorization fails intermittently due to search failure against the LDAP server. Recommendation: Problem Summary: On high volume servers with multiple concurrent logins, authentication or authorization failures may occur occasionally due to concurrent access to the same DirContext used in LDAP search. In an administration server trace, a NullPointException in JNDI code may be seen. Another more common symptom not visible in a trace is corruption of the user's security name or group information. Both symptoms are highly intermittant. Problem Conclusion: LDAP user registry code used to share Directory Contexts between multiple threads. The algorithm has been modified to supply a copy of a directory context to each thread or create a new Directory context if necessary. Test Comments: Circumvention: Temporary Fix: Testing fix provided. Comments: APAR: PQ67679 Version: 400 Abstract: WASSEC - ADDS TAI CAPABILITY TO PQ63020 Error Description: Customer installed Websphere PTF4 to WAS 4.03. Found out that function fixed by previous efix PQ63020 was unavailable. The cause of the failure appears to be TAI. Local Fix: none Logically Dependent Apars: Users Affected: WebSphere Application Server security users who use progra mmatic security with Trust Association and Custom Registry. Problem Description: Wehn using a custom registry with trust association, the AP I getRemoteUser() returns display name rather than security name. Recommendation: Apply this eFix, or use the display name as the same as the s ecurity name. Problem Summary: When using a Custom Registry implementation, security returns the user's display name for getRemoteUser() and getPrincipalName() instead of the user's security name. Problem Conclusion: If getRemoteUser() and getPrincipalName() are required to return the user's security name, a system property has been added to enable this. com.ibm.ejs.security.customregistry.useSecurityName=true Test Comments: Circumvention: make display name the same as security name Temporary Fix: testing eFix provided Comments: APAR: PQ68882 Version: 400 Abstract: CANNOT HAVE , AND () IN ADMIN CONSOLE USER NAME Error Description: The customer cannot use user names that contain the comma character , and parentheses () in the DN as admin console users. The trace file contains the error message 'Invalid LDAP user'. Local Fix: There is no workaround for users whose DN contains a comma and parentheses characters. Logically Dependent Apars: Users Affected: WebSphere Application Server security users who has used a comma (, ASII 44) or open parenthese (ASII 40) or close parenthese (ASII 41) in a user's security name. Problem Description: If a user name or attributes in a DN (if LDAP is the user regi stry) contain comma or open parenthese or close parenthese, authorization for the DN may fail. Recommendation: Problem Summary: If user name or attribute in DN (if using LDAP registry) contain comma or open parenthese or close parenthese, user name was improperly truncated as security use those characters as delimiter, which results authorization error. If a Custom Registry is in use, this also applies to names returned by the following methods: List getUsers() List getUsers(String pattern) String getUserDisplayName(String userName) String getUniqueUserId(String userName) List getUniqueUserIds(String uniqueGroupId) String getUserSecurityName(String uniqueUserId) Problem Conclusion: Comma's and parenthesis were used internally as delimeters. The use of parenthesis as delimiters has been removed. Commas are now treated properly by escaping them. A fix for this APAR will be contained in any security cumulative eFix dated after the closure date of this APAR. Test Comments: Circumvention: Temporary Fix: A test fix was provided. Comments: APAR: PQ68922 Version: 400 Abstract: WAS IS HAVING PROBLEMS AUTHENTICATING WITH IDAR Error Description: Environment: WebSphere Application Server (WAS) 4.0.1 through 4.0.4 (and possibly 4.0.5) iPlanet LDAP server with iDAR . Description: iDAR has a defect which causes WAS to fail authenticating using the JNDI interface. Local Fix: None Logically Dependent Apars: Users Affected: WebSphere Application Server users who have enabled security and are using LDAP as the user registry. Problem Description: Intermittant errors encountered in user authentication. Recommendation: Problem Summary: Intermittant errors encountered in user authentication. The errors were caused by an LDAP search periodically returning the error "could not decode search request". One possible cause for this specific error is the search request returning attributes which do not conform with LDAP standard defined in RFC 2251. Problem Conclusion: The specifc search Wesphre was performaing did not require any attributes to be returned. WebSphere was using an empty string per the JNDI specifications to request no attributes be returned. The LDAP specifications require the string be set to "1.1" if attributes should not be returned. The Sun JNDI LDAP service provider does not properly handle this scenario. WebSphere code was changed to conform with the LDAP specifications instead of the JNDI specifications since LDAP service provider does not handle this scenario properly. A fix for this APAR will be contained in any security cumulative eFix dated after the closure date of this APAR. Test Comments: Circumvention: Temporary Fix: provide testing fix Comments: APAR: PQ71416 Version: 400 Abstract: workaround for search() method Error Description: Fixing the search() method due to what jndi method is returning. Local Fix: search() Logically Dependent Apars: Users Affected: WebSphere Application Server security users who have Distinguished Names (DNs) containing the character '#' on their LDAP server. Problem Description: If a DN contains the special character #, the user may not be authenticated or authorized properly. Recommendation: Problem Summary: If a LDAP entry DN contains the special character '#', a JNDI search does not return relative name, it returns LDAP URL. WebSphere still treats the return value as relative name. Also, if an attribute value in DN starts with '#', the '#' should be escaped. Problem Conclusion: WebSphere will now parse LDAP SearchResult accordingly, either as a relative name or aLDAP URL, and generate a correct DN for the entry. WebSphere also will escape '#' characters if the '#' occurs at the beginging of a String attribute value. Test Comments: Circumvention: Do not start string attribute value in DN with #. Temporary Fix: provide test fix. Comments: APAR: PQ75723 Version: 400 Abstract: Web browser allows multiple LTPAToken cookies to be sent to WSAS causing user not to get past form-based login screen Error Description: Environment: WebSphere Application Server (WSAS) 4.0.x Description: Web browsers are allowing multiple copies of the LTPAToken cookie to be stored in the web browser. When these LTPAToken cookies are sent to WSAS during form-based login, the user is redirected back to the login screen no matter what username and password is used. The fix for this will be included in the WSAS 4.0.7 PTF Local Fix: Logically Dependent Apars: Users Affected: All WebSphere Application Server users who are accessing different WebSphere servers with Single Sign On configured where one server is in a sub-domain of the other server. (e.g. One server is in b.com and the other is in a.b.com) Problem Description: A user is re-prompted to sign on or receives a blank page immediately after signing on. Recommendation: Problem Summary: A user is re-prompted to sign on or receives a blank page immediately after signing on. The root of the problem is that the browser is sending two LtpaToken cookies. WebSphere only attempts to use the first cookie value. If this is not from the same Single Sign On domain and the LTPA keys have not been shared between the servers involved, the token cannot be decrypted and user authentication fails. Problem Conclusion: All LtpaToken cookie values received are now attempted in user authentication. This is done as one cookie is indistinguishable from another cookie if version 0 cookies are used. Version 1 cookies, which contain domain information and can be distinguished, cannot be used for compatibility reasons as older browsers do not recognize them. There are two outstanding issues which will not be addressed in this release of WebSphere. The first is a potential performance degradation to user authentication. This will only occur if two LtpaToken cookies are present in the request and the first cookie is not the proper cookie. The problem will likely only be noticeable on heavily loaded systems. The second will only occur if the WebSphere servers in question have shared LTPA keys. The potential exists for a different user to be logged onto each server. The server in the sub domain may be selected non deterministically as there is no guarantee which cookie will be first in the request. This could cause subsequent requests to this server to be authenticated/authorized by a different user. Test Comments: Circumvention: Do not set up WebSphere servers with SSO configured with one on a sub-domain of another. Temporary Fix: PQ75723_Test.jar Comments: The problem is the browser is not following RFC2109 specification for the handling of cookies. From a WebSphere, perspective the cookies are indistinguishable as the browser does not send any of the cookie attributes that were set by the server on the LtpaToken cookie. The cookie value also does not contain any attributes to distinguish the correct cookie. This value cannot currently be changed due to interoperability issues with other WebSphere servers as well as Domino and WebSeal which may have problems with any new format. Applicable sections of RFC2109: 4.3.2 Rejecting Cookies * The value for the request-host does not domain-match the Domain attribute. * The request-host is a FQDN (not IP address) and has the form HD, where D is the value of the Domain attribute, and H is a string that contains one or more dots. Examples: * A Set-Cookie from request-host y.x.foo.com for Domain=.foo.com would be rejected, because H is y.x and contains a dot. 4.3.4 Sending Cookies to the Origin Server Domain Selection The origin server's fully-qualified host name must domain-match the Domain attribute of the cookie. APAR: PQ76114 Version: 400 Abstract: ADMIN SERVER HANGS WITH SECURITY ON. YOU WILL SEE MANY ORB THREADS IN _REGISTRY_STUB.GETTYPE FROM KILL -3. Error Description: The admin server can sometime hang when security is turned on. When looking at the javacores, you can see many ORB threads are in the following state: at org.omg.CORBA.portable.ObjectImpl._request(ObjectImpl.java: 428) at com.ibm.ejs.security.registry._Registry_Stub.getType( _Registry_Stub.java:605) at com.ibm.ejs.security.ltpa.LTPAServerObject.( LTPAServerObject.java:73) In the middle of the ORB Threads. Now this happens with LDAP, not local OS security. This fix helps improve the performance by making the admin cache this information. Local Fix: Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have security enabled. Problem Description: Threads in the administration server get into a dead locked state. Recommendation: Problem Summary: Threads in the admin server get into a dead locked state. This problem is not seen often and is likely related to a high application server to administration server topology which is not a recommended configuration. The specific system in question had 50+ application servers to one administration server. Problem Conclusion: The dead lock condition was caused by a redundant EJB call. The return value of this call is now cached so that the call only needs to be made once instead of each time a new credential is created. Test Comments: Circumvention: Reduce the application server to administration server ratio. Temporary Fix: PQ76114_Test.jar Comments: APAR: PQ78015 Version: 400 Abstract: WTRN0066W: TRANSACTION COORDINATORIMPL HAS TIMED OUT Error Description: After starting the administration server with security enabled, the customer see the following messages, issued several times in the tracefile: DATE TIME CoordinatorIm I WTRN0066W: Transaction com.ibm.ejs.jts.jts.CoordinatorImpl@1593206813#tid=3335 has timed out after 600 seconds. Local Fix: Logically Dependent Apars: Users Affected: WebSphere Application Server users who have enabled security and configured LTPA. Problem Description: WTRN0066W transaction timeout message if an invalid LTPA Token is used. Recommendation: Problem Summary: WTRN0066W transaction timeout messages occur in tracefile when any invalid Token is received by the security server. A transaction is suspended prior to mapping a Token to a credential and is resumed after credential mapping. However, if an exception is raised during credential mapping, transaction was not resumed which would result in a transaction timeout error. Problem Conclusion: Exception logic was added to resume a transaction after an exception occurs. Test Comments: Circumvention: Temporary Fix: Provided test fix. Comments: APAR: PQ79760 Version: 400 Abstract: INVALID LOGON ATTEMPT USING CUSTOM REGISTRY CAUSES SUBSEQUENT LOGON FAILURES UNTIL SECURITY CACHE TIMEOUT REACHED Error Description: The problem occurs when checking the user id/password combination using the CUR and the CUR throws a CustomRegistryException (e.g. because the backend server does not respond). If this happens, the user id/password combination is cached by the security cache and marked as invalid. This seem to be incorrect because the user id/password combination was not checked at all. This leads to the problem that the user cannot login until the security cache times out even if he uses the correct user id/password combination and the backend is up again. Local Fix: Logically Dependent Apars: Users Affected: All WAS 4.X users. Problem Description: Invalid login attempt using custom registry causes subsequen t logon failures until security cache reached Recommendation: Problem Summary: Invalid login attempt using custom registry causes subsequent logon failures until security cache reached Problem Conclusion: Fixed the code not to cache the failed login. Test Comments: Circumvention: Wait till the cache timeout happens. Temporary Fix: Comments: APAR: PQ65687 Version: 400 Abstract: SECURITY EXCEPTIONS AFTER SERVER REBOOTS Error Description: We are currently having intermittent problems with the IBM WAS Admin service not starting, especially after server reboots. This generates the following entry in sas_server.log: . 1> 2002-07-24 14:23:16.772 , ServerID: 12221999 , SecurityTaggedComponentAssistorImpl.register , Thread Hash: 673378419 Thread P=596272:O=0:CT,5,main : . JSAS0026E: Exception connecting object to the ORB. Check the sas.server.props file to ensure that the SSL keyStore and trustStore properties are set properly. If the problem persists, contact support for assistance. . We did apply the following fix found from the IBM Websphere support site but the problem still recurs. Solution . Put the following parameters in separate lines in the /bin/admin.config and set them to unique unused ports on the machine and restart WAS: . com.ibm.CORBA.SSLPort= com.ibm.CORBA.LSDSSLPort= . Problem occurs on WAS AE 4.03 and 4.04. Local Fix: no workaround available Logically Dependent Apars: Users Affected: All WebSphere Users that have enabled security and restart their adminstration server while their application servers a re up and running. Problem Description: If application server is up and running while the adminst ration server is restarted, then application server continues t o uses a stale IOR without loading the newly created one. Recommendation: Problem Summary: If the WebSphere Application Servers are up and running while the Admin Server is rebooted; then the Application servers continue to use the stale IOR for the Adminstration Server instead of getting the new IOR for the Adminstration Server. With the stale IOR, the Application Servers get OBJECT_NOT_EXIST exceptions. Problem Conclusion: To resolve this problem, the bootstrap repository is read to load the new IORs for the server that was restarted. The logic is to read the bootstrap repository to get the IORs up to three times and ping the server with the new IOR. After three times, if the server does not respond to the new IOR, then OBJECT_NOT_EXIST exception is thrown. Test Comments: Circumvention: Temporary Fix: Comments: APAR: PQ66449 Version: 400 Abstract: AUTH PRINCIPAL NULLS OUT AND CAUSES AUTHORIZATION FAILURE. Error Description: Note from Tom D. at the WCC . Trace and this note on wasdoc0. . - error occurs on thread 5f8245 - prior authentication checks on this thread have succeeded - the failure occurs at 13:06:58 - sequence of events: - EJSSecurityCollaborator.preInvoke - received and invoke creds are both non-null - SetUnauthenticatedCredIfNeeded returns false, meaning that the creds are authorized - SecurityCollaborator.performAuthorization is called - since invokedCred is non-null, curReceived will be set to the value of invokedCred - ejbCheckAuthorization is called with curReceived - WSCheckAccessManager.checkAccess is called. One of the args is a new WSPrincipal, which is constructed with the receivedCred. - in checkAccess, creds is initialized to null. If the principal is non-null, then getCredentials will be called on the principal object. - isGrantedAnyRole is called with the creds object. If the creds object is null, or creds.isUnauthenticated is true, then the error occurs. Since there is one error message for two possible conditions, we do not know which one triggered the error. creds can be null if: 1) the principal passed to checkAccess is null. Then creds will not be set. 2) the call to getCredentials on the principal passed in returns null. - not sure where the code is for isUnauthenticated, so I do not know what conditions would cause this call to return true. * but, upon return to checkAccess from isGrantedAnyRole, a message is printed with the value of principal.toString(), which is UNATHENTICATED. So if the principal is UNAUTHENTICATED, then the call to getCredentials will probably produce null. . QUESTIONS: - why does the same thread seem to be able to create ejbs and have the authentication checks succeed, and then later fail on a subsequent create check? - from looking at two different traces, it does not appear as if the failure only occurs on one ejb. So that could rule out a config/deployment issue. - does an UNAUTHENTICATED invokeCreds object somehow get associated with the call? invokedCred is set by a call to current.get_credentials(). . At this point, I'm thinking that the call to current.get_credentials() in performAuthorization is perhaps not returning the proper credentials, and this leads to the client appearing as unauthenticated. But I still do not have enough evidence to back this up. Local Fix: No workaround available. Logically Dependent Apars: Users Affected: WebSphere Application Server users with security enabled and LocalOS as user registry. Problem Description: The authentication principal is lost which causes authorizatio n failure. Recommendation: Problem Summary: Authentication principal becomes unauthenticated when a valid uid/password is used. Actual Exception: ================= securityName: /UNAUTHENTICATED;accessID: UNAUTHENTICATED is not granted any of the required roles: XXXXX 9/24/02 9:23:37:698 EDT 261218 SecurityColla D Authorization failed accessing EJB com.ibm.ws.security.core.AccessException: securityName: /UNAUTHENTICATED;accessID: UNAUTHENTICATED is not granted any of the required roles: Problem Conclusion: LocalOS credentials were not mapped correctly to the OS user registry when a web request qas received. LocalOS credentials are now correctly mapped. Test Comments: Circumvention: Temporary Fix: available Comments: APAR: PQ67119 Version: 350 Abstract: WEBPSHERE 356 APPSERVER WILL NOT START WHEN SECURITY IS ENABLED ON AN ADMIN AGENT SERVER. Error Description: Upgraded Two Websphere 3.5.4 nodes to Websphere 3.5.6 with security enabled. One node is using a Websphere admin agent thur a firewall to connect to the main Websphere node running the admin server. Various problems either starting an application server on the node running the admin agent and/or connectin the admin server node from the admin agent node. .Running an adminejstrace, it will show that admin server never completes the method: activeObjectInvocation. This method is called from AdminAgentImpl on the admin agent node to update ActiveSecurityConfig on the admin server node.This will cause a deadlock condition within the security code. Local Fix: pmr05680_test.jar Logically Dependent Apars: Users Affected: WebSphere Application Server users of Administration Agent with security enabled. Problem Description: WebSphere AppServer will not start whensecurity is enabled on an adminstrationagent server. Recommendation: Problem Summary: On WebSphere 356, when one node is using administation agent to connect to main WAS node's admin server with both security and auto-start is enabled, the AppServer would failed to start. In the thread dump trace, following thread lock will show. sys_mon_t:0x00239ac0 infl_mon_t: 0x00000000: com.ibm.ISecurityLocalObjectBaseL13Impl.VaultImpl@8E0130/ 8E0138: (Flat locked) owner "Pooled ORB request dispatch WorkerThread" (0xa33b7a0) 1 entry Waiting to be notified: "WebServer-Plugin-Cfg-Thread" (0xac43ca8) sys_mon_t:0x0ab3c658 infl_mon_t: 0x00000000: com.ibm.ISecurityLocalObjectBasicAuthImpl.SecurityContextImpl @9BD940/9BD948: (Flat locked) owner "WebServer-Plugin-Cfg-Thread" (0xac43ca8) 1 entry Waiting to be notified: "Pooled ORB request dispatch WorkerThread" (0xa33b7a0) Problem Conclusion: This is caused by over synchronization in the security component Redundant synchronization code will be removed from security component. Test Comments: Circumvention: Temporary Fix: available Comments: APAR: PQ68078 Version: 400 Abstract: WEBSPHERE SECURITY SAS THROWS NULLPOINTEREXCEPTION WHEN CREDENTIALS EXPIRE Error Description: Customer sees NullPointerException thrown when the credential expires in the sas.server.log file. Error in log shows: 2> 2002-10-08 20:27:25.491 , ServerID: 657623458 , SecureAssociationInterceptorImpl.server_system_exception , Severity code = 0, Error code = 0, Log Exception: java.lang.NullPointerException at com.ibm.ISecurityLocalObjectBaseL13Impl.SecureAssociationInterce ptorImpl.server_system_exception(SecureAssociationInterceptorImp l.java:1824) at com.ibm.CORBA.iiop.RIs.iterateServerException(RIs.java:739) at com.ibm.CORBA.iiop.RIs.iterateServerExceptionYesYesNoNo(RIs.java :758) at com.ibm.CORBA.iiop.ServerResponseImpl.interceptAndWriteHeaderFir st(ServerResponseImpl.java:405) at com.ibm.CORBA.iiop.ServerResponseImpl.write_string(ServerRespons eImpl.java:643) at com.ibm.rmi.util.Utility.writeSystemException(Utility.java:1168) at com.ibm.CORBA.iiop.ServerRequestImpl.createSystemExceptionRespon se(ServerRequestImpl.java:316) at com.ibm.CORBA.iiop.ServerRequestImpl.createUnknownExceptionRespo nse(ServerRequestImpl.java:238) at com.ibm.CORBA.iiop.ExtendedServerDelegate.dispatch(ExtendedServe rDelegate.java:585) at com.ibm.CORBA.iiop.ORB.process(ORB.java:2377) at com.ibm.CORBA.iiop.OrbWorker.run(OrbWorker.java:186) at com.ibm.ejs.oa.pool.ThreadPool$PooledWorker.run(ThreadPool.java: 104) at com.ibm.ws.util.CachedThread.run(ThreadPool.java:137) Local Fix: none. Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have enabled securit y and use LTPA as the authentication mechanism. Problem Description: NullPointerException thrown when on credential expiration. Recommendation: Problem Summary: NullPointerException thrown when the credential expires. Error in log shows: 2> 2002-10-08 20:27:25.491 , ServerID: 657623458 , SecureAssociationInterceptorImpl.server_system_exception , Severity code = 0, Error code = 0, Log Exception: java.lang.NullPointerException at com.ibm.ISecurityLocalObjectBaseL13Impl.SecureAssociationInterce ptorImpl.server_system_exception(SecureAssociationInterceptorImp l.java:1824) at com.ibm.CORBA.iiop.RIs.iterateServerException(RIs.java:739) at com.ibm.CORBA.iiop.RIs.iterateServerExceptionYesYesNoNo(RIs.java :758) at com.ibm.CORBA.iiop.ServerResponseImpl.interceptAndWriteHeaderFir st(ServerResponseImpl.java:405) at com.ibm.CORBA.iiop.ServerResponseImpl.write_string(ServerRespons eImpl.java:643) at com.ibm.rmi.util.Utility.writeSystemException(Utility.java:1168) at com.ibm.CORBA.iiop.ServerRequestImpl.createSystemExceptionRespon se(ServerRequestImpl.java:316) at com.ibm.CORBA.iiop.ServerRequestImpl.createUnknownExceptionRespo nse(ServerRequestImpl.java:238) at com.ibm.CORBA.iiop.ExtendedServerDelegate.dispatch(ExtendedServe rDelegate.java:585) at com.ibm.CORBA.iiop.ORB.process(ORB.java:2377) at com.ibm.CORBA.iiop.OrbWorker.run(OrbWorker.java:186) at com.ibm.ejs.oa.pool.ThreadPool$PooledWorker.run(ThreadPool.java: 104) at com.ibm.ws.util.CachedThread.run(ThreadPool.java:137) Problem Conclusion: Null condition is now checked when processing system_exceptions. A fix for this APAR will be contained in any security cumulative eFix dated after the closure date of this APAR. Test Comments: Circumvention: Temporary Fix: Available Comments: APAR: PQ69078 Version: 400 Abstract: PERFORMANCE DEGRADATION AFTER APPLYING SECURITY CUMULATIVE EFIX FROM 10-07 Error Description: Customer says that they experience degradation in performance after applying WAS Security cumulative efix dated 10-07-02. Local Fix: none Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have enabled securit y and have deployed web applications with protected resource s. Problem Description: Performance degradation in web applications. Recommendation: Problem Summary: Performance degradation in web applications caused by the unnecessary creation of the special credential "UNAUTHENTICATED" with each web request to a protected resource. Problem Conclusion: The UNAUTHENTICATED credential is now cached after it is created the first time. Test Comments: Circumvention: Temporary Fix: Testfix submitted to Portal Server. Comments: APAR: PQ69188 Version: 400 Abstract: UNEXPECTED LOGIN PROMPT WHEN CLICKING ON ADMIN CONSOLE. Error Description: We set the Security timeout to 300 and the LTPA token timeout to 10mn. When the LTPA Token timeout expired, a popup window is displayed asking us to login again and a message is sent in the tracefile in order to warn us that the credential have expired. We expected this behaviour and we were happy. Then we entered the userid/password, we hit "enter". It seems that we are logged in. . Then we click on the console and we saw a new popup window asking us to login again, but this time without any messages in the tracefile. We entered the userid/password and we were able to go through the websphere topology inside the console. We don't think that prompting the second time when clicking on the console is correct. Steps taken: - wait 10mn - popup window - Login/password (msg in tracefile) and hit enter - hit somewhere in the console and you got a second popup window - Login/password (no msg in tracefile) and hit enter - wait 10mn - popup window - Login/password (msg in tracefile) and hit enter - wait 10mn - popup window - Login/password (msg in tracefile) and hit enter . By "message in tracefile", we mean JSAS0435E and CNTR0019E messages are generated. Local Fix: No workaround is known at this time. Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have enabled securit y and use the Administration Console with the Authentication Mechanism set to LTPA. Problem Description: Admin Console displays unexpected login prompt after login. Recommendation: Problem Summary: When the LTPA Token timeout expired, a login window is displayed to login again and a message is sent in the tracefile in order to warn that the credential have expired. After username and password is entered, the console appears to be logged in. However, when the console is clicked again to access resources, another login windows is displayed. Problem Conclusion: This is caused by the expired credential not being cleaned properly. The expired credential is now being removed after expiration. Test Comments: Circumvention: Temporary Fix: avaliable Comments: APAR: PQ70121 Version: 400 Abstract: JAVA.LANG.NULLPOINTEREXCEPTION ERROR ON WAS 4.04 AFTER CUMULATIVE SECURITY EFIX INSTALLED. Error Description: Customer installed WAS 4.04 on AIX 4.3.3. Then installs cumulative security efix dated 01-06-2003. Database is ORacle. After starting admin server, sees in the tracefile: [1/21/03 9:18:20:458 CST] 1c37f64b EJBEngine I WSVR0037I: Starting EJB jar: Tasks java.lang.NullPointerException at com.ibm.ISecurityLocalObjectBaseL13Impl.VaultImpl.get_default_cr edentials(VaultImpl.java(Compiled Code)) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl.get_credenti als(CurrentImpl.java(Compiled Code)) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl.get_credenti als(CurrentImpl.java:313) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl.get_credenti als(CurrentImpl.java:299) at com.ibm.ejs.security.SecurityContext.enable(SecurityContext.java :151) at com.ibm.ws.wlm.admin.config.ServerGroupRefresh$RefreshThread.run (ServerGroupRefresh.java:330) [1/21/03 9:18:26:868 CST] 1c37f64b Server A WSVR0023I: Server __adminServer open for e-business . This error does not appear to affect function of the admin server security. Local Fix: none Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have enabled security. Problem Description: java.lang.NullPointerException in tracefile during server startup after security cumulative fix installed. Recommendation: Problem Summary: java.lang.NullPointerException occured during the server startup. Exception as follows: java.lang.NullPointerException at com.ibm.ISecurityLocalObjectBaseL13Impl.VaultImpl. get_default_credentials(VaultImpl.java(Compiled Code)) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl. get_credentials(CurrentImpl.java(Compiled Code)) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl. get_credentials(CurrentImpl.java:313) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl. get_credentials(CurrentImpl.java:299) at com.ibm.ejs.security.SecurityContext. enable(SecurityContext.java:151) Problem Conclusion: java.lang.NullPointerException occured during the server startup. The null value caused the server credential cache to throw the java.lang.NullPointerException during the startup. However, this exception should not cause any functional loss. A fix for this APAR will be contained in any security cumulative eFix dated after the closure date of this APAR. Test Comments: Circumvention: Temporary Fix: available Comments: APAR: PQ70596 Version: 400 Abstract: ADMIN SERVER USES OLD APP SERVER SSLPORT WHEN APP SERVER IS RESTARTED Error Description: Environment: WebSphere Application Server (WAS) 4.0.4 . Description: With WAS security enabled after an app server is stopped and started, the admin server continues to try to connect to the app server's old SSLPort instead of the new ramdomly created SSLPort resulting in following messages in the admin server's tracefile: . 1/20/03 19:25:59:890 CET 4ed74147 ActiveServerP A ADMS0032I: Started server: SEC_01 (pid 12686) 1/20/03 19:26:31:145 CET 4ed74147 SystemOut U 1> 2003-01-20 19:26:31.134 , ServerID: 1917109904 , VaultImpl.init_security_context : 1/20/03 19:26:31:146 CET 4ed74147 SystemOut U JSAS0070E: Unable to complete secure association at the client. Retry the client program after a few minutes wait. If the problem still persists, contact support for assistance. 1/20/03 19:26:31:564 CET 4ed74147 SystemOut U 3> 2003-01-20 19:26:31.563 , ServerID: 1917109904 , VaultImpl.init_security_context : 1/20/03 19:26:31:565 CET 4ed74147 SystemOut U JSAS0070E: Unable to complete secure association at the client. Retry the client program after a few minutes wait. If the problem still persists, contact support for assistance. 1/20/03 19:26:31:570 CET 4ed74147 ActiveEJBServ W ADSM0104W: Failed to initialize a server: SEC_01 CORBA COMM_FAILURE 1 No; nested exception is: org.omg.CORBA.COMM_FAILURE: minor code: 1 completed: No Local Fix: Set the app server's SSL port to a static value Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have enabled securit y. Problem Description: WebSphere Application Server is unable to reconnect after applic ation server is stopped and restarted. Recommendation: Problem Summary: After an application server is stopped and started, the adminstration server continues to try to connect to the application server's old SSLPort instead of the new ramdomly created SSLPort resulting in following messages in the admin server's tracefile: . 1/20/03 19:25:59:890 CET 4ed74147 ActiveServerP A ADMS0032I: Started server: SEC_01 (pid 12686) 1/20/03 19:26:31:145 CET 4ed74147 SystemOut U 1> 2003-01-20 19:26:31.134 , ServerID: 1917109904 , VaultImpl.init_security_context : 1/20/03 19:26:31:146 CET 4ed74147 SystemOut U JSAS0070E: Unable to complete secure association at the client. Retry the client program after a few minutes wait. If the problem still persists, contact support for assistance. Problem Conclusion: The adminserver now reregisters application server's new ssl port and then makes new connection to the new application server's new port. Test Comments: Circumvention: Temporary Fix: available Comments: APAR: PQ71310 Version: 400 Abstract: getUserPrincipal() getname() returns wrong name rather than logg ed in username accessing from another domain Error Description: Customer logs in to the administration console using the administration username and password The Custom realm is called and accepts the administration username and password. Then logs to the Web Application using a web username and password (username and password are different from administration username and password): The Custom realm is called and accepts the authentication. The Web Application (a servlet) calls getUserPrincipal().getName() from the HttpServletRequest. The Web application receives the administration user identity. Later calls to getUserPrincipal().getName() from the HttpServletRequest return the correct logged user. Local Fix: none Logically Dependent Apars: Users Affected: WebSphere Application Server security users who have deployed servlets and EJBs in different realms. Problem Description: getName() in getUserPrincipal() may not return the right security name during an EJB call. Recommendation: Problem Summary: getName() in getUserPrincipal() may return wrong security name after servlet accesses an EJB in a different security domain. Problem Conclusion: When a servlet accesses EJB in a different realm, security will try to map the invocation credential to the target realm. If the credential mapping fails, the original credential is now returned rather than returning the default credential. Test Comments: Circumvention: Temporary Fix: provided test fix Comments: APAR: PQ72041 Version: 350 APAR: PQ74826 Version: 350 APAR: PQ71397 Version: 400 Abstract: Admin server is trying to use expired tokens from cache - invali d tokens. Error Description: Local Fix: Logically Dependent Apars: Users Affected: Problem Description: Recommendation: Problem Summary: Problem Conclusion: Test Comments: Circumvention: Temporary Fix: Comments: APAR: PQ71308 Version: 400 Abstract: WAS is using invalid tokens from server cache instead of new one Error Description: After an ltpa token expires and the user tries to access a secured resource they get re-directed to the login page(as expected). After entering their user name and password again they get an "Invalid Credential" exception and cannot get access. If they were to wait a period of time, say 5-10 mins, they have no trouble getting back in. Local Fix: Increase LTPA timeout Logically Dependent Apars: Users Affected: Problem Description: Recommendation: Problem Summary: Problem Conclusion: Test Comments: Circumvention: Temporary Fix: Comments: duplicated APAR . APAR: PQ71832 Version: 400 Abstract: NULL SESSION HANDLE ERRORS IN APPLICATION SERVER STDOUT FILE. Error Description: (PQ71832 is included in 5-16-2003 security cumulative fix.) 1.These errors occur repeatedly in the app servers stdout file. . [VaultImpl.get_security_context]: [12/13/02 9:11:12:905 EST] 1214b1 SystemOut U JSAS0170E: Null session handle in session table. Check to see if a server process has terminated just prior to receiving these errors. If a process has terminated, restart the process and retry the operation. Verify that the client userid/password is valid. If the login fails, the session will be deleted on the client side and the credentials will be marked invalid. If a retry occurs, you will likely see this error. Restart the client program after verifying the login info. If the errors persist, contact support for assistance. . 2.The errors happen on different application servers at different times. The customer has 52 application servers running on 2 nodes (36:1 ratio). The error also occurs in his production system which has 8 application servers to 1 node. . 3. When customer enables tracing, the errors do not occur. He is using seclogger40.jar for tracing. He began by filtering for these three classes: . com.ibm.CORBA.securityTraceFilter=SecureAssociationInterceptorI pl,VaultImpl,SecurityConnectionInterceptor . The problem did not occur during tracing. Once tracing was disabled, the problem came back. . He has also tried tracing while filtering for just one class at a time. The problem still does not occur even with tracing only one class. . 4. The customer has increased the value of maxopenconnections to 1000, and the errors still persists. The Sun environment has a limit of 1024 threads per process, so the most this value can be set to is 1024. . 5. The customer has tuned their system according to suggestions from the ORB team as follows. While this has improved other problems, it has not changed this problem. . 6. Two previous suggestions have NOT been tried: John suggested setting in sas.server.props file, the following property: com.ibm.CORBA.NotifyBrokenConnectionEnabled=false. This has not been tried due to concern over side effects (memory leaks). . The other suggestion was to just enable ORB tracing. The customer has concerns about the performance impact of doing this, and he is concerned that the ORB tracing will also mask the problem. . If level 3 thinks that it is essential to attempt these last two items, the customer is willing to attempt them - if their concerns are addressed. . This APAR is being created at the request of L3 security team: Because the null session handle error is a fully recoverable condition, the WebSphere administrator should not be informed of this condition. This APAR will address removing the message and making it a debug only message. Local Fix: Enabling security tracing on just one class causes the errors to disappear. Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have enabled security. Problem Description: JSAS0170E: Null session handle in session table reported in the tracefile and Administration console. Recommendation: Problem Summary: These errors occur repeatedly in the application servers stdout file. [VaultImpl.get_security_context]: [12/13/02 9:11:12:905 EST] 1214b1 SystemOut U JSAS0170E: Null session handle in session table. Check to see if a server process has terminated just prior to receiving these errors. If a process has terminated, restart the process and retry the operation. Verify that the client userid/password is valid. If the login fails, the session will be deleted on the client side and the credentials will be marked invalid. If a retry occurs, you will likely see this error. Restart the client program after verifying the login info. If the errors persist, contact support for assistance. Problem Conclusion: The message currently represents a fully recoverable condition from respect of the code reporting it. The message has been converted into a message for debug tracing only. Test Comments: Circumvention: Temporary Fix: Comments: APAR: PQ72942 Version: 400 Abstract: JSAS0070E AND JSAS0200E ERRORS AFTER ADMIN SERVER RUNS FOR AN HOUR. Error Description: After the admin server has run for approximately an hour, the logs record a huge number of JSAS0070E and JSAS0200E errors. These do not seem to harm the server or any applications; there seems to be no adverse effect at all. Secondary Symptoms: CredentialsImpl.refreshServerCred: Unable to refresh the server's credentials, reset to minimum expiration time. Local Fix: There is no workaround available for this problem. Logically Dependent Apars: Users Affected: WebSphere Application Server security users using the LTPA authentication mechanism. Problem Description: The server credential cannot be refreshed. Recommendation: Problem Summary: The server credential is refreshed at the security cache timeout interval. The LTPA token within the credential is used to perform the operation. If the LTPA token in the credential is expired at the time the credential refresh is attempted, the refresh operation will fail. the following error will be logged multiple times; JSAS0070E. Problem Conclusion: An LTPA credential is no longer used to refresh the server credential. Instead, a BasicAuth credential (containing a user ID and password) is used. Test Comments: Circumvention: Temporary Fix: provided test fix Comments: APAR: PQ75785 Version: 400 Abstract: 4.0.6 Admin Console re-prompting Customer for log-in Error Description: Problem Description: Customer is at 4.0.6 on Solaris. When they start the Admin Console, it prompts them for log-in (as expected), and the console comes up fine. After 10 minutes (LTPA Token expiration time), customer is re-prompted for log-in. The prompt stays there for 5 minutes (com.ibm.CORBA.loginTim eout) and then disappears. After 10 minutes, another pop-up appears and dis-aapears in 5 minutes, and so on. During the 10 minutes after the pop-up disappears, anyone can walk up to the Console and use it. Customer is concerned about this apparent lack of security. Expected Behavior: After the first log-in prompt and successful log-in, there should not have been any re-prompting. Explanation: When the Client (admin console) successfully logs-in, the Client-side WebSphere runtime creates BasicAuth Credentials for the Client (these never expire) and sends the request to the Server. The Server-side creates LTPA credentials for this client and uses those to authorize the Client. After 10 minutes, these credentials expire and the Server goes back to the Client to ask for valid credentials. At this point, the Client-side is supposed to simply re-use the BasicAuth credentials and send the request back to the client. As long as the userid/password, supplied initially, are still valid, everything should just work fine. However, instead of first checking the BasicAuth information the Client-side has, WebSphere instead re-prompts the Client for a log-in. This is unexpected and a defect. When the login times out and the pop-up disappears, the Client-side then realizes that it already has valid BasicAuth Credentials and uses them. In the end, the result is as expected and as designed. But, the behavior of the pop-up needs to be fixed. Local Fix: none. No functional loss. Logically Dependent Apars: Users Affected: WebSphere Application Server users who have enabled security and are using the LTPA authentication mechanism. Problem Description: Java clients re-prompt for user ID and password after LTPAToken expired. Recommendation: Problem Summary: Java clients, including the Administration Console, may prompt for user ID and password information again after LTPA Token expired. This should not occur since Java clients use a basic authentication credential which does not expire. Problem Conclusion: The server now deletes the session on credential expiration. The client retries once using the existing basic authentication credential if the session was deleted for an expired credential. This prevents the client from re-prompting unless the user or password is no longer valid in the user registry. Test Comments: Circumvention: Temporary Fix: test fix provided. Comments: APAR: PQ79742 Version: 400 Abstract: I/O Error while processing the security bootstrap repository. Error Description: I/O Error while processing the security bootstrap repository. Local Fix: Local Fix Logically Dependent Apars: Users Affected: Problem Description: Recommendation: Problem Summary: Problem Conclusion: Test Comments: Circumvention: Temporary Fix: Comments: