Fix (APAR): WAS_Security_10-30-2003_4.0.7-4.0.6-4.0.5_AEs_cumulative_Fix Status: Fix Release: 4.0.7,4.0.6,4.0.5 Operating System: All Supersedes Fixes: WAS_Security_01-06-2003_4.0.5-4.0.4_AEs_cumulative_eFix, WAS_Security_03-17-2003_4.0.5-4.0.4-4.0.3_AEs_cumulative_Fix, WAS_Security_05-16-2003_4.0.5-4.0.4_AEs_cumulative_Fix.jar, WAS_Security_06-17-2003_4.0.6-4.0.5-4.0.4_AEs_cumulative_Fix and WAS_Security_08-15-2003_4.0.6-4.0.5-4.0.4_AEs_cumulative_Fix as well as any fixes for APARs listed below. CMVC Defect: See APAR list below Byte size of APAR: 1277078 Date: 2003-10-30 Abstract: Security cumulative fix 10/30/2003. Description/symptom of problem: See APAR list below. Directions to apply fix: 1) Create temporary "efix" directory to store the jar file: AIX: /tmp/WebSphere/efix Solaris/Linux: /tmp/WebSphere/efix Windows: c:\temp\WebSphere\efix 2) Copy jar file to the directory 3) Shutdown WebSphere 4) Run the jar file with the following command answering questions/prompts as they appear: java -jar 5) Restart WebSphere 6) The temp directory may be removed but the jar file should be saved. Do not remove any files created and stored in the /WebSphere/AppServer/efix/ directories. These files are required if an efix is to be removed. Directions to remove fix: NOTE: EFIXES MUST BE REMOVED IN THE ORDER THEY WERE APPLIED. DO NOT REMOVE AN EFIX UNLESS ALL EFIXES APPLIED AFTER IT HAVE FIRST BEEN REMOVED. YOU MAY REAPPLY ANY REMOVED EFIX. Example: If your system has efix1, efix2, and efix3 applied in that order and efix2 is to be removed, efix3 must be removed first, efix2 removed, and efix3 re-applied. 1) Change directory to the efix location (/WebSphere/AppServer/efix/). 2) Shutdown WebSphere 3) Run the backup jar file with the following command: java -jar 4) Restart WebSphere Directions to re-apply fix: Follow the instructions for applying an efix. If the backup files still exist (from the previous efix application), you will be prompted to overwrite. Answer "yes" at the overwrite prompts. Additional Information: ------------------------------------------------------------------ If the following excption is encountered after applying the fixes, apply the latest JSSE fix. java.net.SocketException: Socket closed at java.net.PlainSocketImpl.socketSetOption(Native Method) at java.net.PlainSocketImpl.setOption(PlainSocketImpl.java:210) at java.net.Socket.setKeepAlive(Socket.java:619) at .... The following properties must be set for PQ56218 to be effective: In the admin.config file: com.ibm.CORBA.ListenerPort=2000 com.ibm.CORBA.SSLPort=2001 com.ibm.CORBA.LSDPort=9000 com.ibm.CORBA.LSDSSLPort=9001 In each of the Application servers system properties: com.ibm.CORBA.ListenerPort=3000 com.ibm.CORBA.SSLPort=3001 Note, each port number must be unique and must not be in use by any other service on the system. To enable PQ63020, set the following property in the admin.config. com.ibm.ejs.security.customregistry.useSecurityName=true Included APARS: APAR: PQ69036 Version: 400 Abstract: APPLYING 11/19 CUMULATIVE SECURITY EFIX ON WAS 4.0.4 CAUSES SECJ0129A AUTHORIZATION FAILURE THAT DIDN'T OCCUR BEFORE EFIX Error Description: Environnment: WebSphere Application Server (WAS) 4.0.4 AE LDAP server . Description: After applying the 11/19/2002 cumulative security eFix, customer starts getting the following exception in the stdout file that didn't occur before the cumulative security efix was applied: . 12/5/02 10:29:02:301 CST 6e93ebd8 WebCollaborat A SECJ0129A: Authorization failed for while invoking POST on default_host:, Authorization failed, Not granted any of the required roles: Local Fix: Remove the cumulative security eFix Logically Dependent Apars: Users Affected: WebSphere Application Server users who have enabled security and use LDAP for the user registry. Problem Description: Authorization failure (403) received after security cache time out is exceeded. Recommendation: Problem Summary: After the cache timeout is exceeded, authorization failures (403) could occur. The reason is while creating new credentials, the group type was not properly appended to the group name which cuased the authorization code to fail in finding the proper group name in security roles. Problem Conclusion: The group type is now properly appended to the group name. A fix for this APAR will be contained in any security cumulative eFix dated after the closure date of this APAR. Test Comments: Circumvention: Temporary Fix: A test fix was provided. Comments: APAR: PQ69643 Version: 400 Abstract: AUTHORIZATION (403) FAILURES FORWARDING FROM AN UNPROTECTED SERVLET OR JSP TO A PROTECTED ONE. Error Description: Authorization (403) failures when forwarding from an unprotected servlet or JSP to a protected one. This issue can also be seen if the contect root is not protected but the default page is protected as a forward is implicit in this scenario. Local Fix: Protect the initially requested page. Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have enabled sec urity and are using RequestDispatcher.forward() to forward from an unprotected servlet or JSP to a prot ected one. Problem Description: Authorization failure (403) is received. Recommendation: Problem Summary: An authorization failure is received when using RequestDispatcher.forward() to forward from an unprotected servlet or JSP to a protected one. Problem Conclusion: The servlet 2.3 specification, section 12.2, specifies that the security model does not apply when using a RequestDispatcher. Therefore, the recommended resolution to this issue is to protect the URI which is invoking RequestDispatcher.forward(). This prepares the application for migration to WebSphere 5.X. If this is not possible then setting the following property on each application server will yield a challenge when forwarding from an unprotected URI to a protected one. com.ibm.ws.security.RequestDispatcherChallenge=true Code implementing this property will be contained in any security cumulative eFix dated after the closure date of this APAR as well as the cumulative eFix dated 01-06-2003. Internal defect number 155475. Test Comments: Circumvention: Protect the URI which is invoking RequestDispatcher.forward(). Temporary Fix: Comments: APAR: PQ72357 Version: 400 Abstract: 403 FORBIDDEN ERRORS 20% OF THE TIME Error Description: . When accessing the /wps/portal page, 403 (forbidden) errors are sometimes returned. In a 62 hour test run, the 403 error occurred aproximately 20% of the time. Local Fix: None Logically Dependent Apars: Users Affected: WebSphere Application Server users who have enabled security and have secured at least one URI. Problem Description: Authorization fails without being challenged. Recommendation: Problem Summary: When multiple concurrent requests are received by the Web container for protected URIs, authorization failures may occur without any challenge. Problem Conclusion: A timing issue existed with the use of an instance variable. The variable was changed to a method variable which insures that two simultaneous threads do not update the variable concurrently. Test Comments: Circumvention: Temporary Fix: send testing fix. Comments: APAR: PQ74897 Version: 400 Abstract: Intermittent authentication & authorization failures concurrent logins on 4.02. CNTR0019E findByPrivilegeAttributeId Error Description: Users encounter intermittent authorization failures under load. Caused by multithreading defect in WebSphere 4.0x using LDAP registry, fixed in cumulative efix for 4.05, but backport to WebSphere 4.02 is required by customer. Local Fix: Use admin.config setting to disable caching Logically Dependent Apars: Users Affected: WebSphere Application Server users who have enabled security and configured an LDAP user registry. Problem Description: Authentication or authorization fails intermittently due to search failure against the LDAP server. Recommendation: Problem Summary: On high volume servers with multiple concurrent logins, authentication or authorization failures may occur occasionally due to concurrent access to the same DirContext used in LDAP search. In an administration server trace, a NullPointException in JNDI code may be seen. Another more common symptom not visible in a trace is corruption of the user's security name or group information. Both symptoms are highly intermittant. Problem Conclusion: LDAP user registry code used to share Directory Contexts between multiple threads. The algorithm has been modified to supply a copy of a directory context to each thread or create a new Directory context if necessary. Test Comments: Circumvention: Temporary Fix: Testing fix provided. Comments: APAR: PQ65687 Version: 400 Abstract: SECURITY EXCEPTIONS AFTER SERVER REBOOTS Error Description: We are currently having intermittent problems with the IBM WAS Admin service not starting, especially after server reboots. This generates the following entry in sas_server.log: . 1> 2002-07-24 14:23:16.772 , ServerID: 12221999 , SecurityTaggedComponentAssistorImpl.register , Thread Hash: 673378419 Thread P=596272:O=0:CT,5,main : . JSAS0026E: Exception connecting object to the ORB. Check the sas.server.props file to ensure that the SSL keyStore and trustStore properties are set properly. If the problem persists, contact support for assistance. . We did apply the following fix found from the IBM Websphere support site but the problem still recurs. Solution . Put the following parameters in separate lines in the /bin/admin.config and set them to unique unused ports on the machine and restart WAS: . com.ibm.CORBA.SSLPort= com.ibm.CORBA.LSDSSLPort= . Problem occurs on WAS AE 4.03 and 4.04. Local Fix: no workaround available Logically Dependent Apars: Users Affected: All WebSphere Users that have enabled security and restart their adminstration server while their application servers a re up and running. Problem Description: If application server is up and running while the adminst ration server is restarted, then application server continues t o uses a stale IOR without loading the newly created one. Recommendation: Problem Summary: If the WebSphere Application Servers are up and running while the Admin Server is rebooted; then the Application servers continue to use the stale IOR for the Adminstration Server instead of getting the new IOR for the Adminstration Server. With the stale IOR, the Application Servers get OBJECT_NOT_EXIST exceptions. Problem Conclusion: To resolve this problem, the bootstrap repository is read to load the new IORs for the server that was restarted. The logic is to read the bootstrap repository to get the IORs up to three times and ping the server with the new IOR. After three times, if the server does not respond to the new IOR, then OBJECT_NOT_EXIST exception is thrown. Test Comments: Circumvention: Temporary Fix: Comments: APAR: PQ66449 Version: 400 Abstract: AUTH PRINCIPAL NULLS OUT AND CAUSES AUTHORIZATION FAILURE. Error Description: Note from Tom D. at the WCC . Trace and this note on wasdoc0. . - error occurs on thread 5f8245 - prior authentication checks on this thread have succeeded - the failure occurs at 13:06:58 - sequence of events: - EJSSecurityCollaborator.preInvoke - received and invoke creds are both non-null - SetUnauthenticatedCredIfNeeded returns false, meaning that the creds are authorized - SecurityCollaborator.performAuthorization is called - since invokedCred is non-null, curReceived will be set to the value of invokedCred - ejbCheckAuthorization is called with curReceived - WSCheckAccessManager.checkAccess is called. One of the args is a new WSPrincipal, which is constructed with the receivedCred. - in checkAccess, creds is initialized to null. If the principal is non-null, then getCredentials will be called on the principal object. - isGrantedAnyRole is called with the creds object. If the creds object is null, or creds.isUnauthenticated is true, then the error occurs. Since there is one error message for two possible conditions, we do not know which one triggered the error. creds can be null if: 1) the principal passed to checkAccess is null. Then creds will not be set. 2) the call to getCredentials on the principal passed in returns null. - not sure where the code is for isUnauthenticated, so I do not know what conditions would cause this call to return true. * but, upon return to checkAccess from isGrantedAnyRole, a message is printed with the value of principal.toString(), which is UNATHENTICATED. So if the principal is UNAUTHENTICATED, then the call to getCredentials will probably produce null. . QUESTIONS: - why does the same thread seem to be able to create ejbs and have the authentication checks succeed, and then later fail on a subsequent create check? - from looking at two different traces, it does not appear as if the failure only occurs on one ejb. So that could rule out a config/deployment issue. - does an UNAUTHENTICATED invokeCreds object somehow get associated with the call? invokedCred is set by a call to current.get_credentials(). . At this point, I'm thinking that the call to current.get_credentials() in performAuthorization is perhaps not returning the proper credentials, and this leads to the client appearing as unauthenticated. But I still do not have enough evidence to back this up. Local Fix: No workaround available. Logically Dependent Apars: Users Affected: WebSphere Application Server users with security enabled and LocalOS as user registry. Problem Description: The authentication principal is lost which causes authorizatio n failure. Recommendation: Problem Summary: Authentication principal becomes unauthenticated when a valid uid/password is used. Actual Exception: ================= securityName: /UNAUTHENTICATED;accessID: UNAUTHENTICATED is not granted any of the required roles: XXXXX 9/24/02 9:23:37:698 EDT 261218 SecurityColla D Authorization failed accessing EJB com.ibm.ws.security.core.AccessException: securityName: /UNAUTHENTICATED;accessID: UNAUTHENTICATED is not granted any of the required roles: Problem Conclusion: LocalOS credentials were not mapped correctly to the OS user registry when a web request qas received. LocalOS credentials are now correctly mapped. Test Comments: Circumvention: Temporary Fix: available Comments: APAR: PQ69078 Version: 400 Abstract: PERFORMANCE DEGRADATION AFTER APPLYING SECURITY CUMULATIVE EFIX FROM 10-07 Error Description: Customer says that they experience degradation in performance after applying WAS Security cumulative efix dated 10-07-02. Local Fix: none Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have enabled securit y and have deployed web applications with protected resource s. Problem Description: Performance degradation in web applications. Recommendation: Problem Summary: Performance degradation in web applications caused by the unnecessary creation of the special credential "UNAUTHENTICATED" with each web request to a protected resource. Problem Conclusion: The UNAUTHENTICATED credential is now cached after it is created the first time. Test Comments: Circumvention: Temporary Fix: Testfix submitted to Portal Server. Comments: APAR: PQ70121 Version: 400 Abstract: JAVA.LANG.NULLPOINTEREXCEPTION ERROR ON WAS 4.04 AFTER CUMULATIVE SECURITY EFIX INSTALLED. Error Description: Customer installed WAS 4.04 on AIX 4.3.3. Then installs cumulative security efix dated 01-06-2003. Database is ORacle. After starting admin server, sees in the tracefile: [1/21/03 9:18:20:458 CST] 1c37f64b EJBEngine I WSVR0037I: Starting EJB jar: Tasks java.lang.NullPointerException at com.ibm.ISecurityLocalObjectBaseL13Impl.VaultImpl.get_default_cr edentials(VaultImpl.java(Compiled Code)) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl.get_credenti als(CurrentImpl.java(Compiled Code)) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl.get_credenti als(CurrentImpl.java:313) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl.get_credenti als(CurrentImpl.java:299) at com.ibm.ejs.security.SecurityContext.enable(SecurityContext.java :151) at com.ibm.ws.wlm.admin.config.ServerGroupRefresh$RefreshThread.run (ServerGroupRefresh.java:330) [1/21/03 9:18:26:868 CST] 1c37f64b Server A WSVR0023I: Server __adminServer open for e-business . This error does not appear to affect function of the admin server security. Local Fix: none Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have enabled security. Problem Description: java.lang.NullPointerException in tracefile during server startup after security cumulative fix installed. Recommendation: Problem Summary: java.lang.NullPointerException occured during the server startup. Exception as follows: java.lang.NullPointerException at com.ibm.ISecurityLocalObjectBaseL13Impl.VaultImpl. get_default_credentials(VaultImpl.java(Compiled Code)) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl. get_credentials(CurrentImpl.java(Compiled Code)) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl. get_credentials(CurrentImpl.java:313) at com.ibm.ISecurityLocalObjectBaseL13Impl.CurrentImpl. get_credentials(CurrentImpl.java:299) at com.ibm.ejs.security.SecurityContext. enable(SecurityContext.java:151) Problem Conclusion: java.lang.NullPointerException occured during the server startup. The null value caused the server credential cache to throw the java.lang.NullPointerException during the startup. However, this exception should not cause any functional loss. A fix for this APAR will be contained in any security cumulative eFix dated after the closure date of this APAR. Test Comments: Circumvention: Temporary Fix: available Comments: APAR: PQ71310 Version: 400 Abstract: getUserPrincipal() getname() returns wrong name rather than logg ed in username accessing from another domain Error Description: Customer logs in to the administration console using the administration username and password The Custom realm is called and accepts the administration username and password. Then logs to the Web Application using a web username and password (username and password are different from administration username and password): The Custom realm is called and accepts the authentication. The Web Application (a servlet) calls getUserPrincipal().getName() from the HttpServletRequest. The Web application receives the administration user identity. Later calls to getUserPrincipal().getName() from the HttpServletRequest return the correct logged user. Local Fix: none Logically Dependent Apars: Users Affected: WebSphere Application Server security users who have deployed servlets and EJBs in different realms. Problem Description: getName() in getUserPrincipal() may not return the right security name during an EJB call. Recommendation: Problem Summary: getName() in getUserPrincipal() may return wrong security name after servlet accesses an EJB in a different security domain. Problem Conclusion: When a servlet accesses EJB in a different realm, security will try to map the invocation credential to the target realm. If the credential mapping fails, the original credential is now returned rather than returning the default credential. Test Comments: Circumvention: Temporary Fix: provided test fix Comments: APAR: PQ71832 Version: 400 Abstract: NULL SESSION HANDLE ERRORS IN APPLICATION SERVER STDOUT FILE. Error Description: (PQ71832 is included in 5-16-2003 security cumulative fix.) 1.These errors occur repeatedly in the app servers stdout file. . [VaultImpl.get_security_context]: [12/13/02 9:11:12:905 EST] 1214b1 SystemOut U JSAS0170E: Null session handle in session table. Check to see if a server process has terminated just prior to receiving these errors. If a process has terminated, restart the process and retry the operation. Verify that the client userid/password is valid. If the login fails, the session will be deleted on the client side and the credentials will be marked invalid. If a retry occurs, you will likely see this error. Restart the client program after verifying the login info. If the errors persist, contact support for assistance. . 2.The errors happen on different application servers at different times. The customer has 52 application servers running on 2 nodes (36:1 ratio). The error also occurs in his production system which has 8 application servers to 1 node. . 3. When customer enables tracing, the errors do not occur. He is using seclogger40.jar for tracing. He began by filtering for these three classes: . com.ibm.CORBA.securityTraceFilter=SecureAssociationInterceptorI pl,VaultImpl,SecurityConnectionInterceptor . The problem did not occur during tracing. Once tracing was disabled, the problem came back. . He has also tried tracing while filtering for just one class at a time. The problem still does not occur even with tracing only one class. . 4. The customer has increased the value of maxopenconnections to 1000, and the errors still persists. The Sun environment has a limit of 1024 threads per process, so the most this value can be set to is 1024. . 5. The customer has tuned their system according to suggestions from the ORB team as follows. While this has improved other problems, it has not changed this problem. . 6. Two previous suggestions have NOT been tried: John suggested setting in sas.server.props file, the following property: com.ibm.CORBA.NotifyBrokenConnectionEnabled=false. This has not been tried due to concern over side effects (memory leaks). . The other suggestion was to just enable ORB tracing. The customer has concerns about the performance impact of doing this, and he is concerned that the ORB tracing will also mask the problem. . If level 3 thinks that it is essential to attempt these last two items, the customer is willing to attempt them - if their concerns are addressed. . This APAR is being created at the request of L3 security team: Because the null session handle error is a fully recoverable condition, the WebSphere administrator should not be informed of this condition. This APAR will address removing the message and making it a debug only message. Local Fix: Enabling security tracing on just one class causes the errors to disappear. Logically Dependent Apars: Users Affected: All WebSphere Application Server users who have enabled security. Problem Description: JSAS0170E: Null session handle in session table reported in the tracefile and Administration console. Recommendation: Problem Summary: These errors occur repeatedly in the application servers stdout file. [VaultImpl.get_security_context]: [12/13/02 9:11:12:905 EST] 1214b1 SystemOut U JSAS0170E: Null session handle in session table. Check to see if a server process has terminated just prior to receiving these errors. If a process has terminated, restart the process and retry the operation. Verify that the client userid/password is valid. If the login fails, the session will be deleted on the client side and the credentials will be marked invalid. If a retry occurs, you will likely see this error. Restart the client program after verifying the login info. If the errors persist, contact support for assistance. Problem Conclusion: The message currently represents a fully recoverable condition from respect of the code reporting it. The message has been converted into a message for debug tracing only. Test Comments: Circumvention: Temporary Fix: Comments: