Fix (APAR): PK90343 Status: Fix Release: 6.1.0.27, 7.0.0.5 Operating System: AIX,HP-UX,i5/OS,IBM i,Linux,Solaris,Windows Supersedes Fixes: CMVC Defect: PK90343 Byte size of APAR: 33521 Date: 2009-11-19 Abstract: The server may hang on server or application startup. Description/symptom of problem: PK90343 resolves the following problem: ERROR DESCRIPTION: The servant hangs while starting up due to a deadlock in the startup threads in the websphere servant JVM. A javacore of the servant shows the following: 1LKDEADLOCK Deadlock detected !!! --------------------- 2LKDEADLOCKTHR Thread "server.startup : 2" 3LKDEADLOCKWTR is waiting for: 4LKDEADLOCKMON sys_mon_t:0x1ACD1DE8 infl_mon_t: com/ibm/ws/bootstrap/WsLogManager@1EC5FC50/1EC5FC5C: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "Thread-18" (0x53989800) 3LKDEADLOCKWTR which is waiting for: 4LKDEADLOCKMON sys_mon_t:0x532C7A70 infl_mon_t: 0x532C7AB0: 4LKDEADLOCKOBJ java/util/HashMap@30668500/3066850C: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "server.startup : 2" (0x53778500) The partial traceback of "server.startup: 2" is: at java/util/logging/LogManager.access$900 at java/util/logging/LogManager$RootLogger.getHandlers at java/util/logging/Logger.log at com/ibm/ws/logging/WsLogger.deliverOrBuffer at com/ibm/ws/logging/WsLogger.log at com/ibm/ws/logging/WsLogger.logp at com/ibm/ws/ffdc/IncidentStreamImpl.initialize at com/ibm/ws/ffdc/IncidentStreamManager.getIncidentStream at com/ibm/ws/ffdc/DiagnosticEngine.processIncident at com/ibm/ws/ffdc/FFDCFilter.filterEngine at com/ibm/ws/ffdc/FFDCFilter.processException atcom/ibm/ws/classloader/SinglePathClassProvider. buildContentsCache . . at com/ibm/ws/classloader/ClassLoaderManager.initialize at com/ibm/ws/classloader/ClassLoaderManager. at com/ibm/ws/runtime/component/DeployedApplicationImpl.start . . com/ibm/wsspi/runtime/component/ WsComponentImpl$_AsynchInitializer.run com/ibm/ws/util/ThreadPool$Worker.run The partial traceback for "Thread-18" is at com/ibm/ws/classloader/SinglePathClassProvider.openZipFile at com/ibm/ws/classloader/SinglePathClassProvider.openZipFile at com/ibm/ws/classloader/SinglePathClassProvider.access$200 at com/ibm/ws/classloader/SinglePathClassProvider$3.run at com/ibm/ws/security/util/AccessController.doPrivileged at com/ibm/ws/classloader/SinglePathClassProvider.getResource at com/ibm/ws/classloader/CompoundClassLoader.findResources at java/lang/ClassLoader.getResources at com/ibm/ws/logging/WsLoggerConfigurator$3.run at com/ibm/ws/security/util/AccessController.doPrivileged at com/ibm/ws/logging/WsLoggerConfigurator.getResources at com/ibm/ws/logging/WsLoggerConfigurator. getCurrentLoggingConfiguration at com/ibm/ws/logging/WsLoggerConfigurator. getLoggerConfiguration at com/ibm/ws/logging/WsLoggerConfigurator.configureLogger . . at com/ibm/ws/wswebcontainer/webapp/WebGroup.addWebApplication at com/ibm/ws/wswebcontainer/VirtualHost.addWebApplication at com/ibm/ws/wswebcontainer/WebContainer.addWebApp at com/ibm/ws/wswebcontainer/WebContainer.run at java/lang/Thread.run PK72252 that shipped in 6.1.0.23, 7.0.0.3, and 6.0.2.33 introduced a zipfile cache in the websphere runtime code to improve performance. The possible deadlock occurs in this new code. This apar will address this problem. LOCAL FIX: None PROBLEM SUMMARY USERS AFFECTED: All users of IBM WebSphere Application Server PROBLEM DESCRIPTION: The server may hang on server or application startup. RECOMMENDATION: None On Application Server startup, application jars are loaded and put in a global cache to reduce the frequency of opening and closing jar files. To maintain the consistency of the global cache, access is restricted to one thread at a time. Attempts to log messages and errors while the global cache has been locked may cause a deadlock. PROBLEM CONCLUSION: The class loading code has been changed to delay the logging of messages and errors until the global cache has been unlocked. APAR PK90343 is currently targeted for inclusion in Service Level (Fix Pack) 6.1.0.29 of WebSphere Application Server V6.1 of the WebSphere Plugin. Please refer to URL: //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970 for Fix Pack availability. Directions to apply fix: Fix applies to Editions: Release 6 X_ Application Server (Express or BASE) X_ Network Deployment (ND) __ WebSphere Business Integration Server Foundation (WBISF) __ Edge Components __ Developer X_ Extended Deployment (XD) Install Fix to: Method: __ Application Server Nodes __ Deployment Manager Nodes X_ Both Directions to apply fix to V7: Fix applies to Editions: Release 7 X_ Application Server (Express or BASE) X_ Network Deployment (ND) __ Edge Components __ Developer NOTE: The user must: * Have Administrative rights in Windows, or be the Actual Root User in a UNIX environments. * Logged in with the same authority level when unpacking a fix, fix pack or refresh pack. * Be at V6.1.0.13 or newer of the Update Installer. Certain iFixes may require a newer version of the Update Installer and the Update Installer will inform you during the installation process if a newer version is required. This can be checked by reviewing the level of the Update Installer in file /updateinstaller/version.txt. The Update Installer can be downloaded from the following link: http://www.ibm.com/support/docview.wss?rs=180&uid=swg21205991 For detailed instructions to Extract the Update Installer see the following Technote: http://www.ibm.com/support/docview.wss?rs=180&uid=swg21205400 Note that there are two different methods for delivering iFixes, depending on the contents. The fix may be delivered either as a single file with a .pak extension (such as 6.1.0.11-WS-WAS-IFPK12345.pak) or a single file with a .zip extension (such as 6.1.0.11-WS-WAS-IFPK12345.zip) which then contains one or more files with a .pak extension. 1) If your iFix is delivered as a single file with a .pak extension, Copy the .pak file directly to the maintenance directory. If your iFix is delivered as a single file with a .zip extension, unzip the file into the maintenance directory. 2) Shutdown WebSphere Manually execute setupCmdLine.bat in Windows or . ./setupCmdLine.sh in Unix from the WebSphere instance that maintenance is being applied to. 3) Launch Update Installer and click the Next button on the Welcome page. 4) Enter the directory path of the installation location of the WebSphere product you want to update, and click the Next button. 5) Select the "Install maintenance package" operation and click the Next button. 6) Enter the directory path of your maintenance directory where you have the maintenance packages (.pak files) and click the Next button. 7) The Available Maintenance Package to Install page should list all maintenance packages (.pak files) that it finds in the directory path provided in the previous step. The Update Installer will select the correct maintenance packages based on your system configuration and will not allow an invalid combination to be installed. Please keep the Update Installer recommendations and click the Next button and continue with the installation of the maintenance package. 8) Please note that in the future, if a Feature Pack is installed or uninstalled, a different set of iFixes will be needed. Use the Update Installer again at that time, with the maintenance directory location where these maintenance packages are stored, to determine the required interim fixes for the new WebSphere and Feature Pack(s) combination. 9) The maintenance packages could have one of a set of names, and these names will help determine which maintenance package you need to install. The APAR name (PKxxxxx) should appear as part of the filename. Between the APAR number and the .pak extension there will be 0 to 2 characters added. The table below indicates the usage of each of the maintenance packages with respect to which Feature Packs, if any, are installed. |.pak File Names |No Feature Packs|EJB3 Only|WebServices Only| Both | |6.1.0.x-WS-WAS-IFPK12345 | X | X | X | X | |6.1.0.x-WS-WAS-IFPK12345C | X | | | | |6.1.0.x-WS-WAS-IFPK12345CE | | X | X | X | |6.1.0.x-WS-WAS-IFPK12345W | X | X | | | |6.1.0.x-WS-WAS-IFPK12345WE | | | X | X | |6.1.0.x-WS-WAS-IFPK12345E | X | | X | | |6.1.0.x-WS-WAS-IFPK12345EE | | X | | X | 10) Restart WebSphere Directions to remove fix: NOTE: * The user must have Administrative rights in Windows, or be the Actual Root User in a UNIX environments. * FIXES MUST BE REMOVED IN THE ORDER THEY WERE APPLIED * DO NOT REMOVE A FIX UNLESS ALL FIXES APPLIED AFTER IT HAVE FIRST BEEN REMOVED * YOU MAY REAPPLY ANY REMOVED FIX Example: If your system has fix1, fix2, and fix3 applied in that order and fix2 is to be removed, fix3 must be removed first, fix2 removed, and fix3 re-applied. 1) Shutdown WebSphere Manually execute setupCmdLine.bat in Windows or . ./setupCmdLine.sh in Unix from the WebSphere instance that uninstall is being run against. 2) Start Update Installer 3) Enter the installation location of the WebSphere product you want to remove the fix. 4) Select "Uninstall maintenance package" operation. 5) Select file name of the maintenance package to uninstall (for example, 6.1.0.11-WS-WAS-IFPK12345.pak). 6) Verify the Uninstallation Summary and continue with the uninstall. 7) Restart WebSphere Directions to re-apply fix: 1) Shutdown WebSphere. 2) Follow the Fix instructions to apply the fix. 3) Restart WebSphere. Additional Information: