Fix (APAR): PM11713 Status: Fix Release: 6.0.2.37,6.0.2.39,6.0.2.41,6.1.0.31,6.1.0.29,6.1.0.27,7.0.0.3,7.0.0.5,7.0.0.7,7.0.0.9 Operating System: AIX,HP-UX,i5/OS,IBM i,Linux,Solaris,Windows Supersedes Fixes: PK98076 CMVC Defect: PM11713 Byte size of APAR: 38863 Date: 2010-04-28 Abstract: Users can experience Out Of Memory exceptions following large spikes in load with large transactions. Description/symptom of problem: PM11713 resolves the following problem: ERROR DESCRIPTION: The primary symptom is an OutOfMemoryError. This may cause CNTR0019E or CNTR0020E errors, since the memory that is not being cleaned up is EJB cache memory. . The heapdump will resemble the following. com/ibm/ejs/util/cache/BackgroundLruEvictionStrategy, which has a com/ibm/ejs/util/cache/CacheElementEnumerator, which has an array of com/ibm/ejs/util/cache/Element objects. Each com/ibm/ejs/util/cache/Element has a com/ibm/ejs/container/activator/TransactionKey, which has a com/ibm/ejs/container/ContainerTx, which then has an ArrayList which contains thousands of com/ibm/ejs/container/ContainerManagedBeanO objects. . The problem was reported using IBM WebSphere Commerce Suite, so the beans in the EJB cache are EJBs from that product. The issue is not with the CommerceSuite beans, but with how the WebSphere Application Server cache is handling them. . The key symptom in terms of the heap is the Array of thousands of ContainerManagedBeanO objects tied to a CacheElementEnumerator. .-------------------------------------------------------- This shows that the memory that is potentially being leaked is tied to the EJB cache that WebSphere AppServer maintains. 317,803,376 (33%) [176] 5 com/ibm/ejs/util/cache/BackgroundLruEvictionStrategy 0x72ebe0c8 317,803,064 (33%) [56] 3 com/ibm/ejs/util/cache/CacheElementEnumerator 0x8243f630 317,802,912 (33%) [472] 26 array of java/lang/Object 0xa70484b8 .-------------------------------------- Here is one of the elements in the EJB Cache. 238,307,992 (25%) [40] 2 com/ibm/ejs/util/cache/Element 0xb6ae1778 238,307,256 (25%) [24] 2 com/ibm/ejs/container/activator/TransactionKey 0xb6ae17a0 238,306,896 (25%) [136] 10 com/ibm/ejs/container/ContainerTx 0x88a960a0 164,283,672 (17%) [24] 1 java/util/ArrayList 0x88d8f5c8 . Here is that giant array of beans. It is directly tied to a cached EJB. The array consists of lots and lots of com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXML DefinHomeBean_c498873a beans. What this says to me is that the order objects are not being cleaned up and are accumulating. There is an array of 185,387 ContainerManagedBeanO objects. 164,283,648 (17%) [794,328] 185,387 array of java/lang/Object 0xa4bd5008 21,504 (0%) [176] 10 com/ibm/ejs/container/ContainerManagedBeanO 0x761eb0c8 21,504 (0%) [176] 10 com/ibm/ejs/container/ContainerManagedBeanO 0x96b5b6e0 21,504 (0%) [176] 10 com/ibm/ejs/container/ContainerManagedBeanO 0x9a4ea550 21,224 (0%) [48] 4 com/ibm/commerce/order/objects/TradingPositionContainerXMLDefini tionBean 0x9a4ea600 104 (0%) [24] 1 com/ibm/ejs/j2c/HandleList 0x9a4ea538 825,919,352 (88%) [200] 33 com/ibm/ejs/container/EJSContainer 0x8240da48 959,128 (0%) [160] 18 com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXML DefinHom eBean_c498873a 0x8f0e5d38 957,072 (0%) [56] 4 com/ibm/ejs/csi/PoolImplThreadSafe 0x8f0e8070 1,576 (0%) [248] 15 com/ibm/ws/pmi/server/modules/BeanModule 0x8f0e59c0 872 (0%) [80] 16 array of java/lang/String 0x7a456348 128 (0%) [72] 5 com/ibm/commerce/order/objects/EJSJDBCPersisterCMPTradingPositio nContainerXMLDefinitionBean_c498873a 0x8f0c2948 24 (0%) [24] 2 com/ibm/ejs/container/activator/OptCEntityActivationStrategy 0x82495070 16 (0%) [16] 0 com/ibm/ejs/container/lock/LockStrategy 0x9326ee08 21,504 (0%) [176] 10 com/ibm/ejs/container/ContainerManagedBeanO 0x8f0fe698 21,224 (0%) [48] 4 com/ibm/commerce/order/objects/TradingPositionContainerXMLDefini tionBean 0x8f0fe748 104 (0%) [24] 1 com/ibm/ejs/j2c/HandleList 0x8f0fe680 825,919,352 (88%) [200] 33 com/ibm/ejs/container/EJSContainer 0x8240da48 959,128 (0%) [160] 18 com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXML DefinHomeBean_c498873a 0x8f0e5d38 LOCAL FIX: None PROBLEM SUMMARY USERS AFFECTED: All users of IBM WebSphere Application Server containing large or long-running transactions. PROBLEM DESCRIPTION: Users can experience Out Of Memory exceptions following large spikes in load with large transactions. RECOMMENDATION: None Out Of Memory issues may result after large spikes in load. Analysis of the heap dumps should expose large "inactive" transactions with EJB references being held by the BackgroundLruEvictionStrategy. This APAR is a follow-on APAR to PK98076 that addresses a separate issue in the same area of code. PROBLEM CONCLUSION: Modified the EJBContainer code to completely clear the copy of the cache bucket so that all items will be garbage collected prior to the next valid sweep. The fix for this APAR is currently targeted for inclusion in fix packs 7.0.0.11, 6.1.0.33, Feature Pack for EJB3.0 6.1.0.33, and 6.0.2.43. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980 Directions to apply fix: Fix applies to Editions: Release 6 X_ Application Server (Express or BASE) X_ Network Deployment (ND) __ WebSphere Business Integration Server Foundation (WBISF) __ Edge Components __ Developer __ Extended Deployment (XD) Install Fix to: Method: _X_ Application Server Nodes __ Deployment Manager Nodes __ Both Directions to apply fix to V7: Fix applies to Editions: Release 7 X_ Application Server (Express or BASE) X_ Network Deployment (ND) __ Edge Components __ Developer NOTE: The user must: * For V6, have Administrative rights in Windows, or be the Actual Root User in a UNIX environments. * For V7, logged in with the same authority level when unpacking a fix, fix pack or refresh pack. * For V6, be at V6.1.0.13 or newer of the Update Installer and for V7, be at V7.0.0.0 or newer of the Update Installer. Certain iFixes may require a newer version of the Update Installer and the Update Installer will inform you during the installation process if a newer version is required. This can be checked by reviewing the level of the Update Installer in file /updateinstaller/version.txt. The Update Installer can be downloaded from the following link: http://www.ibm.com/support/docview.wss?rs=180&uid=swg21205991 For detailed instructions to Extract the Update Installer see the following Technote: http://www.ibm.com/support/docview.wss?rs=180&uid=swg21205400 Note that there are two different methods for delivering iFixes, depending on the contents. The fix may be delivered either as a single file with a .pak extension (such as 6.1.0.11-WS-WAS-IFPK12345.pak) or a single file with a .zip extension (such as 6.1.0.11-WS-WAS-IFPK12345.zip) which then contains one or more files with a .pak extension. 1) If your iFix is delivered as a single file with a .pak extension, Copy the .pak file directly to the maintenance directory. If your iFix is delivered as a single file with a .zip extension, unzip the file into the maintenance directory. 2) Shutdown WebSphere Manually execute setupCmdLine.bat in Windows or . ./setupCmdLine.sh in Unix from the WebSphere instance that maintenance is being applied to. 3) For V7 IBM i users, use the update command to install and uninstall the interim fix. The IBM Information Center can provide additional details, if needed, on the use of this command. http://www14.software.ibm.com/webapp/wsbroker/redirect?version=compass&product=was-nd-iseries&topic=rins_update. For V7 non-IBM i users and V6 users, launch the Update Installer and click the Next button on the Welcome page. 4) Enter the directory path of the installation location of the WebSphere product you want to update, and click the Next button. 5) Select the "Install maintenance package" operation and click the Next button. 6) Enter the directory path of your maintenance directory where you have the maintenance packages (.pak files) and click the Next button. 7) The Available Maintenance Package to Install page should list all maintenance packages (.pak files) that it finds in the directory path provided in the previous step. The Update Installer will select the correct maintenance packages based on your system configuration and will not allow an invalid combination to be installed. Please keep the Update Installer recommendations and click the Next button and continue with the installation of the maintenance package. To determine why some maintenance packages have been identified as not applicable, see description in log found in /logs/tmp*/updatelogs.txt 8) Please note that in the future, if a Feature Pack is installed or uninstalled, a different set of iFixes will be needed. Use the Update Installer again at that time, with the maintenance directory location where these maintenance packages are stored, to determine the required interim fixes for the new WebSphere and Feature Pack(s) combination. 9) The maintenance packages could have one of a set of names, and these names will help determine which maintenance package you need to install. The APAR name (PKxxxxx) should appear as part of the filename. Between the APAR number and the .pak extension there will be 0 to 2 characters added. The table below indicates the usage of each of the maintenance packages with respect to which Feature Packs, if any, are installed. |.pak File Names |No Feature Packs|EJB3 Only|WebServices Only| Both | |6.1.0.x-WS-WAS-IFPK12345 | X | X | X | X | |6.1.0.x-WS-WAS-IFPK12345C | X | | | | |6.1.0.x-WS-WAS-IFPK12345CE | | X | X | X | |6.1.0.x-WS-WAS-IFPK12345W | X | X | | | |6.1.0.x-WS-WAS-IFPK12345WE | | | X | X | |6.1.0.x-WS-WAS-IFPK12345E | X | | X | | |6.1.0.x-WS-WAS-IFPK12345EE | | X | | X | 10) For V7, all platforms except Windows. In pre-install summary panel, use the "verify permission" feature to verify the user has permissions to apply updates to files associated with the selected maintenance. Correct any file permissions before clicking next to start the install. 11) Restart WebSphere Directions to remove fix: NOTE: * The user must have Administrative rights in Windows, or be the Actual Root User in a UNIX environments. * FIXES MUST BE REMOVED IN THE ORDER THEY WERE APPLIED * DO NOT REMOVE A FIX UNLESS ALL FIXES APPLIED AFTER IT HAVE FIRST BEEN REMOVED * YOU MAY REAPPLY ANY REMOVED FIX Example: If your system has fix1, fix2, and fix3 applied in that order and fix2 is to be removed, fix3 must be removed first, fix2 removed, and fix3 re-applied. 1) Shutdown WebSphere Application Server. Manually execute setupCmdLine.bat in Windows or . ./setupCmdLine.sh in Unix from the WebSphere instance that uninstall is being run against. 2) Start Update Installer 3) Enter the installation location of the WebSphere product you want to remove the fix. 4) Select "Uninstall maintenance package" operation. 5) Select file name of the maintenance package to uninstall (for example, 6.1.0.11-WS-WAS-IFPK12345.pak). 6) Verify the Uninstallation Summary and continue with the uninstall. 7) Restart WebSphere Directions to re-apply fix: 1) Shutdown WebSphere Application Server. 2) Follow the Fix instructions to apply the fix. 3) Restart WebSphere Application Server. Additional Information: