Fix (APAR): PK72336 Status: Fix Release: 6.1.0.19 Operating System: AIX,HP-UX,i5/OS,IBM i,Linux,Linux pSeries,Linux Red Hat - pSeries,Linux zSeries Supersedes Fixes: CMVC Defect: xxxxxx Byte size of APAR: 24868 Date: 2008-10-10 Abstract: WebContainer threads hang and/or missing response data resulting from a forwarded request. Description/symptom of problem: PK72336 resolves the following problem: ERROR DESCRIPTION:? WebContainer threads hang and/or missing response data resulting from a forwarded request. For example: WebContainer threads become unusable after the following exception is logged. java.lang.NullPointerException at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:111) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:195) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:743) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:873) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1473) LOCAL FIX:? Note that the problems only happen when response data is sent asynchronously. As a result it is possible to prevent the problems by forcing all response data to sent be synchronously. This can be done by setting the webcontainer custom property: com.ibm.ws.webcontainer.channelwritetype=sync PROBLEM SUMMARY:? USERS AFFECTED: Users of WebSphere Application Server Version 6.1.0.19 PROBLEM DESCRIPTION: Various issues including hanging threads and missing response data resulting from a forwarded request. RECOMMENDATION: None PK63328 introduced an issue where a http connection can be closed too early for a response from a forwarded request. Depending on the timing of when the http connection is closed several different symptoms have been observed and these include: - Null pointer exceptions - Missing response data - Hanging threads Other symptoms may also be possible. Note that the problem only occurs when response data is sent asynchronously and as a result are more likely to occur under load and will not happen when the following WebContainer custom property is set: com.ibm.ws.webcontainer.ChannelWriteType=sync One example of a NullPointerException exception resulting from the problem is: java.lang.NullPointerException at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:11) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:95) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResuHandler.java:743) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:87 at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1473 Another example of a NullPointerException exception resulting from the problem is: java.lang.NullPointerException at com.ibm.ws.tcp.channel.impl.NioSocketIOChannel.attemptWriteToSocketUsingNIO(NioSocketIOChannel.java:158) at com.ibm.ws.tcp.channel.impl.SocketIOChannel.attemptWriteToSocket(SocketIOChannel.java:744) at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:628) at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:979) at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1064) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1462 PROBLEM CONCLUSION:? The WebContainer has been updated to only close a connection after the sending of the response data from forward request is complete. The fix for this APAR is currently targeted for inclusion in fixpack 6.1.0.21.. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980 Directions to apply fix: NOTE: Choose the: 1) Release the fix applies to 2) The Editions that apply 3) Delete the Editions & Methods that do not apply and this Note Fix applies to Editions: Release 6.0 __ Application Server (Express or BASE) __ Network Deployment (ND) __ WebSphere Business Integration Server Foundation (WBISF) __ Edge Components __ Developer __ Extended Deployment (XD) Install Fix to: Method: __ Application Server Nodes __ Deployment Manager Nodes __ Both NOTE: The user must: * Have Administrative rights in Windows, or be the Actual Root User in a UNIX environments. * Logged in with the same authority level when unpacking a fix, fix pack or refresh pack. * Be at V6.1.0.13 or newer of the Update Installer. Certain iFixes may require a newer version of the Update Installer and the Update Installer will inform you during the installation process if a newer version is required. This can be checked by reviewing the level of the Update Installer in file /updateinstaller/version.txt. The Update Installer can be downloaded from the following link: http://www.ibm.com/support/docview.wss?rs=180&uid=swg21205991 For detailed instructions to Extract the Update Installer see the following Technote: http://www.ibm.com/support/docview.wss?rs=180&uid=swg21205400 Note that there are two different methods for delivering iFixes, depending on the contents. The fix may be delivered either as a single file with a .pak extension (such as 6.1.0.11-WS-WAS-IFPK12345.pak) or a single file with a .zip extension (such as 6.1.0.11-WS-WAS-IFPK12345.zip) which then contains one or more files with a .pak extension. 1) If your iFix is delivered as a single file with a .pak extension, Copy the .pak file directly to the maintenance directory. If your iFix is delivered as a single file with a .zip extension, unzip the file into the maintenance directory. 2) Shutdown WebSphere Manually execute setupCmdLine.bat in Windows or . ./setupCmdLine.sh in Unix from the WebSphere instance that maintenance is being applied to. 3) Launch Update Installer and click the Next button on the Welcome page. 4) Enter the directory path of the installation location of the WebSphere product you want to update, and click the Next button. 5) Select the "Install maintenance package" operation and click the Next button. 6) Enter the directory path of your maintenance directory where you have the maintenance packages (.pak files) and click the Next button. 7) The Available Maintenance Package to Install page should list all maintenance packages (.pak files) that it finds in the directory path provided in the previous step. The Update Installer will select the correct maintenance packages based on your system configuration and will not allow an invalid combination to be installed. Please keep the Update Installer recommendations and click the Next button and continue with the installation of the maintenance package. 8) Please note that in the future, if a Feature Pack is installed or uninstalled, a different set of iFixes will be needed. Use the Update Installer again at that time, with the maintenance directory location where these maintenance packages are stored, to determine the required interim fixes for the new WebSphere and Feature Pack(s) combination. 9) The maintenance packages could have one of a set of names, and these names will help determine which maintenance package you need to install. The APAR name (PKxxxxx) should appear as part of the filename. Between the APAR number and the .pak extension there will be 0 to 2 characters added. The table below indicates the usage of each of the maintenance packages with respect to which Feature Packs, if any, are installed. |.pak File Names |No Feature Packs|EJB3 Only|WebServices Only| Both | |6.1.0.x-WS-WAS-IFPK12345 | X | X | X | X | |6.1.0.x-WS-WAS-IFPK12345C | X | | | | |6.1.0.x-WS-WAS-IFPK12345C Directions to remove fix: NOTE: * The user must have Administrative rights in Windows, or be the Actual Root User in a UNIX environments. * FIXES MUST BE REMOVED IN THE ORDER THEY WERE APPLIED * DO NOT REMOVE A FIX UNLESS ALL FIXES APPLIED AFTER IT HAVE FIRST BEEN REMOVED * YOU MAY REAPPLY ANY REMOVED FIX Example: If your system has fix1, fix2, and fix3 applied in that order and fix2 is to be removed, fix3 must be removed first, fix2 removed, and fix3 re-applied. 1) Shutdown WebSphere Manually execute setupCmdLine.bat in Windows or . ./setupCmdLine.sh in Unix from the WebSphere instance that uninstall is being run against. 2) Start Update Installer 3) Enter the installation location of the WebSphere product you want to remove the fix. 4) Select "Uninstall maintenance package" operation. 5) Enter the file name of the maintenance package to uninstall (PKxxxxx.pak). 6) UnInstall maintenance package. 7) Restart WebSphere Directions to re-apply fix: 1) Shutdown WebSphere. 2) Follow the Fix instructions to apply the fix. 3) Restart WebSphere. Additional Information: