Fix (APAR):  PI68803

Status:  Fix

Release:  8.5.5.10,8.5.5.9,8.5.5.8,8.5.5.7

Operating System:  z/OS

Supersedes Fixes:  

CMVC Defect:  xxxxxx

Byte size of APAR:  10464278

Date: 2016-12-02

Abstract:  Apache on z/OS CPU usage increases in release 8.5.5.5 or beyond

Description/symptom of problem:  

PI68803 resolves the following problem:

ERROR DESCRIPTION:
High CPU usage observed in Apache webserver processes following 
upgrade to service level 8.5.5.5 or beyond.                     
Under the same inbound workload the CPU usage might be 2x or 3x 
higher at the later service level.                              
Dumps show intermittent long periods in z/OS UNIX module        
BPXVRCAN which supports asyncio() cancel operations.            
High CPU is due to increasing length of the z/OS UNIX Arq       
chain, which is maintained on a per process basis. An Arq is    
used to represent an individual asycnio() request.              
Apache code paths involved with asyncio() activity              
were updated to resolve a couple of issues.                     
(a) a fix to detect and fix one path which might lead to        
multiple asyncio() cancel operations for one fd                 
(b) a fix to cope with BPX4AIO cancel return code of 0x117803fd 
With these fixes in place it is anticipated that no leaking of  
Arq elements should occur.                                      
NB. Diagnosis might be difficult since the periods in           
BPXVRCAN occur as a result of an asyncio() cancel operation     
and may be in the range of 3-20ms wallclock duration. As a      
result it's possible that the system trace in a specific dump   
may not capture the period when this occurs.                    
Note to IBM L2: Please see z-page for further advice            
                                                                
PROBLEM SUMMARY:
IHS regularly cancels asyncio operations using an AIOCB 
allocated from the stack whose "unused" fields were not 
explicitly intiailized. This would result in occasional 
failures to cancel the outstanding asyncio calls. 
If many asyncio calls became orphaned in this way, legitimate 
calls to cancel would begin to take more and more CPU time. 

PROBLEM CONCLUSION:
IHS was updated to always zero out the AIOCB structure used to 
perform asyncio cancels.  Additionally, some extra safeguards we 
added around reuse of some related datastructures.

Note: This interim fix also includes fixes for the following APARs:
 PI59561 :Add Pre/Post password hooks to mod_authnz_saf (MFA base support)
 PI64628 :IBM HTTP Server on z/OS is deleting the wrong IPC message queue
 PI65855 :CVE-2016-5387 (HTTPOXY) for IBM HTTP Server
 PI66183 :When MFA is configured, SAFRunAs fails with a permission error
 
The fix for this issue is targeted for the following IHS fixpack 
 - 9.0.0.2 
 - 8.5.5.11 
 - (not applicable to earlier releases) 




Directions to apply fix:  
Special Instructions: None

NOTE: The user must:
* Be at V1.4.3 or newer of the Installation Manager. Certain iFixes may require
  a newer version of the Installation Manager and the Installation Manager will
  inform you during the installation process if a newer version is required.
* Be logged in with the same authority level when unpacking a fix, fix pack, or
  refresh pack.

The IBM Knowledge Center can provide details, if needed, on the use of the
Installation Manager to apply the interim fix:
http://publib.boulder.ibm.com/infocenter/install/v1r4/index.jsp.

1) Shutdown IBM HTTP Server
2) Apply the interim fix using Installation Manager
3) Restart IBM HTTP Server




Directions to remove fix:  
The IBM Knowledge Center can provide details, if needed, on the use of the
Installation Manager to remove the interim fix:
http://publib.boulder.ibm.com/infocenter/install/v1r4/index.jsp.

1) Shutdown IBM HTTP Server
2) Remove the interim fix using Installation Manager
3) Restart IBM HTTP Server




Directions to re-apply fix:  
1) Stop IBM HTTP Server.
2) Follow the directions to apply the fix.
3) Restart IBM HTTP Server.




Additional Information: