MustGather: Kill -3 command does not generate javacore
 Technote (FAQ)
 
Problem
MustGather for when your WebSphere® Application Server is not responding (hangs) on AIX® and the Kill -3 command does not generate a javacore file. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.
 
Solution
If you have already contacted support, continue on to the Hang/Performance degradation MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.


No javacore generated specific MustGather information
The following lists what is needed to troubleshoot a hung Java™ virtual machine (JVM™) on an AIX® platform when the command kill -3 does not generate a javacore.txt file.

Issuing a kill -3 against a Java process normally results in the generation of a javacore.txt file. However, there are scenarios when the JVM gets into a state in which javacores are not generated as a result of the kill -3. Follow these instructions if kill -3 normally gives you javacores, but when the JVM becomes hung or degraded a kill -3 does not produce javacores.

Machine setup:
  1. export DISABLE_JAVADUMP=true

    This should set an environment variable in any script that starts a Java process.

  2. Restart all Java processes.


When hang occurs
Use the attached pdump and checkcore scripts in the following procedures.
  1. netstat -an > netstat.out

    Note: If the Web server is remote, gather netstat output on the Application Server machine AND Web server machine.

  2. vmstat 5 12 > vmstat.out

  3. This step differs based upon the version of AIX:
    1. For AIX V4.3.3:  svmon -P -m <pid> -i 600 > svmon.out
    2. For AIX V5.1:  svmon -P -m -r <pid> -i 600 > svmon.out
    3. For AIX V5.2:  svmon -P <pid> -m -r  -i 600 > svmon.out

  4. Can you serve a static document from the Web server?

  5. tprof -k -s -e -x sleep 60
    1. pdump.sh java_pid
    2. kill -11 java_pid

  6. chmod 755 checkcore

  7. checkcore [core] > checkcore.out

    Where:
    [core] is the core file generated.

  8. Gather the following information:
    • The pdump output file.
    • The core file.
    • All libraries in the list generated from step 6 above (checkcore.out).
    • The Java binary located in the directory,
      usr/WebSphere/AppServer/java/jre/bin/java
    • Everything in install_root/logs directory
      (from the WebSphere Application Server machine)
    • If the Web server is remote, send the following from the Web server machine:
      For releases of V4.0: native.log
      For releases of V5.0, V 5.1 and V6.0: http_plugin.log
    • All netstat.out
    • All vmstat.out
    • All smvon.out
    • Files generated by tprof (especially _prof.all). These files are in the directory from which the tprof command was run and begin with an underscore ( _ ). Some versions of AIX might produce just a sleep.prof file.

  9. Follow instructions to Submitting Diagnostic Information to IBM Technical Support for Problem Determination.

For a listing of all technotes, downloads, and educational materials specific to the Hangs/Performance Degradation component, search the WebSphere Application Server support site.
 
Related information
Java™ virtual machine (JVM) crash on an AIX® platform
Steps to getting support
Submitting information to IBM Support
MustGather: Read first
Troubleshooting guide
 
checkcorepdump.sh
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Hangs/Performance Degradation
Operating system(s): AIX
Software version: 6.0
Software edition:
Reference #: 1145345
IBM Group: Software Group
Modified date: Apr 5, 2006