MustGather: Kill -3 command does not generate javacore during no response (hang) or performance degradation on AIX

Technote (FAQ)
Problem
MustGather for when your WebSphere® Application Server is not responding (hangs) on AIX® and the Kill -3 command does not generate a javacore file. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.
Solution
If you have already contacted support, continue on to the component's specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

No javacore generated specific MustGather:

The list documents what is needed to troubleshoot a hung Java™ virtual machine (JVM™) on an AIX® platform when the command kill -3 does not generate a javacore.txt file

Issuing a kill -3 against a Java process normally results in the generation of a javacore.txt file. However, there are scenarios when the JVM gets into a state in which javacores are not generated as a result of the kill -3. Follow these instructions if kill -3 normally gives you javacores, but when the JVM becomes hung or degraded a kill -3 does not produce javacores.

Machine setup

  1. export DISABLE_JAVADUMP=true

    This should set an environment variable in any script that starts a Java process.

  2. Restart all Java processes.


When hang occurs
  1. netstat -an > netstat.out

    Note: If the Web server is remote, gather netstat output on the Application Server machine AND Web server machine.

  2. vmstat 5 12 > vmstat.out

  3. This step differs based upon the version of AIX:
    1. For V4.3.3:svmon -P -m <pid> -i 600 > svmon.out
    2. For V5.1:svmon -P -m -r <pid> -i 600 > svmon.out
    3. For V5.2:svmon -P <pid> -m -r -i 600 > svmon.out

  4. Can you serve a static document from the Web server?

  5. tprof -k -s -e -x sleep 60
    1. pdump.sh java_pid
    2. kill -11 java_pid

  6. chmod 755 checkcore

  7. checkcore [core]> checkcore.out

    where:
    [core] is the core file generated.


Submit the following diagnostic information to IBM Support:
  1. The pdump output file.

  2. The core file.

  3. All libraries in the list generated from step 6 above (checkcore.out).

  4. The Java binary located in the directory,

    usr/WebSphere/AppServer/java/jre/bin/java

  5. Everything in install_root/logs directory (from the WebSphere Application Server machine).

  6. If the Web server is remote, send the following from the Web server machine:

    For releases of V3.5: trace.log.<webserver name>.<date>
    For releases of V4.0: native.log
    For releases of V5.0: http_plugin.log

  7. All netstat.out

  8. All vmstat.out

  9. All smvon.out

  10. Send all files generated by tprof (especially _prof.all). These files are in the directory from which the tprof command was run and begin with an underscore ( _ ). Some versions of AIX might produce just a sleep.prof file.
  11. Follow instructions tosend diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to the Hangs/Performance Degradation component, search the WebSphere Application Server support site.

Related information
Java™ virtual machine (JVM) crash on an AIX® platform

Steps to Getting Support












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Hangs/Performance Degradation
Operating system(s): AIX
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Edition Independent
Reference #: 1145345
IBM Group: Software Group
Modified date: 2004-12-08