MustGather: Kill -3 command does not generate javacore
 Technote (troubleshooting)
 
Problem(Abstract)
Collecting data for when your WebSphere® Application Server is not responding (hangs) on AIX® and the Kill -3 command does not generate a javacore file. Gathering this information before calling IBM® support will help familiarize you with the troubleshooting process and save you time.
 
Resolving the problem
If you have already contacted support, continue on to the Hang/Performance degradation MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.
No javacore generated specific MustGather information
The following lists what is needed to troubleshoot a hung Java™ virtual machine (JVM) on an AIX operating system when the kill -3 command does not generate a javacore.txt file.

Issuing a kill -3 against a Java process normally results in the generation of a javacore.txt file. However, there are scenarios when the JVM gets into a state in which javacores are not generated as a result of the kill -3. Follow these instructions if the kill -3 command normally gives you javacores, but when the JVM becomes hung or degraded, a kill -3 command does not produce javacores. To determine the status of the Java threads during the hang, you will generate AIX system core files which can be processed to provide the needed information. The instructions in the Machine Setup section will insure that your server is configured to generate a full core file.

Machine Setup:
  1. Run the following command:

    export DISABLE_JAVADUMP=true

    This should set an environment variable in any script that starts a Java process.

  2. Use one of the following 2 methods to insure that AIX is configured to produce a full core file:
    • Use the System Management Interface Tool (SMIT) to create full core dumps by starting SMIT and setting the following to "TRUE":

      System Environments > Change/Show Characteristics of Operating System > Enable Full CORE dump

    • Or, enter the following commands:

      lsattr -Elsys0 | grep full

      To set full core to true:

      chdev -a fullcore=true -lsys0

  3. Ensure that your ulimit -c and ulimit -f is set high enough.

    Note: Units are in 512-byte blocks. Set to unlimited to ensure that you have enough free file system space in your install_root/bin directory (at least 750 MB to 2 GB).

    To check the current settings, run the following commands:

    ulimit -c
    ulimit -f

    To set to unlimited, run the following commands:

    ulimit -c unlimited
    ulimit -f unlimited

  4. Ensure that the WebSphere Application Server process can write to the install_root\bin directory (default, unless the workingDirectory variable is changed).

  5. Check the file, /etc/security/limits. The stanza for the user that runs the process should have fsize = -1 and core = -1. Setting these values to -1 changes the setting to unlimited; you must ensure that you have enough free space to handle a large core file.

  6. Restart all Java processes.

When the Hang occurs:
Collect the documentation as detailed in the following steps:
  1. Run the following command:

    netstat -an > netstat.out

    Note: If the Web server is remote, gather netstat output on the Application Server machine and Web server machine.

  2. Run the following command:

    vmstat 5 12 > vmstat.out

  3. This step differs based upon the version of AIX:
    • For AIX V5.2:

      svmon -P <pid> -m -r  -i 600 > svmon.out

    • For AIX V5.1:

      svmon -P -m -r <pid> -i 600 > svmon.out

    • For AIX V4.3.3:

      svmon -P -m <pid> -i 600 > svmon.out

  4. Can you serve a static document from the Web server?

  5. Collect CPU usage statistics by issuing the following command:

    tprof -k -s -e -x sleep 60

  6. Collect a process dump of the Java process by executing the pdump.sh script (attached below):

    pdump.sh java_pid

  7. Collecting system core files:
    • If you are running AIX version 5.2 or later, execute the following steps to produce a series of three AIX core files:
      1. Run the following command:

        gencore java_pid core1.process_ID

      2. Wait for 3 to 5 minutes to allow for core file generation.

      3. Repeat the preceding steps a. and b. to generate two additional core files at 3 to 5 minute intervals. It is important to remember to use a different core file name each time you run the command (for example: core2.process_id, core3.process_id).

    • If you are running AIX version 5.1 or earlier, run the following command to generate an AIX operating system core file:

      kill -11 java_pid

  8. Execute the getlib.sh script (attached below) to collect the system libraries. Using the core files and the system libraries, IBM support will be able to create Java thread stacks for the JVM. To run the script:

    ./getlib.sh java_executable_fullpath core_file_name

    This creates the getlib_archive.tar.Z file. Note: You must have dbx installed to run this command.

  9. If the Developer Kit is 131 SR5 (131-20030618) or later, run the following command against each core file:

    ./jextract -o coreN.sdff [CORE_PATH]

    Where,
    jextract is found in the install_root/java/jre/bin directory
    coreN.sdff is the output file and "N" is the number (for example: 1, 2, 3, and so on)
    [CORE_PATH] is the path and file name for the core file

  10. Gather the following information:
    • For WebSphere Application Server V6.0:
      • Include all of the files from the following directory:

        profile_root/logs/server_name

        Include the following Application Server log files if they are located in a different location:
        • Systemout
        • Systemerr
        • Native_stderr
        • Native_stdout

      • The server.xml file located in the following directory:

        profile_root/config/cells/cell_name/
        nodes/
        node_name/servers/server_name

    • For WebSphere Application Server V5.1 and 5.0:
      • Include all of the files from the following directory:

        install_root/logs/server_name

        Include the following Application Server log files if they are located in a different location:
        • Systemout
        • Systemerr
        • Native_stderr
        • Native_stdout

      • The server.xml file located in the following directory:

        install_root/config/cells/cell_name/
        nodes/
        node_name/servers/server_name

    • For all releases of WebSphere Application Server
      • The pdump output file.
      • The core files. Ensure you compress these files as they can be very large.
      • All standard data file format (*.SDFF) files
      • All getlib_archive.tar.Z files
      • If the Web server is remote, send the http_plugin.log file from the Web server machine. 
      • All netstat.out files
      • All vmstat.out files
      • All smvon.out files
      • Files generated by tprof (especially _prof.all). These files are in the directory from which the tprof command was run and begin with an underscore ( _ ). Some versions of AIX might produce just a sleep.prof file.

  11. Follow instructions to submit diagnostic information to IBM Technical Support for problem determination.

For a listing of all technotes, downloads, and educational materials specific to a hang or performance degradation, search the WebSphere Application Server support site.
 
Related information
JVM crash on an AIX operating system
Steps to getting support
Submitting information to IBM Support
MustGather: Read first
Troubleshooting guide
 
pdump.shgetlib.sh
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers WebSphere Application Server - Express Hangs/performance degradation AIX 6.0, 5.1, 5.0
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Hangs/Performance Degradation
Operating system(s): AIX
Software version: 6.0
Software edition:
Reference #: 1145345
IBM Group: Software Group
Modified date: Feb 19, 2007