MustGather: Crash on AIX produces no core or a truncated core
 Technote (FAQ)
 
Problem
This document outlines what needs to be done to ensure that a full core file is produced on AIX® when the Application Server crashes. Gathering this information before calling IBM support helps familiarize you with the troubleshooting process and saves you time.
 
 
Solution
If you already contacted support, continue to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.


Ensuring that AIX is set up to accept a full core
  1. Go into SMIT and ensure that full core is enabled; or, from a command line:

    lsattr -Elsys0|grep full

    To set full core to true:

    chdev -a fullcore=true -lsys0

  2. Ensure that your ulimit -c and ulimit -f is set high enough. Note that units are in 512-byte blocks. Set to unlimited to ensure that you have enough free file system space in your install_root\bin directory (at least 750 MB to 2 GB).

    To check the current settings run the following commands:

    ulimit -c
    ulimit -f

    To set to unlimited, run the following commands:

    ulimit -c unlimited
    ulimit -f unlimited

  3. Ensure that the WebSphere® Application Server process can write to the install_root\bin directory (default), unless the workingDirectory variable is changed.

  4. Check /etc/security/limits file. The stanza for the user that runs the process should have fsize = -1 and core = -1. Setting these values to -1 changes the setting to unlimited; you must ensure that you have enough free space to handle a large core file.

  5. Issue a kill -11 (against PID of the Java™ process) to test getting a core file.

  6. If you still do not get a core file after setting these parameters correctly, do the following:
    1. Add the following lines to the beginning of startupServer.sh:

      DISABLE_JAVADUMP=true
      export DISABLE_JAVADUMP

    2. Restart all WebSphere java processes.

    3. Issue a kill -3 against the PID of the java process to test getting a core file.

      Note: Normally a kill -3 generates a javacore.txt file. However, setting DISABLE_JAVADUMP=true causes a kill -3 to generate a system core file.

    4. If you can get a core from the test, wait for a crash to occur during an error condition

  7. If you are using MQ, it might trap the signal and not propagate to the Java process. Disable MQ signal handling, by setting env variable.

    Example:

    export MQS_NO_SYNC_SIGNAL_HANDLING=true

  8. If core file is still not generated, use the attached pdump.sh script that you can run against the process prior to running the kill command. This will create a file pdump.java.###.txt file. Locate the line containing sigcatch ( # grep -p sigcatch pdump.java.*). If SEGV is listed in output, then something is handling this signal.

    Example:

    pdump.sh <Java_PID>


Note: Search for core files in the following directories:
  1. install_root/bin
  2. Configured Application Server working directory.
  3. /tmp

If you cannot find a core file in any of these locations, search your entire machine for core* files.

Follow instructions to send diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to the Crash component, search the WebSphere Application Server support site.
 
Related information
Additional commands to ensure core generation
Submitting information to IBM support
Steps to getting support
MustGather: Read first
Troubleshooting guide
 
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Crash
Operating system(s): AIX
Software version: 6.0
Software edition:
Reference #: 1052642
IBM Group: Software Group
Modified date: Mar 17, 2005