MustGather: No response (hang) on AIX 5.2 and higher versions produces no javacore
 Technote (troubleshooting)
 
Problem(Abstract)
MustGather for when your WebSphere® Application Server is not responding (hangs) on the AIX® 5.2 platform and a "kill -3" command does not produce javacore files. Gathering this information before calling IBM® support will help familiarize you with the troubleshooting process and save you time.
 
Resolving the problem
If you already contacted support, continue to the component-specific MustGather information. Otherwise, click on the following link: MustGather: Read first for all WebSphere Application Server products.
No javacore generated specific MustGather information
The following steps will ensure that AIX is set up to accept a full core:
  1. Go into SMIT and ensure that full core is enabled; or, from a command line, enter the following:

    lsattr -Elsys0 | grep full

    To set full core to true:

    chdev -a fullcore=true -lsys0

  2. Ensure that your ulimit -c and ulimit -f is set high enough.

    Note: Units are in 512-byte blocks. Set to unlimited to ensure that you have enough free file system space in your install_root\bin directory (at least 750 MB to 2 GB).

    To check the current settings run the following commands:

    ulimit -c
    ulimit -f

    To set to unlimited, run the following commands:

    ulimit -c unlimited
    ulimit -f unlimited

  3. Ensure that the WebSphere Application Server process can write to the install_root\bin directory (default, unless workingDirectory variable is changed).

  4. Check the file, /etc/security/limits. The stanza for the user that runs the process should have fsize = -1 and core = -1. Setting these values to -1 changes the setting to unlimited; you must ensure that you have enough free space to handle a large core file.

  5. Enter the following command:

    gencore ProcessID FileName

    Example: To generate a core file, named core.1095, for the process with process ID 1095, enter the following command:

    gencore 1095 core.1095

    The gencore command creates a core file of the process specified by the ProcessID without terminating the process.

  6. If you still do not get a core file after setting these parameters correctly, do the following:
    1. Add the following lines to the beginning of the file, startupServer.sh:

      export DISABLE_JAVADUMP
      export IBM_NOSIGHANDLER=true

    2. Restart all WebSphere Application Server java processes.

    3. Enter the following command against the PID of the java process to test getting a core file:

      gencore ProcessID Filename

  7. If you can get a core from the test, wait for a hang to occur during an error condition.

    If you are using WebSphere MQ, it might trap the signal and not propagate to the Java™ process.

    Disable the WebSphere MQ signal handling by setting the following environment variable:

    MQS_NO_SYNC_SIGNAL_HANDLING

    Example:

    export MQS_NO_SYNC_SIGNAL_HANDLING=true

  8. When the Java process hangs, perform the following steps:
    1. Enter the following command:

      gencore ProcessID core1.ProcessID

    2. Wait for 3 to 5 minutes to allow for core file generation.

    3. Enter the following command:

      gencore ProcessID core2.ProcessID

    4. Wait for 3 to 5 minutes to allow for core file generation.

    5. Enter the following command:

      gencore ProcessID core3.ProcessID

    6. Enter the following command:

      ./getlib.sh java_executable_fullpath core_filename

      This creates getlib_archive.tar.Z.

      Note: You must have dbx installed to run this command.

    7. Enter the following command:

      tar cvf core_file_name.tar core_file_name

      This creates core_file_name.tar.

    8. If the Developer Kit is 131 SR5 (131-20030618) or later, run the following command against each core file:

      ./jextract -o coreN.sdff [CORE_PATH]

      Where,
      jextract is found in the install_root/java/jre/bin directory coreN.sdff is the output file and "N" is the number (for example: 1, 2, 3, and so on)
      [CORE_PATH] is the path and file name for the core file

  9. Collect the following diagnostic information:
    • All dbxtrace.out files
    • All standard data file format (SDFF) files
    • errpt.out
    • All getlib_archive.tar.Z files
    • All core.tar files
    • All WebSphere Application Server log files for V5.0 and V5.1:
      • Include all of the files from the following directory:

        install_root\logs\server_name

        Include the following Application Server log files if they are located in a different location:
        • SystemOut
        • SystemErr
        • native_stdout
        • native_stderr

      • Send the server.xml file located in the following directory:

        install_root\config\cells\cell_name\
        nodes\
        node_name\servers\server_name

    • All WebSphere Application Server log files for V6.0:
      • Include all of the files from the following directory:

        install_root\profiles\profile_name\logs\server_name

        Include the following Application Server log files if they are located in a different location:
        • SystemOut
        • SystemErr
        • native_stdout
        • native_stderr

      • Send the server.xml file located in the following directory:

        install_root\profiles\profile_name\
        configcells\
        cell_name\nodes\node_name\
        servers\
        server_name

    Note: See the attached scripts dbxtrace_aix.sh and getlib.sh. To run the dbxtrace_aix.sh file, you must have the dbx component of AIX installed on the machine.

  10. Follow instructions to send diagnostic information to IBM support.


For a listing of all technotes, downloads, and educational materials specific to the Hangs/Performance Degradation component, search the WebSphere Application Server support site.
 
Related information
MustGather: Crash on AIX
Submitting information to IBM support
Steps to getting support
MustGather: Read first
Troubleshooting guide
 
dbxtrace_aix.shgetlib.sh
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Hangs/Performance Degradation
Operating system(s): AIX
Software version: 6.0
Software edition:
Reference #: 1205260
IBM Group: Software Group
Modified date: Feb 13, 2007