MustGather: Out of Memory errors on Linux, Part 2a - native leak

Technote (FAQ)
Problem
MustGather for native leak problems with the WebSphere® Application Server Out of Memory component on Linux®. Gathering this information before calling IBM® support will help familiarize you with the troubleshooting process and save you time.
Solution
The following steps outline how to troubleshoot java.lang.OutOfMemoryError errors on a Linux platform caused by the inability of the Java™ virtual machine (JVM™) to allocate native memory. This suspicion is based upon your analysis of the documentation collected from the technote: MustGather: Out of Memory Errors on Linux, Part 1.

If you already contacted support, continue to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server Products.

Out of Memory (native leak) specific MustGather information:
  1. Enable tracing of native memory allocation usingIBM_MALLOCTRACE and MALLOC_TRACE to create mtrace.log:

    • For WebSphere Application Server V6.0 release:
      1. Enable these environment variables through the administrative console by selecting the following:
        Servers > Application Servers >server_name> Java and Process Management > Process Definition > Environment Entries > New

      2. Add the following Name and Value pairs:


        Name: IBM_MALLOCTRACE
        Value:1


        Name: MALLOC_TRACE
        Value:memory_log_path


        Where memory_log_path is install_root/profiles/profile_name/logs/server_name/mtrace.log

    • For WebSphere Application Server V5.0 and V5.1 releases:
      1. Enable these environment variables through the administrative console by selecting the following:
        Servers > Application Servers >server_name> Process Definition > Environment Entries > New

      2. Add the following Name and Value pairs:


        Name: IBM_MALLOCTRACE
        Value:1


        Name: MALLOC_TRACE
        Value:memory_log_path


        Where memory_log_path is install_root/logs/server_name/mtrace.log

    • For WebSphere Application Server V3.5 and V4.0 releases:
      Add the following lines to the top of the startupServer.sh file in the install_root/bin directory:
      export IBM_MALLOCTRACE=1
      export MALLOC_TRACE=
      memory_log_path

      Where memory_log_path is install_root/logs/mtrace.log
  2. Follow instructions for Enabling verbosegc in WebSphere Application Server.

  3. Delete all WebSphere Application Server and FFDC log files. Back up them first, if needed.

  4. After starting WebSphere Application Server, run the Linux_memory_leak.sh shell script (attached) while trying to reproduce the problem. Ensure that you update the following lines of the script before running it:
    1. log=file
    2. pid=PID_of_Java_process_that_leaks_memory
    3. Change the amount of time the script sleeps, "sleep 600", based on how long it takes to reproduce the problem.

      For example, if the problem can be reproduced in 1 hour, update the sleep interval to 300 so that the documentation is collected every 5 minutes instead of 10. For a problem that takes 2 days to reproduce, 1200 is a good sleep interval.

  5. Wait for problem to re-occur.

  6. Stop the Application Server and the script.

  7. Collect the following files:
    • For WebSphere Application Server V6.0 release:
      • The systemErr.log, systemOut.log, native_stderr.log and native_stout.log files from install_root/profiles/profile_name/logs/server_namedirectory
      • Everything in the install_root/profiles/profile_name/logs/ffdc directory
      • Server.xml file from the directory:
        install_root/profiles/profile_name/config/cells/cell_name/nodes/node_name/servers/server_name
      • All documents requested for all releases below

    • For WebSphere Application Server V5.0 and V5.1 releases:
      • The systemErr.log, systemOut.log, native_stderr.log, native_stout.log and startServer.log files from install_root/logs/server_namedirectory
      • Everything in the install_root/logs/ffdc directory
      • Server.xml file from the directory:
        install_root/config/cells/cell_name/nodes/node_name/servers/server_name
      • All documents requested for all releases below

    • For WebSphere Application Server V3.5 and V4.0 releases:
      • The stderr, stdout and tracefile files from install_root/logs/
      • Full XMLConfig export.
      • All documents requested for all releases below

    • For all releases
      • mtrace.log
      • The Linux_memory_leak.sh log file
      • List of all other software and version information on the system. For example, any databases and WebSphere MQ®.

  8. Follow instructions to send diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to the Out of Memory component, search the WebSphere Application Server support site.

Related information
MustGather: Out of Memory Errors on Linux - Part 1

How to enable verbosegc in WebSphere

Submitting information to IBM support

Steps for getting support

MustGather: Readme first

Troubleshooting guide












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Out of Memory
Operating system(s): Linux
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Advanced, Base, Express, Network Deployment, Single Server, Standard
Reference #: 1138462
IBM Group: Software Group
Modified date: 2004-12-15