MustGather: Out of Memory errors on AIX, Part 2a - native leak

Technote (FAQ)
Problem
MustGather for native leak problems with the WebSphere® Application Server Out of Memory component on AIX® platforms. Gathering this information before calling IBM® support will help familiarize you with the troubleshooting process and save you time.
Solution
The following steps outline how to troubleshoot java.lang.OutOfMemoryError errors on an AIX platform when you suspect that there is a native memory leak. This suspicion is based upon your analysis of the data collected from technote: MustGather: Out of Memory errors on AIX, Part 1.

If you already contacted support, continue to the component-specific MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

Out of Memory (native leak) specific MustGather information:


Set up, before problem occurs:
  1. Enabling MMAP for Java™ heap:


    • For V6.0 release:
      To enable the MMAP Java heap, add the following line to the top of the startServer.sh file in the install_root/profiles/profile_name/bin directory:

      export IBM_JAVA_MMAP_JAVA_HEAP=true


    • For V5.0 and V5.1 releases:

      To enable the MMAP Java heap, add the following line to the top of the startServer.sh file in the install_root/bin directory:

      export IBM_JAVA_MMAP_JAVA_HEAP=true

    • For V3.5 and V4.0 releases:

      To enable the MMAP Java heap, add the following line to the top of the startupServer.sh file in the install_root/bin directory:

      export IBM_JAVA_MMAP_JAVA_HEAP=true

  2. Changing the Native Heap:
    • For releases of V6.0:
      1. Verifiy the Java Virtual Machine (JVM™) is 32-bit, since that is what WebSphere Application Server supports.
      2. Issue bootinfo -K from the AIX command line.
      3. Add the following line to the top of the startServer.sh file in the install_root/profiles/profile_name/bin directory:

        export LDR_CNTRL=MAXDATA=0xn0000000

        where n= (10 - round(Xmx/256)

    • For releases of V5.0 and V5.1:
      1. Verifiy the JVM is 32-bit, since that is what WebSphere Application Server supports.
      2. Issue bootinfo -K from the AIX command line.
      3. Add the following line to the top of the startServer.sh file in the install_root/bin directory:

        export LDR_CNTRL=MAXDATA=0xn0000000

        where n= (10 - round(Xmx/256)

    • For releases of V3.5 and V4.0:
      1. Verifiy the JVM is 32-bit, since that is what WebSphere Application Server supports.
      2. Issue bootinfo -k from the AIX command line.
      3. Add the following line to the top of the startupServer.sh file in the install_root/bin directory:

        export LDR_CNTRL=MAXDATA=0xn0000000

        where n= (10 - round(Xmx/256))

    • Examples for all releases
      1. Xmx = 1024
        n = (10 - round(1024/256))
        n = (10 - round(4))
        n = (10 - 4)
        n = 6

      2. Xmx = 640
        n = (10 - round(640/256))
        n = (10 - round(2.5))
        n = (10 - 3)
        n = 7

  3. Follow instructions for Enabling verbosegc in WebSphere Application Server.

  4. Clear all application server log files before starting the Application Server to capture data.

  5. After starting the Application Server, run the AIX_433_memory_leak.sh or AIX_51_memory_leak.sh shell script (see attached ;script for your version of AIX) while trying to reproduce the problem.

    AIX_memory_leak.sh -p pid-f base_filename-i interval

    Where:
    pid is the process id to be profiled
    base_filename is a path and identifier for the output
    interval is the time in seconds between each iteration

    Example:
    AIX_memory_leak.sh -p 123456 -f /logs/leak -i 120

After problem re-occurs:

  1. Stop the Application Server and the script.

  2. Collect the following:
    • For V6.0 release:
      • All files in install_root/profiles/profile_name/logs/server_name directory.
      • All files in install_root/profiles/profile_name/logs/ffdc directory.
      • A copy of server.xml located in install_root/profiles/profile_name/config/cells/cell_name/nodes/node_name/servers/server_namedirectory.

    • For V5.0 and V5.1 releases:
      1. Include all of the files from the install_root/logs/server_namedirectory.
      2. All files in install_root/logs/ffdc directory.
      3. A copy of server.xml located at install_root/config/cells/cell_name/nodes/node_name/servers/server_name directory

    • For V3.5 and V4.0 releases:
      1. Include all of the files from the install_root/logs directory.
      2. A copy of XMLExport for the server configuration

    • For all releases:
      • If you have configured the application server to write logs into a different location, send them accordingly.
      • The log file output from memory leak script.
      • List of all other software and version information on the system. For example, databases, WebSphere MQ, and so forth.

  3. Follow instructions to send diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to the Out of Memory component, search the WebSphere Application Server support site.
Related information
MustGather: Out of Memory errors on AIX - Part 1

How to enable verbosegc in WebSphere

Submitting information to IBM support

Steps to getting support

MustGather: Readme first

Troubleshooting guide












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Out of Memory
Operating system(s): AIX
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Advanced, Base, Express, Network Deployment, Single Server
Reference #: 1138456
IBM Group: Software Group
Modified date: 2004-12-17