MustGather: Out of memory errors on Solaris, part 2b - heap leak

Technote (FAQ)
Problem
MustGather for heap leak problems with the WebSphere® Application Server Out of Memory component on Solaris™. Gathering this information before calling IBM® support helps familiarize you with the troubleshooting process and saves you time.
Solution
The following steps outline how to troubleshoot java.lang.OutOfMemoryError errors on the Solaris platform when you suspect a memory leak in the Java™ heap. This suspicion is based upon your analysis of the documentation collected from the technote:
MustGather: Out of Memory Errors on Solaris, Part 1.

If you have already contacted support, continue on to the Out of Memory MustGather information. Otherwise, click: MustGather: Read first for all WebSphere Application Server products.

Out of Memory for Solaris specific MustGather information:

Debugging the Java virtual machine (JVM™) that is running out of Java heap

If it is suspected that there is a memory leak in the Java™ heap, the first step is to eliminate any misconfiguration or tuning as a potential cause.
  1. Increase the Maximum Java Heap Value (-Xmx) and test again.

  2. Ensure that the heap generations (NEW and PERM) are sized appropriately. Default values for MaxPermSize (Permanent Region) are often insufficient for applications. The Permanent Region holds class data and other very long lived resources. MaxPermSize (default 64MB) should be set to a quarter of max heap.

    For example: -XX:MaxPermSize=128m

    MaxNewSize, the Young generation is intended for short-lived objects where Java objects are created and age, from where they are collected without a Full garbage collection (GC) cycle. Young Generation (default 32MB) of the heap should be set to a quarter of max heap size.

    For example: -XX:MaxNewSize=128m

    For information on setting these properties, please see the Generic JVM Arguments section of the following document: Setting java command line arguments in WebSphere 5.x.

    For information on setting these properties, please see the Generic JVM Arguments section of the following document: Setting java command line arguments in WebSphere Application Server V6.0. (NOTE: This link will be updated soon)

    To set this in V4.0 release:
    1. In the administrative console, select the Application Server and go to the JVM Settings tab.
    2. Click on the Advanced JVM Settings button.
    3. Enter the value(s) in the Generic JVM Arguments section.
    4. Click OK
    5. Click Apply
    6. Stop and re-start the Application Server.

  3. Run in HotSpot Server mode(-server). Running in HotSpot client mode halves the size of the Permanent Region and therefore increases the stress on this part of the heap in the Sun® JVM.

    For more details read, Setting up a HotSpot server or client mode on a Java 2 SDK.

  4. Disable System GCs. Another way applications can interact with garbage collection (GC) is by invoking GCs explicitly, such as through the System.gc call. These calls force major collection, and inhibit scalability on large systems. The performance impact of explicit GCs can be measured using the unsupported flag: -XX:+DisableExplicitGC

  5. Switch to the Alternate Threading Library. The Solaris 8 operating environment supports an alternate version of libthread that binds threads to LWPs directly; this might help avoid starvation of the finalization thread. To try this, set the environment variable LD_LIBRARY_PATH to include /usr/lib/lwp before launching the JVM.

    For more details read, Changing to the alternate, many-to-many threading library on Solaris™ 8 improves performance.

  6. If making configuration changes does not resolve the problem and a Java™ heap leak is still present, proceed reading this document.

    IBM support is proficient at interpreting heap dumps by analyzing them with the IBM-developed HeapRoots tool.

    To generate a heapdump for analysis by HeapRoots, you must set up an agent on the JVM prior to reproducing your problem. To obtain this agent, and directions to configure it, open up a Problem Management Record (PMR) with WebSphere Application Server support. If you are in the United States, call 1-800-IBM-SERV. If you are outside of the United States, please call your local IBM support center.

    Disclaimer: The HeapRoots tool is an experimental agent provided on an as-is basis with very limited support. It attempts to interact as safely as possible with classic and hotspot JVM, but might cause crashes or hangs, particularly when the JVM is under stress. Because of this, IBM support recommends that it not be used in a production or critical environment.
  7. Follow instructions tosend diagnostic information to IBM support.

For a listing of all technotes, downloads, and educational materials specific to the Out of Memory component, search the WebSphere Application Server support site.

Related information
Enabling verbosegc in WebSphere® Application Server

Out of Memory Errors on Solaris - Part 1

Java HotSpot VM Options

Tuning Garbage Collection with the 1.3.1 Java Virtual M

Steps to Getting Support

Submitting information to IBM support

MustGather: Readme first

Troubleshooting guide












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Out of Memory
Operating system(s): Solaris
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Advanced, Base, Express, Network Deployment, Single Server, Standard
Reference #: 1145349
IBM Group: Software Group
Modified date: 2004-12-15