Interpreting svmon output to determine if a native memory leak is the cause of a java.lang.OutOfMemoryError

Technote (FAQ)
Problem
A java.lang.OutOfMemoryError issued by the Java™ Virtual Machine (JVM™) can indicate a memory leak.

Monitoring heap memory usage in the verboseGC output shows decreasing amounts of free space on the Java heap if the java.lang.OutOfMemoryError exception is do to a leak of Java objects. However, if the verboseGC output does not show this behavior, it is probable that the java.lang.OutOfMemoryError is caused by a native memory leak.

This technote describes how to interpret svmon data to determine if a native memory leak has caused the java.lang.OutOfMemoryError.
Cause
A native memory leak is caused by a native library that is allocating memory, but not freeing it.
Solution

This technote assumes you have already followed MustGather: Out of Memory errors on AIX, Part 2a - native leak and collected the necessary documentation.

Use the svmon data to determine which segments of the process are growing. If the growth is in the native heap segments of the process, the problem is due to a native memory leak. Here is a sample output from SVMON:


-----------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage
90190 java 198140 3252 1857 191470 N Y N

Vsid Esid Type Description LPage Inuse Pin Pgsp Virtual
33899 4 work working storage - 65536 0 0 65536
1760 3 work working storage - 65519 0 0 65519
2f136 6 work working storage - 44158 0 0 44158
149aa d work loader segment - 8622 0 19 8679
0 0 work kernel segment - 4779 3115 1838 5631
359ba - pers /dev/lv04:2053 - 1839 0 - -
13949 5 work working storage - 798 0 0 798
116c8 - pers /dev/lv01:2125 - 571 0 - -
3991c 7 work working storage - 560 0 0 560


This is the output from a screenshot of the svmon utility:
  • ESID is the segment ID for the segment being used. The ESID corresponds to the segment numbers referenced above.

  • INUSE is the memory (in bytes) used by the segment. The maximum INUSE size for 1 segment is 65536.

Determining if there is a native memory leak
Check INUSE value by taking svmon snapshots over time. A memory leak exists if INUSE increases for any ESIDs that correspond to the native heap in each subsequent snapshot.

This sample shows svmon data for segment ESID from five consecutive svmon snapshots. Notice that the INUSE value of ESID 5 (also known as segment 5) appears to be increasing over time. This is evidence of a memory leak in that segment:


Vsid Esid Type Description LPage Inuse Pin Pgsp Virtual
13949 5 work working storage - 798 0 0 798
13949 5 work working storage - 1798 0 0 798
13949 5 work working storage - 3798 0 0 798
13949 5 work working storage - 5798 0 0 798
13949 5 work working storage - 10000 0 0 798
etc...


INUSE growth can indicate a problem with:
  • Java SDK Java Native Interface (JNI) methods

  • Native calls elsewhere

    For example, JNI calls from a user application, JDBC™ drivers for DB2®, WebSphere® MQ libraries, or third party native code.

    Check for:
    • Nonstandard libraries in the application; determine if the native leak is related to these.
    • Which segment is leaking memory; this can be an indication of the application that is leaking memory. For example, DB2 CLI code uses segment 3, while MQ uses segment 8.
  • The Just In-Time (JIT) compiler might be using the native memory. Disabling JIT might eliminate the memory leak. If this does prevent a memory leak, this is indicative of a problem in the JIT compiler of the JDK. Report this to IBM support.











Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Java SDK
Operating system(s): AIX
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Edition Independent
Reference #: 1174676
IBM Group: Software Group
Modified date: 2004-07-20