Interpreting svmon output to determine if a native memory leak is the cause of a java.lang.OutOfMemoryError
 Technote (troubleshooting)
 
Problem(Abstract)
A java.lang.OutOfMemoryError issued by the Java™ Virtual Machine (JVM™) can indicate a memory leak.

Monitoring heap memory usage in the verboseGC output shows decreasing amounts of free space on the Java heap if the java.lang.OutOfMemoryError exception is do to a leak of Java objects. However, if the verboseGC output does not show this behavior, it is probable that the java.lang.OutOfMemoryError is caused by a native memory leak.

This technote describes how to interpret svmon data to determine if a native memory leak has caused the java.lang.OutOfMemoryError.
 
Cause
A native memory leak is caused by a native library that is allocating memory, but not freeing it.
 
Resolving the problem
This technote assumes you have already followed
"MustGather: Out of Memory errors on AIX, Linux, or Windows" and collected the necessary documentation.

Use the svmon data to determine which segments of the process are growing. If the growth is in the native heap segments of the process, the problem is due to a native memory leak. Here is a sample output from SVMON:


-----------------------------------------------------------------
   Pid Command  Inuse   Pin   Pgsp  Virtual 64-bit Mthrd LPage
 90190 java     198140  3252  1857  191470     N     Y     N

    Vsid  Esid    Type Description      LPage   Inuse  Pin  Pgsp  Virtual
   33899    4     work working storage    -     65536  0    0      65536
    1760    3     work working storage    -     65519  0    0      65519
   2f136    6     work working storage    -     44158  0    0      44158
   149aa    d     work loader segment     -     8622   0    19     8679
       0    0     work kernel segment     -     4779   3115 1838   5631
   359ba    -     pers /dev/lv04:2053     -     1839   0    -      -
   13949    5     work working storage    -     798    0    0      798
   116c8    -     pers /dev/lv01:2125     -     571    0    -      -
   3991c    7     work working storage    -     560    0    0      560


This is the output from a screenshot of the svmon utility:
  • ESID is the segment ID for the segment being used. The ESID corresponds to the segment numbers referenced above.

  • INUSE is the memory (in pages) used by the segment. Each page is 4KB. The maximum number of pages that may be taken up by a segment is 65536.

Determining if there is a native memory leak
Check INUSE value by taking svmon snapshots over time. A memory leak exists if INUSE increases for any ESIDs that correspond to the native heap in each subsequent snapshot.

This sample shows svmon data for segment ESID from five consecutive svmon snapshots. Notice that the INUSE value of ESID 5 (also known as segment 5) appears to be increasing over time. This is evidence of a memory leak in that segment:


Vsid Esid Type Description LPage Inuse Pin Pgsp Virtual
13949 5 work working storage - 798 0 0 798
13949 5 work working storage - 1798 0 0 798
13949 5 work working storage - 3798 0 0 798
13949 5 work working storage - 5798 0 0 798
13949 5 work working storage - 10000 0 0 798
and so on...


INUSE growth can indicate a problem with:
  • Java SDK Java Native Interface (JNI) methods

  • Native calls elsewhere

    For example, JNI calls from a user application, JDBC™ drivers for DB2®, WebSphere® MQ libraries, or third party native code.

    Check for:
    • Nonstandard libraries in the application; determine if the native leak is related to these.
    • Which segment is leaking memory; this can be an indication of the application that is leaking memory. For example, DB2 CLI code uses segment 3, while MQ uses segment 8.
  • The Just In-Time (JIT) compiler might be using the native memory. Disabling JIT might eliminate the memory leak. If this does prevent a memory leak, this is indicative of a problem in the JIT compiler of the JDK. Report this to IBM support.
 
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Java SDK
Operating system(s): AIX
Software version: 6.0
Software edition:
Reference #: 1174676
IBM Group: Software Group
Modified date: Sep 8, 2008