|
Problem |
Blocks of free heap space less than 512 bytes in length
and lying between two allocated objects are known as "dark matter." This
space is are not released by the normal mark/sweep phase of the IBM® Java™
SDK garbage collection cycle. This space is only collected during a
compaction phase or when the objects adjacent to them are collected. In
some cases, the amount of dark matter can grow over time leading to
performance issues and heap fragmentation. |
|
Cause |
The allocation and creation of an object on the Java heap
will often leave space between objects that is less than 512 bytes. This
small portion of the heap is removed from the free space list, and to the
garbage collector, appears like a small object. This dark matter is not
collected during the normal mark/sweep phase of the IBM Java SDK garbage
collection (GC) cycle. Dark matter will remain in the heap until a
compaction phase occurs or the adjacent objects are collected. Compaction
cycles can be lengthy and can consume a large amount of the system
resources, so they are not initiated by the JVM unless it is necessary. A
compaction cycle will occur if any of the following criteria are met:
- -Xcompactgc has been specified.
- Following the mark/sweep phase, not enough contiguous free space is
available to satisfy the allocation request.
- A System.gc() has been requested and the last allocation
failure garbage collection did not perform a compaction.
- At least half the previously available memory has been consumed by
thread local heap (TLH), cache, allocations (ensuring an accurate sample)
and the average TLH size falls below 1000 bytes.
- Less than 5% of the active heap is free.
- Less than 128 KB of the active heap is free.
If the heap does not meet the above criteria and the amount of Dark Matter
continues to increase, performance issues associated with excessive or
long running GC cycles can be experienced. The JVM may begin to experience
frequent GC cycles which consume a majority of the processing cycles
resulting in severe performance degradation on the application server. It
may also result in a very long running GC cycle where a compaction is
performed causing intermittent pauses in application responsiveness.
To determine if dark matter is the source of a GC issue, enable verbose GC
and examine the GC patterns during a performance issue. If dark matter is
involved, you will should see a series of GC cycles where no compaction
takes place and a large amount of heap space is not freed. This is
followed by a GC cycle where a compaction takes place and a large amount
of heap space is reclaimed.
For example:
<AF[92]: Allocation Failure. need 300016 bytes, 3907 ms since last
AF>
<AF[92]: managing allocation failure, action=2
(20023632/1073740288)>
<GC(110): GC cycle started Thu Apr 07 08:17:20 2005
<GC(110): freed 24254168 bytes, 4% free (44277800/1073740288), in
4173 ms>
<GC(110): mark: 4055 ms, sweep: 118 ms, compact: 0 ms>
<GC(110): refs: soft 0 (age >= 32), weak 5769, final
293, phantom 0>
<AF[92]: managing allocation failure, action=3
(44277800/1073740288)>
<AF[92]: managing allocation failure, action=4
(44277800/1073740288)>
<AF[92]: clearing all remaining soft refs>
<GC(111): GC cycle started Thu Apr 07 08:17:24 2005
<GC(111): freed 196248 bytes, 4% free (44474048/1073740288), in
4106 ms>
<GC(111): mark: 3987 ms, sweep: 119 ms, compact: 0 ms>
<GC(111): refs: soft 13 (age >= 32), weak 1, final 0,
phantom 0>
<GC(112): GC cycle started Thu Apr 07 08:17:29 2005
<GC(112): freed 624 bytes, 4% free (44474672/1073740288), in 4851
ms>
<GC(112): mark: 4733 ms, sweep: 118 ms, compact: 0 ms>
<GC(112): refs: soft 0 (age >= 32), weak 0, final 0,
phantom 0>
<AF[92]: completed in 13153 ms>
<AF[93]: Allocation Failure. need 300016 bytes, 13911 ms since
last AF>
<AF[93]: managing allocation failure, action=2
(6488344/1073740288)>
<GC(113): GC cycle started Thu Apr 07 08:19:47 2005
<GC(113): freed 644892792 bytes, 60% free (651381136/1073740288),
in 20605 ms>
<GC(113): mark: 4735 ms, sweep: 140 ms, compact: 15730 ms>
<GC(113): refs: soft 0 (age >= 32), weak 28771, final 864,
phantom 0>
<GC(113): moved 7268044 objects, 270012456 bytes, reason=5, used
5272 more bytes>
<AF[93]: completed in 20619 ms>
From this data, we see that Allocation Failure (AF) 92 is able to free
around 25M of heap with no compaction attempted. Fourteen seconds later,
AF 93 performs a compaction and frees 644 MB of memory taking 20.6 seconds
to complete. The collecting of a relatively large amount of space during a
GC cycle that invokes a compaction is a clear symptom that dark matter is
involved. |
|
Solution |
Beginning with the 1.4.0 release of the IBM Java SDK, a
new feature was added to deal with the problem of long pauses caused by
the compaction phase. The 1.4 Java SDK will perform incremental
compactions by default, spreading the compaction process across the GC
cycles. The -Xpartialcompactgc parameter, which can be added
as a Generic JVM Argument for the 1.4 Java SDK, causes an incremental
compaction to occur on every GC cycle. Incremental compaction works by
dividing the heap into sections and running the compaction on one given
section at a time. The 1.4 release of the Java SDK is used beginning with
the WebSphere® Application Server V5.1 release. |
|
|