Enterprise applications written in the Java language involve complex object relationships and utilize large numbers of objects. Although, the Java language automatically manages memory associated with object life cycles, understanding the application usage patterns for objects is important. In particular, verify the following:
Understanding the effect of garbage collection is necessary to apply these management techniques.
The garbage collection bottleneck
Examining Java garbage collection gives insight to how the application is utilizing memory. Garbage collection is a Java strength. By taking the burden of memory management away from the application writer, Java applications are more robust than applications written in languages that do not provide garbage collection. This robustness applies as long as the application is not abusing objects. Garbage collection normally consumes from 2% to 5% of total execution time of a properly functioning application. If not managed, garbage collection is one of the biggest bottlenecks for an application. The Java virtual machine (JVM) uses a parallel garbage collector to fully exploit an SMP during most garbage collection cycles.
The garbage collection gauge
You can use garbage collection to evaluate application performance health. By monitoring garbage collection during the execution of a fixed workload, you gain insight as to whether the application is over-utilizing objects. Garbage collection can even detect the presence of memory leaks.
You can monitor garbage collection statistics using the verbose:gc JVM configuration setting. The verbose:gc format is not standardized between different JVMs or release levels.
For this type of investigation, set the minimum and maximum heap sizes to the same value. Choose a representative, repetitive workload that matches production usage as closely as possible, user errors included.
To ensure meaningful statistics, run the fixed workload until the application state is steady.
Detecting over-utilization of objects
You can check if the application is overusing objects, by observing the counters for the JVM runtime. To do this, set the -Xrunhprof command line option, as well as the JVM module maximum level in order to enable the Java virtual machine profiler interface (JVMPI) counters. The best result for the average time between garbage collections is at least 10 times the average duration of a single garbage collection. If you do not achieve this number, the application is spending more than 10% of its time in garbage collection. For more information about the -Xrunhprof option, see IBM Developer Kit and Runtime Environment, Java 2 Technology Edition, Diagnostics Guide .
If the information indicates a garbage collection bottleneck, there are two ways to clear the bottleneck. The most cost-effective way to optimize the application may be to implement object caches and pools. If your application is generating a huge amount of garbage every second, check for common problems like overuse of String appends (String X = oldString+"abcde"). Rather than appending to a String, you can append to a StringBuffer, significantly reducing the amount of garbage you generate. In some cases, it may also help to implement object caches or pools. However, take care to avoid memory leaks and to tune your caches/pools for optimal performance. Use a Java profiler to determine which objects to target.
If you can not optimize the application, increasing the size of your Java heap may help. By increasing the heap size, you can reduce the number of garbage collection cycles and thereby reduce the total cost of GC. If you seem to need a larger heap than you can create in a single address space, you can also spread your work across more servant regions. However, before you increase your Java heap size, make sure that your real storage is not already constrained. If you already have significant pressure on real storage, increasing the size of your Java heap may do more harm than good.
Other tools you may want to consider include the Tivoli Performance Viewer (TPV), which can be used to monitor WebSphere Application Server for z/OS transactions using the PMI (Performance Monitoring Infrastructure), and the WSAD Profiler, which can help you determine why an application is not meeting response time, CPU utilization, or storage usage requirements. For more information, see the TPV (TD101242 and TD101645) and the WSAD Profiler (TD101199) white papers at IBM Support - Techdocs Library .
Detecting memory leaks
Memory leaks in the Java language are a dangerous contributor to garbage collection bottlenecks. Memory leaks are more damaging than memory overuse, because a memory leak ultimately leads to system instability. Over time, garbage collection occurs more frequently until the heap is exhausted and the Java code fails with a fatal Out of Memory exception. Memory leaks occur when an unused object has references that are never freed. Memory leaks most commonly occur in collection classes, such as Hashtable because the table always has a reference to the object, even after real references are deleted.
High workload can sometimes cause applications to crash immediately after deployment in the production environment. This is especially true for leaking applications where the high workload accelerates the magnification of the leakage and a memory allocation failure occurs. However, in other cases, an application will appear to work fine, but every so often the servant region will terminate with an Out of Memory exception. The easiest way to detect potential memory leaks in a running application is to capture verbose:gc data or use a performance monitor to determine how much heap storage is available after each GC. If the free heap continuously declines over time, this may indicate a memory leak.
Memory Leak Testing
The goal of memory leak testing is to magnify numbers. Memory leaks are measured in terms of the amount of bytes or kilobytes that cannot be garbage collected. The delicate task is to differentiate these amounts between expected sizes of useful and unusable memory. This task is achieved more easily if the numbers are magnified, resulting in larger gaps and easier identification of inconsistencies. The following list contains important conclusions about memory leaks:
Memory leak problems can manifest only after a period of time, therefore, memory leaks are found easily during long-running tests. Short running tests can lead to false alarms. It is sometimes difficult to know when a memory leak is occurring in the Java language, especially when memory usage has seemingly increased either abruptly or monotonically in a given period of time. The reason it is hard to detect a memory leak is that these kinds of increases can be valid or might be the intention of the developer. You can learn how to differentiate the delayed use of objects from completely unused objects by running applications for a longer period of time. Long-running application testing gives you higher confidence for whether the delayed use of objects is actually occurring.
In many cases, memory leak problems occur by successive repetitions of the same test case. The goal of memory leak testing is to establish a big gap between unusable memory and used memory in terms of their relative sizes. By repeating the same scenario over and over again, the gap is multiplied in a very progressive way. This testing helps if the number of leaks caused by the execution of a test case is so minimal that it is hardly noticeable in one run.
You can use repetitive tests at the system level or module level. The advantage with modular testing is better control. When a module is designed to keep the private module without creating external side effects such as memory usage, testing for memory leaks is easier. First, the memory usage before running the module is recorded. Then, a fixed set of test cases are run repeatedly. At the end of the test run, the current memory usage is recorded and checked for significant changes. Remember, garbage collection must be suggested when recording the actual memory usage by inserting System.gc() in the module where you want garbage collection to occur, or using a profiling tool, to force the event to occur.
Some memory leak problems can occur only when there are several threads running in the application. Unfortunately, synchronization points are very susceptible to memory leaks because of the added complication in the program logic. Careless programming can lead to kept or unreleased references. The incident of memory leaks is often facilitated or accelerated by increased concurrency in the system. The most common way to increase concurrency is to increase the number of clients in the test driver.
Consider the following points when choosing which test cases to use for memory leak testing:
Java heap parameters
The Java heap parameters also influence the behavior of garbage collection. Increasing the heap size supports more object creation. Because a large heap takes longer to fill, the application runs longer before a garbage collection occurs. However, with a larger heap garbage collection takes longer. Initial and maximum Java heap sizes can be specified using the -Xms and -Xmx options. The general recommendation is that these values should be the same.
Note that Java Heap information is contained in SMF records and can be viewed dynamically using the console command DISPLAY,JVMHEAP.
Starting with the default value for initial and maximum heap size, capture verbose:gc output (or equivalent performance monitor data) running your standard workload. Examine the verbose:gc output for garbage collections which occurred after the workload reaches a steady state. Compare the following statistics:
If the heap free space settles at 85% or more and the percentage of time in GC processing is low, consider decreasing the initial and maximum heap size values because the application server and the application are under-utilizing the memory allocated for heap. If system real storage is constrained, overall performance may be improved by reducing the Java heap size, even if the percentage of time in GC is higher than desired.
For current information available from IBM Support on known problems and their resolution, see the IBM Support page.
IBM Support has documents that can save you time gathering information needed to resolve this problem. Before opening a PMR, see the IBM Support page.