How to collect performance data on Linux

Collect the following information when high CPU consumption is with IBM Java process:
  1. Enable garbage collection trace to see whether Java garbage collection is thrashing if possible. If you want to enable Java garbage collection trace on IBM WebSphere Application Server, please refer to the following document: Enabling verbose garbage collection (verbosegc) in WebSphere Application Server

  2. Run the following command:

    top -d delaytime -c -b > top.log

    Where delaytime is the number of seconds to delay. This must be 60 seconds or greater, depending on how soon the failure is expected.

  3. Create a script file, vmstat.sh with the following content:
    #vmstat.sh
    #output file name
    VMSTAT_LOG=$1
    LIMIT=288
    #sleep for 5 miniutes
    SLEEP_TIME=300
    while true
    do
     i=0
     echo >$VMSTAT_LOG
     while [ $i -le "$LIMIT" ];
     do
      date >> $VMSTAT_LOG;
      vmstat 5 12 >> $VMSTAT_LOG;
      i=`expr $i + 1`;
      sleep $SLEEP_TIME;
     done
    done
    
  4. Create a script, ps.sh with the following content:
    #ps.sh
    #output file name
    PS_LOG=$1
    LIMIT=288
    #sleep for 5 miniutes
    SLEEP_TIME=300
    while true
    do
     i=0
     echo >$PS_LOG
     while [ $i -le "$LIMIT" ];
     do
      date >> $PS_LOG;
      ps -eLf >> $PS_LOG;
      i=`expr $i + 1`;
      sleep $SLEEP_TIME;
     done
    done
    
  5. Run the scripts:

    ./ps.sh ps_eLf.log
    ./vmstat.sh vmstat.log

    Notes: . The scripts ps.sh and vmstat.sh, as provided, roll over every 24 hours. . You might need to modify the scripts to meet your needs. . The preceding scripts will run forever. After the error condition is reached, you will have to terminate them.

  6. When high CPU consumption occurs, collect the following logs:

    netstat -an > netstat1.out

  7. If the Web server is remote, run the following on the Web server system:

    netstat -an > netstatwebserver1.out

  8. Run the following:

    kill -3 [PID_of_problem_JVM]

    The kill -3 commands create javacore*.txt files

    Note: If you are not able to determine which JVM process is experiencing the high CPU usage then you should issue the kill -3 PID for each of the JVM processes.

  9. Wait two minutes.

  10. Run the following:

    kill -3 [PID_of_problem_JVM]

  11. Wait two minutes.

  12. Run the following:

    kill -3 [PID_of_problem_JVM]

  13. Wait two minutes.

  14. Run the following:

    netstat -an > netstat2.out


  15. If the Web server is remote, run the following on the Web server system:

    netstat -an > netstatwebserver2.out


  16. If you are unable to generate javacore files, then perform the following:

    kill -11 [PID_of_problem_JVM]

    WARNING: kill -11 will terminate the JVM process, produce a core file, and possibly a javacore.

  17. Review all output files and collect the following files for IBM Performance Analysis Tool for Java for Linux

If you want to analyze the Java thread dumps, download the IBM Thread and Monitor Dump Analyzer for Java (TMDA). TMDA is one of top alphaWorks technologies that can analyze thread dumps from Java virtual machine. It is useful for identifying deadlocks, contention, bottlenecks, and to summarize the state of threads within Java virtual machine.

If garbage collection activity seems to be causing performance degradation or high processor time consumption, Verbose GC logging can be enabled. Enabling the generation of Verbose GC logging is done using the command line option: -verbose:gc. This causes the Verbose GC logging to be written to stderr or stdout.

If you want to analyze the Java Verbose GC log, download the IBM Pattern Modeling and Analysis Tool for Java Garbage Collector (PMAT). PMAT is one of top alphaWorks technologies that can parses verbose GC trace, analyzes Java heap usage, and recommends key configurations based on pattern modeling of Java heap usage.