Correlating TPROF output to a specific thread in the JAVACORE using DBXTrace -a output
 Technote (troubleshooting)
 
Problem(Abstract)
How to match a thread ID (TID) in TPROF to the pthread_t in "DBXTrace -a" output and to native ID in a javacore file.
 
Resolving the problem
The following examples of documentation are gathered as a result of the following technotes:
The following commands gather the output used in this example:
  • TPROF:

    tprof -k -s -e -x sleep 60

  • DBXTrace: (Not needed for 5.0 or higher Java™ Virtual Machine)

    ./dbxtrace_aix.sh -a [AppServer PID] > dbxtrace.out

  • JAVACORE:

    kill -3 [Appserver PID]

The following is sample output from TPROF:

Process      PID      TID    Total   Kernel     User   Shared
=======    =======    ===    =====   ======     =====  ======
 java       14254     83603   10675   10675         0       0  
 java       14254     80450    9677     887         0    6730

1.You can see that a large amount of CPU is used by the Java™ process. The amount under the Total column is the total number of ticks that process used. These should be ordered from highest to lowest.

2. For Java Virtual Machine V1.4 and V1.3, search the dbxtrace output for the TID. In this case, the value is "83603". The following is the result of a search:

thread  state-k  wchan state-u  k-tid  mode  held  scope  function
 $t9     run            running  83603   u     no   sys  lkmonitor
 
      general:
         pthread addr = 0x30ccf4f8   size    = 0x18c
         vp addr      = 0x30ba4104   size    = 0x284
         thread errno = 2
         start pc     = 0x300555a0
         joinable     = no
         pthread_t    = 809
      scheduler:
         kernel       =
         user         = 1 (other)
      event :
         event        = 0x0
         cancel       = enabled, deferred, not pending
      stack storage:
         base         = 0x30c8f000    size   = 0x40000
         limit        = 0x30ccf4f8
         sp           = 0x30ccdb18


In this case, the thread is "$t9". Search for the corresponding pthread_t value to find the corresponding thread in the JAVACORE file. In this example, the corresponding pthread_t value is "809". The pthread_t value will correlate to the native ID. It should be proceeded with a "0x":

"Pooled ORB request dispatch WorkerThread" (TID:0x800334a8,
    sys_thread_t:0x30b816f8, state:R, native ID:0x809) prio=5
at java.lang.reflect.Method.copy(Method.java:105)
at java.lang.reflect.Constructor.getParamTypes(Constructor.java:77)


3.For Java Virtual Machine V5.0 or higher, convert decimal TID (83603) to hex TID(14693) in TPROF and search javacore for the hex TID (14693). The hex TID (14693) in TPROF will correlate to the native ID in javacore.


"Pooled ORB request dispatch WorkerThread" (TID:0x800334a8,
    sys_thread_t:0x30b816f8, state:R, native ID:0x00014693) prio=5
at java.lang.reflect.Method.copy(Method.java:105)
at java.lang.reflect.Constructor.getParamTypes(Constructor.java:77)


5.Review the stack trace and identify threads that have high CPU usage.
 
Related information
MustGather: 100% CPU Usage on AIX Platforms
MustGather: No response (hang) or performance degradati
 
 
Cross Reference information
Segment Product Component Platform Version Edition
Application Servers Runtimes for Java Technology Java SDK
 
 


Document Information


Product categories: Software > Application Servers > Distributed Application & Web Servers > WebSphere Application Server > Java SDK
Operating system(s): AIX
Software version: 6.1
Software edition:
Reference #: 1138359
IBM Group: Software Group
Modified date: Sep 28, 2007