Jinsight README
===============

0. Contents
===========

1. Overview

2. Files

3. Supported Platforms and JVMs

4. Modes of Operation

5. Client/Server and Headless

6. Client Installation
  6.1 JAVA2_HOME
  6.2 JINSIGHT_HOME
  6.3 JINSIGHT_HEAP

7. Server Installation and Invocation Overview
  7.1 Java 1.4.2 and 1.5 (JVMPI)
  7.2 Java 1.6 (JVMTI)
  7.3 Installation
    7.3.1 IBM z/OS 
    7.3.2 Linux on System z
    7.3.3 IBM AIX
    7.3.4 x86 Linux
    7.3.5 Microsoft Windows

8. Profiling
  8.1 Overview
  8.2 Command line Java (Generic Instructions)
    8.2.1 Headless/Local mode
    8.2.2 Client Server mode

9. A Typical WebSphere Profiling Session
  9.1 Arrange exclusive access to a WebSphere Application server
  9.2 Select the transactions that you wish to be profiled
  9.3 Understand how to drive the transactions manually
  9.4 Install the Jinsight server agents and the remote client
  9.5 Setup WebSphere using the Admin Console
  9.6 If WebSphere does not start
  9.7 When WebSphere restarts
  9.8 Run the selected transactions once to "warm-up" the JVM
  9.9 Start Profiling
  9.10 Re-run the selected transactions
  9.11 Stop Profiling
  9.12 Review status and Repeat
  9.14 Stop WebSphere and remove Jinsight

10. CPU Measurement
  10.1 Overview
  10.2 Invoking CPU Mode
  10.3 Config File Format
  10.4 Obtaining Class and Method Names
  10.5 Example
  10.6 Sample Output
  10.7 Identities

11. Jinsight Agent Options
  11.1 Profiling Mode: (default)
  11.2 CPU Measurement Mode (disables profiling)
  11.3 Generic Options
  11.4 Profiling Config File

12. Dealing with Large Trace Files
  12.1 TraceSplitter

13. Programatically Controling Profiling



1. Overview
===========


  Jinsight is a Java profiling tool, it is unique in its visual nature. It
allows the operator to consume enormous amounts of data easily and naturally,
and leads to a faster and deeper understanding of a modern Java program. It is
the pinnacle of application profiling.

  Jinsight comprises a Java GUI client for profile collection and analysis as
well as the platform specific .dll's (.so's) that plug-into the JVM to be
profiled.

  As with all other Java profiling tools it uses the JVMPI and now JVMTI
interfaces, these C APIs allow tool vendors to create "agents" that follow the
execution of Java code. 

  JVMPI was the traditional profiling interface to the JVM (arguably driven by
Jinsight's needs in 1996/7). It calls the registered agents on a significant
event such as method entry or exit, Object allocate/collect, thread start/stop.
The agent receives these callbacks and does with the data what it will, most
record to disk or transport over the network to a client.

  For numerous reasons it was decided to transition and combine the JVMPI
(Profiling), JVMDI (Debug) and other interfaces to a single JVMTI (Tool)
interface. The transition was planned to be from 1.4.2 to Java6, with Java5
being the fulcrum.

  So Java 1.4.2 supports (JVM)PI, Java 5 supports both PI and (JVM)TI and Java6
and beyond support only TI.

  The significance of this is that PI required the JVM to call the agent on
every event such as very frequent and basic operations like Object.hashcode()
entry/exit. 

  TI however requires the agent to manipulate the Java Bytecodes of classes
as they are loaded and insert java instructions to callback the agent
for whatever reason and at whatever point the agent requires. 

  Jinsight uses visualisation to draw patterns from the execution data
collected via JVMPI/TI. These patterns can both deliver the methods level detail
one gets from other Java profiling tools but in addition allows for rapid
understanding of large code bases in context of the use case/transaction,
specifically 3rd party code which is prevalent in modern Java.

  The significance of all of this is that regular profiling tools render
entry/exit events in text lists/ tree widgets, conversely Jinsight exploits
human visual pattern recognition, our most powerful tool. Where a textual
display can render 70 lines of text/method calls Jinsight can render 40 million.


2. Files
========

  JinsightClient.zip       The Jinsight client
  JinsightAIX.tar.gz       Server agents for AIX 
  JinsightWin.zip          Server agent for Microsoft Windows 
  JinsightxLinux.tar.bz2   Server agent for x86 Linux
  JinsightzLinux.tar.bz2   Server agents for Linux on System z
  JinsightzOS.tar.Z        Server agents for IBM z/OS


3. Supported Platforms and JVMs
===============================

  Jinsight currently the following JVMs:

    IBM 1.4.2 service release 9 and above (not 64 bit)

    IBM 1.5.0 service release 2 and above

    IBM 1.6.0 service release 3 and above

  On the following platforms:

    IBM z/OS (31/64 bit)

    Linux on System z (31/64bit)

    IBM AIX (32/64 bit)

    x86 Linux (32/64 bit)

    Microsoft Windows (32 bit)


4. Modes of Operation
=====================

  Jinsight has two modes of operation a) profiling, b) CPU measurement.
Profiling is the primary and default mode in which every Java method entry and
exit, Object creation and collection, thread start and end and other events are
collected on the "server" and visualised for analysis in the Jinsight client.

  The overhead incurred in profiling mode is high and therefore is not suitable
for production environments. The second mode, CPU measurement, allows a small
number of nominated methods derived by analysis of the profile to be measured at
full speed with little or no overhead. This accurate CPU information in concert
with the full profile enables highly efficient code analysis and tuning.


5. Client/Server and Headless
=============================

  Although referred to as the "server" in this README the the JVM being
profiled/measured could be on the same physical machine as the client.

  The CPU measurement mode does not involve the Jinsight client, the methods to
be measured are supplied to the agent in a config file on the server and data is
logged to stderr or a named log file also on the server.

  Two profiling methods are supported, client/server and headless. Client/server
involves enabling the server agent on the JVM to be profiled to listen on
network port, launching the client and connecting from client to server.
Profiling can then be started and stopped from the client and profile data is
sent to the client to be saved to disk and/or visualised.

  Headless profiling does not require the Jinsight client, the server agent is
told to save profile data to a local file and a small executable (jinctl*) is
used on the server to start and stop profiling. The file produced is then
downloaded to a client for visualisation. This techique can be useful if the
network bandwidth between the client and server is low.

  When in Client/Server mode the operator has the option of:

    a) Just saving profile data to disk for later analysys
    b) Saving to disk and loading into memory for immediate visualisation
    c) Not saving to disk but just loading for visualisation

  As the size of the profile data may well be unknown and seeing that the Client
will have finite memory it is recommended to use techique a) and just save to
disk for later analysis. If technique b) is used then the Client may run out of
memory putting into jeopardy the collection session. Technique c) is rare as
profiling data will be lost when the client is closed.

* jinctl is not available for Microsoft Windows, Client/server must be used


6. Client Installation
======================

  The Jinsight client is itself a Java application. Unzip JinsightClient.zip,
this will produce a directory "JinsightLive" on disk containing 3 files:

    jinsightlive.bat   A Microsoft Window batch file to start the client
    jinsightlive.sh    A Unix shell script to start the client
    jinsightLive.jar   A Java jar file containing the client code

  If installing on Windows edit jinsightlive.bat, or for Unix edit
jinsightlive.sh and change the following three environment variable in the file:


    6.1 JAVA2_HOME
    ===============

      This needs to be set to the location of a Java JVM (SDK/JRE). /bin/java
will be appended to this location to execute the Java JVM therefore the variable
needs to be set to a location that precedes /bin/java. For example: if the full
path to the java executable is C:\Program Files\IBM\Java\jre\bin\java then the
variable would be set to C:\Program Files\IBM\Java\jre

    On Microsoft windows it is advisable to wrap the variable in quotes in case
it has embedded spaces, i.e.

    "JAVA2_HOME=c:\Program Files\IBM\Java5\jre"


    6.2 JINSIGHT_HOME
    =================

      This variable need to be set to the full path to the directory that was
created when extracting the client zip file, e.g.

    "JINSIGHT_HOME=C:\tools\JinsightLive"  -  On Windows

    or

    JINSIGHT_HOME=/opt/JinsightLive  -  On Linux/Unix


    6.3 JINSIGHT_HEAP
    =================

      When visualising a profile the client can consume a large amount of
physical RAM. To limit the potential issues with completely consuming the Java
heap available the client will monitor it's own memory usage and stop loading
new data when it reaches a threshold of 90%. This variable is used to set the
maximum heap used buy the client JVM and is also passed to the client as a
command line option so it is aware of how much it has available.

      This option is limited to values in raw bytes, Kilobytes with the K
suffix Megabytes with the M suffix or Gigabytes with the G suffix, e.g.

      JINSIGHT_HEAP=1280M

      Clearly it should be set lower to a value lower that the physical RAM on
the client machine, one also has to consider that a 32bit client JVM may not be
able to address over 2GB even if physically available depending operating
system.

  Save and close the .bat/.sh and execute it, a small Java window should appear
in the top left of the screen. If not check the environment variables and the
error message returned when executing the script.

7. Server Installation and Invocation Overview
==============================================

  The Jinsight "server" agent .dll/.so uses either the JVM Profiling Interface
(JVMPI) or JVM Tool Interface (JVMTI) to communicate with the JVM and collect
profiling information. Profiling mode used JVMPI on Java 1.4.2 and 1.5 and JVMTI
on Java 1.6, CPU measurement mode uses JVMPI on 1.4.2 and JVMTI on 1.5 and 1.6.

  The agent is a shared library written in C that implements functions that are
called by the JVM on start-up and subsequently calls JVM functions when
profiling/measuring. 


  7.1 Java 1.4.2 and 1.5 (JVMPI)
  ==============================

    The JVM has to be told to load and call the agent initialisation functions,
this is done by adding command line options to the JVM. For Java 1.4.2 and 1.5
(JVMPI) the form is -XrunXXXXX:options where XXXXX is the name of the .dll/.so
file with the 'lib' prefix if it exists and the '.dll' or '.so' suffix removed,
e.g.

    the file libjinsight-zOS-31.so is loaded using the command line

      java -Xrunjinsight-zOS-31 ...otherJVMoptions

    or

    the file jinsight-win-32.dll is loaded using the command line

      java -Xrunjinsight-win-32 ...otherJVMoptions


    To specify Options that the agent can read a colon is appended to the
agent name followed by the option string, options are comma separated and cannot
contain spaces:

      java -Xrunjinsight-AIX-64:option1,option2=value ...otherJVMoptions


    JVMPI does not have the ability to specify the full path name of the agent
shared library on the Java command line, therefore the agent .dll/.so has to be
locatable by appending the directory it is in to the LIBPATH, LD_LIBRARY_PATH
or PATH environment variables depending on platform, i.e.

    Microsoft Windows:

      set PATH=%PATH%;C:\jinsight

    Linux:
  
      export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/jinsight

    AIX, z/OS:

      export LIBPATH=$LIBPATH:/opt/jinsight


  7.2 Java 1.6 (JVMTI)
  ====================

    Java 1.6 (JVMTI) uses a different command line format and additionally can
optionally specify the full path to the agent therefore removing the
requirement to set environment variables.

    The form is -agentlib:XXXXX=options or -agentpath:XXXXX=options.

    It is important to notice that JVMTI uses '=' to separate agent name and
options whereas JVMPI uses ':'

    agentlib is used if the location of the agent .dll/.so has been set using
the LIBPATH, LD_LIBRARY_PATH or PATH environment variables and therefore is
locatable by the dynamic linker or -agentpath if not.

    When using -agentlib the name of the agent is the file name adjusted as
above, removing lib prefix and .dll/.so suffix. When using -agentpath the fully
qualified name is used.

      export LIBPATH=$LIBPATH:/opt/jinsightLive

      java -agentlib:jinsight-AIX-32=options ...otherJVMoptions


      or

      java -agentpath:/opt/jinsight/libjinsight-AIX-32=options ...otherJVMOpts


  7.3 Installation
  ================

    7.3.1 IBM z/OS 
    =============

      Make a directory on the server to contain the jinsight agent binaries,
for example /opt/jinsight
    
      Upload as binary JinsightzOS.tar.Z and extract into the new directory
using:

        cd /opt/jinsight
        tar -xzXvf /path/to/JinsightzOS.tar.Z


      The X option on the tar command will attempt to restore the a and p
extended attribute flags so that the agents can be used to profile within an
authorised address space such as the WebSphere control region. This may not
be possible given your users authority, the flags can be re-applied at a later
date with proper authority using with:


        cd /opt/jinsight
        extattr +ap libjinsight-zOS-31.so
        extattr +ap libjinsight-zOS-64.so


      Confirm that the permissions on the files are correctly set to rwxr-xr-x
(755):

        cd /opt/jinsight
        ls -l

        -rwxr-xr-x   libjinsight-zOS-31.so
        -rwxr-xr-x   libjinsight-zOS-64.so
        -rwxr-xr-x   jinctl-31
        -rwxr-xr-x   jinctl-64

      Verify the installation by testing Jinsight as described in section 7.4 

    7.3.2 Linux on System z
    =======================

      Make a directory on the server to contain the jinsight agent binaries,
for example /opt/jinsight
    
      Upload as binary JinsightzLinux.tar.bz2 and extract into the new directory
using:

        cd /opt/jinsight
        tar -xjvf /path/to/JinsightzLinux.tar.bz2


      Confirm that the permissions on the files are correctly set to rwxr-xr-x
(755):

        cd /opt/jinsight
        ls -l

        -rwxr-xr-x   libjinsight-zLinux-31.so
        -rwxr-xr-x   libjinsight-zLinux-64.so
        -rwxr-xr-x   jinctl-31
        -rwxr-xr-x   jinctl-64

      Verify the installation by testing Jinsight as described in section 7.4 


    7.3.3 IBM AIX
    =============

      Make a directory on the server to contain the jinsight agent binaries,
for example /opt/jinsight
    
      Upload as binary JinsightAIX.tar.gz and extract into the new
directory
using:

	gunzip /path/to/JinsightAIX.tar.gz
	cd /opt/jinsight
	tar -xvf /path/to/JinsightAIX.tar


      Confirm that the permissions on the files are correctly set to rwxr-xr-x
(755):

	cd /opt/jinsight
	ls -l

	-rwxr-xr-x   libjinsight-AIX-32.so
	-rwxr-xr-x   libjinsight-AIX-64.so
	-rwxr-xr-x   jinctl-32
	-rwxr-xr-x   jinctl-64

      Verify the installation by testing Jinsight as described in section 7.4 

    7.3.4 x86 Linux
    ===============

      Make a directory on the server to contain the jinsight agent binaries,
for example /opt/jinsight
    
      Upload as binary JinsightxLinux.tar.bz2 and extract into the new directory
using:

        cd /opt/jinsight
        tar -xjvf /path/to/JinsightxLinux.tar.bz2


      Confirm that the permissions on the files are correctly set to rwxr-xr-x
(755):

        cd /opt/jinsight
        ls -l

        -rwxr-xr-x   libjinsight-xLinux-32.so
        -rwxr-xr-x   libjinsight-xLinux-64.so
        -rwxr-xr-x   jinctl-32
        -rwxr-xr-x   jinctl-64

      Verify the installation by testing Jinsight described in section 7.4 


    7.3.5 Microsoft Windows
    =======================

      Make a directory on the server to contain the jinsight agent binaries,
for example C:\jinsight

      Upload as binary JinsightWin.zip and extract into the new directory using
an unzip tool.

      Verify the installation by testing Jinsight as described in section 7.4 


  7.4 Verifying Installation
  ==========================

    To confirm that the binary agents have been installed successfully one can
execute java at the command line supplying the jinsight command line options.

    Firstly determine the version of Java by executing:


      java -version

    If no executable java is found you will have to specify a path fully or add
a JVM to your PATH.

    The version returned will be similar to the following:

      Java(TM) SE Runtime Environment (build pxi3260sr6-20090925_01(SR6))
      IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux x86-32
      jvmxi3260sr6-20090923_42924 (JIT enabled, AOT enabled)
      J9VM - 20090923_042924
      JIT  - r9_20090902_1330ifx1
      GC   - 20090817_AA)
      JCL  - 20090924_01

      or

      java version "1.5.0"
      Java(TM) 2 Runtime Environment, (build pwi32dev-20070201 (SR4))
      IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32
      j9vmwi3223-20070201 (JIT enabled)
      J9VM - 20070131_11312_lHdSMR
      JIT  - 20070109_1805ifx1_r8
      GC   - 200701_09)
      JCL  - 20070131

      or

      java version "1.4.2"
      Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2)
      Classic VM (build 1.4.2, J2RE 1.4.2 IBM z/OS Persistent Reusable VM build
      cm142-20070708 (SR9) (JIT enabled: jitc))


    The version info should indicate and IBM 1.4.2, 1.5 or 1.6 JVM at a Service
Release (SR?) on or above the levels described in Section 3. Also note the
"bittage" (31, 32 or 64) from the version info.

    Next set the path to the jinsight agents

    Microsoft Windows:

      set PATH=%PATH%;C:\jinsight

    Linux:
  
      export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/jinsight

    AIX, z/OS:

      export LIBPATH=$LIBPATH:/opt/jinsight
  

    Depending on the version of JVM next execute java again enabling the
jinsight agent and setting minimal local tracing options


    for Java 1.4.2 and 1.5: (choose the line for your platform and bittage)

  java -Xrunjinsight-zOS-31:localFileName=trace.trc,profileImmediately

  java -Xrunjinsight-zOS-64:localFileName=trace.trc,profileImmediately

  java -Xrunjinsight-zLinux-31:localFileName=trace.trc,profileImmediately

  java -Xrunjinsight-zLinux-64:localFileName=trace.trc,profileImmediately

  java -Xrunjinsight-AIX-32:localFileName=trace.trc,profileImmediately
  
  java -Xrunjinsight-AIX-64:localFileName=trace.trc,profileImmediately

  java -Xrunjinsight-xLinux-32:localFileName=trace.trc,profileImmediately

  java -Xrunjinsight-xLinux-64:localFileName=trace.trc,profileImmediately

  java -Xrunjinsight-win-32:localFileName=.\trace.trc,profileImmediately


    for Java 1.6: (choose the line for your platform and bittage)

  java -agentlib:jinsight-zOS-31=localFileName=trace.trc,profileImmediately

  java -agentlib:jinsight-zOS-64=localFileName=trace.trc,profileImmediately

  java -agentlib:jinsight-zLinux-31=localFileName=trace.trc,profileImmediately

  java -agentlib:jinsight-zLinux-64=localFileName=trace.trc,profileImmediately

  java -agentlib:jinsight-AIX-32=localFileName=trace.trc,profileImmediately

  java -agentlib:jinsight-AIX-64=localFileName=trace.trc,profileImmediately

  java -agentlib:jinsight-xLinux-32=localFileName=trace.trc,profileImmediately

  java -agentlib:jinsight-xLinux-64=localFileName=trace.trc,profileImmediately

  java -agentlib:jinsight-win-32=localFileName=.\trace.trc,profileImmediately


    This should produce something similar to the following:


      Combined Jinsight profiling agent.
          Copyright International Business Machines 1997-2009.
          Version ????.
          Using the JVM?I Interface.
          Started at ??? ??? ?? ??:??:?? ????

          Local trace file (trace.trc) opened

          .....

          (followed by standard java usage text)


    If it does not, check the command line and then confirm the installation
steps paying attention to permissions

    If successful a small trace file (trace.trc) will be produced in the current
directory


8. Profiling
=============

  8.1 Overview
  ============


  8.2 Command line Java (Generic Instructions)
  ============================================

  As described there are two ways to profile, headless and client server.
Headless mode is useful for small, short running command line java
applications, where the bandwidth between the client and server is low or the
profiling data produced will be very large (Gigabytes). Client server is a
more convenient method for long running applications such as application
servers when the client is on the same LAN (or machine) as the server.


    8.2.1 Headless/Local mode
    =========================

      The jinsight option localFileName switches the agent into local or
headless mode, in this mode there are two options for initiating profiling.
firstly the profileImmediately option starts data collection as soon as the JVM
is initialised, this is ideal for very short running Java programs, once
profile data collection has started it cannot be stopped.


  java -agentlib:jinsight-win-32=localFileName=my.trc,profileImmediately MyClass


      The other way to initiate collection in local mode is to use the jinctl
executable. By setting the option localID the agent will listen over shared
memory for jinctl to start or stop profiling.

  java -agentlib:jinsight-zOS-31=localFileName=my.trc,localID=1 MyClass


  then

  ./jinctl-31 start 1

  then 

  ./jinctl-31 stop 1
    

      With local profiling driven by jinctl it is possible to stop data
collection and start it again when required therefore building a trace file
containing slices of execution profile data.

      At the moment the trace file cannot be closed and a new one opened
within a running JVM. Therefore, considering the visualisation client memory
requirements it is recommended not to create a trace file above 2GB.

      When the JVM terminates the trace file will be properly closed, if the
JVM (or jinsight) fails during execution then the file will not be correctly
terminated however what ever already collected should still be valid and
readable by the client.

      Once the trace file is complete it can be transferred to a client and
loaded and visualised, it is possible to copy/transfer the trace file whilst it
is still open and the JVM is still running, it should sill be readable by the
visualiser.


    8.2.2 Client Server mode
    ========================

    Client/Server mode is the regular way of profiling with Jinsight. The agent
is plugged into the servers JVMPI/TI interface ans opens a server TCP/IP socket
for client connections. The client issues the commands to start and stop
collecting profile data and sends this data to the client for storage
to disk and analysis.

    To start a JVM in client/server mode you need to pass the port number that
the agent will listen on on the java command line:

      for Java 1.4.2 and 1.5

        -Xrunjinsight-AAA-BB:port=NNNNN

      for Java 1.6

        -agentlib:jinsight-AAA-BB=port=NNNNN

    Where AAA is the operating system identifier, BB is the platform bittage and
NNNNN is a unique port number for this JVM to listen on example:

      -Xrunjinsight-win-32:port=12345
   
      -agentlib:jinsight-AIX-64=port=6000

      *note: If there is a firewall between the server and client you will have
to open a port or use an already open slot.

      *note: On z/OS it is possible that the opening of server ports requires
special authorisation

      *note: When no options at all are supplied to the Jinsight agent, or
alternatively if the specified port is unavailable, Jinsight will allow the
operating system to select a port, this will be reported in the Jinsight log,
which defaults to stderr, or the log file specified with the logFile option.

      Launch the Jinsight Client installed in Section 6.

      Select File->Connect to a Java VM

      Enter the IP address or hostname of the server (127.0.0.1 or localhost) if
the profiled JVM is on the same machine as the client.

      Assuming you are using profiling technique a) described in Section 5:

      Enter the port specified in above or alternatively the port that
the operating system selected for Jinsight.

      Enter the name of a file on the client machine to save the profiling
data to

      Importantly select "Only save to file, don't process events"

      Click OK.

      You should now be connected to the server JVM, if not it may take up to 30
seconds for the attempt to fail.

      If the connection fails then confirm that Jinsight has not been forced to
let the operating system select a port and that there are no firewall or
security issues.

      Warm-up the JVM by allowing it to execute work/transactions at least once
then click "Start One Burst" on the Jinsight client.

      Allow the JVM to perform the work/transactions you wish to profile and
notice the number of "events read" increase in the Jinsight client window.

      Click "Stop This Burst"

      At this point the Jinsight client can be close, the JVM stopped and a new
Jinsight client opened to load and analyse the collected data

      Alternatively the client can remain connected and "Start One Burst"
clicked at a later time to collect more profile information. In this case the
profile file will contain two or more "bursts" of data.




9. A Typical WebSphere Profiling Session
=========================================


  9.1 Arrange exclusive access to a WebSphere Application server
  ==============================================================

    i) Due to the intrusiveness of profiling exclusive access is required to a
server that is non-critical, such as development/test for up to 2 hours 

    ii) It can take between 15 minutes and 2 hours to profile under normal
circumstances, issues that can extend this time generally centre around
restrictive authorisation and access to those who authorise


  9.2 Select the transactions that you wish to be profiled
  ========================================================

    i) Normally in an application the majority of transactions share a common
form, therefore even if there are 300 discrete transactions there may be as few
as 4 distinct forms and 95% of the paths are repeated.

    ii) The effort in profiling 300 transactions is significant and the
volume of data produced troublesome. A selection of 5 to 10 of the transactions
that are believed to be frequent or cause concern are the best to start with.


  9.3) Understand how to drive the transactions manually
  ======================================================

	i) there can be as many as 20 million "events" per transaction each
event when recorded is 48 bytes and the JVM slows by as much as 100x when
Jinsight is actively profiling. As we only nee to see one invocation of a
transaction to understand the path of all other invocation then this and the
data volume preclude "load" testing. Sometimes applications are only tested at
load with tools such as load-runner and manual driving is not a familiar task,
it needs to be an available option.


  9.4 Install the Jinsight server agents and the remote client
  ============================================================

  See Sections 6. and 7.3.


  9.5 Setup WebSphere using the Admin Console
  ===========================================

    i) Go to the admin console and select:


      for z/OS:
      Servers->Server Types->WebSphere Application Servers->Your Server->Servant

      for other platforms
      Servers->Server Types->WebSphere Application Servers->Your Server->


    ii) Half way down the right of the screen you will see a section called
"Java and Process Management", expand this and select "Process Definition"


    iii) On the right select "Environment Entries", add a new entry, or add to
an existing entry with the name:

      for z/OS, AIX:

        LIBPATH

      for Linux:

        LD_LIBRARY_PATH

      for Microsoft Windows

        PATH

      Set the value for this new entry to the directory containing the Jinsight
agent binaries. Click save, going back to "Process Definition"


    iv) On the right select "Java Virtual Machine"


    v) Scroll down the page and look for "Generic JVM Arguments"


    vi) This box may already contain some entries, the entries that will
clash with jinsight begin with "-Xrun", "-agentpath" or "-agentlib", remove
these entries*.

    *note: If Wiley Introscope or ITCAM are being used for this server
consider removing any classpath entries at the top of this JVM settings page

    *note: The agent for Microsoft Windows does not support local mode at the
moment

    vii) Add to "Generic JVM Arguments" the following:

    a) for Local (jinctl) profiling:

      for Java 1.4.2 (WebSphere 6.0)

        -Xrunjinsight-AAA-BB:localFileName=/path/to/trace.trc,localID=MM

      for Java 1.5 (WebSphere 6.1)

        -Xrunjinsight-AAA-BB:localFileName=/path/to/trace.trc,localID=MM

      for Java 1.6 (WebSphere 7)

        -agentlib:jinsight-AAA-BB=localFileName=/path/to/trace.trc,localID=MM

    Where AAA is the operating system identifier, BB is the platform bittage and
MM is a unique ID for this JVM, for example:

      -Xrunjinsight-zLinux-32:localFileName=/jinsight/trace.trc,localID=1
   
      -agentlib:jinsight-zOS-64=localFileName=/tmp/trace.trc,localID=10


    b) for Client/Server (network) profiling:

      for Java 1.4.2 (WebSphere 6.0)

        -Xrunjinsight-AAA-BB:port=NNNNN

      for Java 1.5 (WebSphere 6.1)

        -Xrunjinsight-AAA-BB:port=NNNNN

      for Java 1.6 (WebSphere 7)

        -agentlib:jinsight-AAA-BB=port=NNNNN

    Where AAA is the operating system identifier, BB is the platform bittage and
NNNNN is a unique port number for this JVM to listen on example:

      -Xrunjinsight-win-32:port=12345
   
      -agentlib:jinsight-AIX-64=port=6000

      *note: If there is a firewall between the server and client you will have
to open a port or use an already open slot.

      *note: On z/OS it is possible that the opening of server ports requires
special authorisation

      *note: When no options at all are supplied to the Jinsight agent, or
alternatively if the specified port is unavailable, Jinsight will allow the
operating system to select a port, this will be reported in the Jinsight log,
which defaults to stderr, or the log file specified with the logFile option.
   

    viii) Make sure that if this server is in a cluster that all other
cluster members are stopped. Additionally on z/OS go back to "Java and Process
Management", expand it again and select "Server Instance". Deselect "Multiple
Server Instances" and to make sure to drop the minimum and maximum both to 1


    ix) Save all changes and restart WebSphere


  9.6 If WebSphere does not start
  ===============================

      If WebSphere does not start it is probably due to not being able to find
or load the .dll/.so agent binary. 

      If this is a stand-alone WebSphere server instance you may have to locate
and edit the server.xml under the WebSphere directory to correct any mistakes.

      If this is WebSphere ND then go back to the admin console and check all
settings. Additionally check and verify installation by re-visiting Sections
7.3 and 7.4.


  9.7 When WebSphere restarts
  ===========================

    If in local mode Jinsight will write to the trace file a 45 byte header,
confirm that you see this and locate and prepare jinctl for execution.

    If in client/server mode launch the Jinsight client and select:

      File->Connect to a Java VM

      Enter the IP address or hostname of the server (127.0.0.1 or localhost) if
the profiled JVM is on the same machine as the client.

      Assuming you are using profiling technique c) described in Section 5:

      Enter the port specified in section 9.5 or alternatively the port that
the operating system selected for Jinsight.

      Enter the name of a file on the client machine to save the profiling
data to

      Importantly select "Only save to file, don't process events"

    Click OK.

    You should now be connected to the server JVM, if not it may take up to 30
seconds for the attempt to fail.

    If the connection fails then confirm that Jinsight has not been forced to
let the operating system select a port and that there are no firewall or
security issues.


  9.8 Run the selected transactions once to "warm-up" the JVM
  ===========================================================

    As the application code has not yet been run in this fresh JVM if we
were to start profiling and then run the transactions for the first time the
profile would be full of Class loading and application initialisation that
would not be part of production transactions. Therefore it is important to run
the transactions once to "warm-up" the JVM.

    Ideally the transactions should be driven manually from a browser to
increase the control one has over the volume of profile data collected in a
single file.

    However you can if you wish run the transactions from a load test script but
there is no need for load with Jinsight, just one script iteration from a
single user/thread is enough.


  9.9 Start Profiling
  ===================

    If in local mode:

      Go to the directory that contains the jinctl executable and execute in the
following way to start profiling:

        ./jinctl-BB start MM

      where BB is the bittage of the operating system and MM is the unique
localID set in the Generic JVM Arguments in Section 9.5.

      for example

        ./jinctl-31 start 10


    If in client/server mode:

      Go to the connected jinsight client window and click "Start One Burst"


  9.10 Re-run the selected transactions
  =====================================

    As before load is NOT needed. Re-run the same transaction as in Section
9.8 either manually from a browser of a single iteration with a single
user/thread from the load test script.

    The execution speed of the JVM whilst profiling will be reduced
significantly, this may affect browser time-outs, increase time-out values if
required.

    In local mode you should see the specified trace file increasing in size,
if in client/server mode you will see the number of "events read" in the
Jinsight client window increase.


  9.11 Stop Profiling
  ===================

  When the transactions have completed their execution you stop profiling.

    If in local mode:

      Go to the directory that contains the jinctl executable and execute in the
following way to start profiling:

        ./jinctl-BB stop MM

      where BB is the bittage of the operating system and MM is the unique
localID set in the Generic JVM Arguments in Section 9.5.

      for example

        ./jinctl-64 stop 10


    If in client/server mode:

      Go to the connected jinsight client window and click "Stop This Burst"


  9.12 Review status and Repeat
  =============================

    The trace file, either on the client or server, should now contain a
substantial amount of data ranging from a few MB to over 10GB.

    As the client has limits on the size of trace file it can visualise you may
consider opening a new trace file before continuing with more transactions:

      If in local mode:

        Stop WebSphere copy the trace file to a different location. Restart
WebSphere, this will open a new trace file, return to section 9.8.


      If in Client/Server mode:

        Close the Jinsight client, and restart it. Re-connect as in Section
9.7 but select a different trace file name, return to Section 9.8.


  9.14 Stop WebSphere and remove Jinsight
  =======================================

    When the profiling session is over WebSphere should be stopped. If this is
a stand-alone server you may with to reverse the actions made in the Admin
Console in Section 9.5 before stopping. If in local mode make sure that the
local trace file is copied off to another location/name as if WebSphere were to
restart it would over-write the data. 


10. CPU Measurement
===================


  10.1 Overview
  ============

  Jinsight profiling gives a unique and detailed view of the execution of Java
code and can itself lead to the insight required to improve performance.
However the overhead introduced my any Java profiler can skew the reported
times in the profile enough to render them unhelpful. 

  Jinsight has a second mode of operation in which, guided by the profile,
methods of interest can be declared in a config file and this file used to
instruct the jinsight agent to instrument and measure just those methods for
CPU and elapsed times.

  This means that with only a handful of methods instrumented that the
application can run at near production speed, the true costs of the methods
measured and "overlaid" on the profile to inform and direct tuning efforts.

  If a method looks inefficient when a profile is visualised it can be added
the a config file and the application run again with jinsight in cpuMode. The
costs of the method are printed out as XML with little or no overhead. 


  10.2 Invoking CPU Mode
  ======================

    To switch the Jinsight agent into CPU mode pass the option cpuMode on the
java command line and supply a config file with the configFile option.
Optionally you can log the XML output to a file using the logFile option.

    for Java 1.4.2

      java -Xrunjinsight-AAA-BB=cpuMode,configFile=my.cfg,logFile=cpu.xml

    for Java 1.5 and Java 1.6

      java -agentlib:jinsight-AAA-BB=cpuMode,configFile=my.cfg,logFile=cpu.xml


  10.3 Config File Format
  =======================

    Each line in the CPU config file is made up of 5 components in the following
form:

      role class method signature identity

        role: The type measurement, transaction, fragment or identity
  
        class: The name of the class to instrument

        method: The name of the method to instrument

        signature: The signature of the method to instrument

        identity: The unique name for this declaration

    The elements of the line are space/tab separated. Pay attention to the line
terminator, if created or edited on Microsoft Windows then the lines will be
terminated with CR and LF, the agent is expecting a Unix style line terminator
of just LF, many Windows editors can support this convention.

    Method signatures are in the JVM format described in Section 4.3.3 of the
VM Spec "Method Descriptors"

  
  10.4 Obtaining Class and Method Names
  =====================================

    The signatures (or descriptors) used in the config file are in JVM format
and can be non-intuitive. To ease the creation of the config file the
application can be run with the Jinsight agent passing the dumpClasses and
logFile options. This will log every class and method as they load and the
method signature in the correct format for the config file.

    for Java 1.4.2

      java -Xrunjinsight-AAA-BB=dumpClasses,logFile=classes.dump

    for Java 1.5 and Java 1.6

      java -agentlib:jinsight-AAA-BB=dumpClasses,logFile=classes.dump


    Make sure that you actually run the application code that you wish to
measure so that it's classes are loaded and dumped.

    The config file can then be created and the JVM restarted with the cpuMode
option as described in Section 10.2

    An alternative method for obtaining correct signatures is to open a profile
of the application in the Jinsight client, locate a method and over over it
with the mouse, hit Ctrl-C and this will copy the method details into the
clipboard which can them be pasted into an editor and the config file created.

  10.5 Example
  ============

    #Example config file
    #
    transaction HelloWorld main ([Ljava/lang/String;)V helloWorldMain
    fragment java/io/PrintStream println (Ljava/lang/String;)V printHello


    HelloWorld.java:

      public class HelloWorld {

        public static void main(String[] args) {

          System.out.println("Hello World");

        }

      }

    Invocation:

      for Java 1.4.2

        java -Xrunjinsight-AAA-BB=cpuMode,configFile=example.cfg HelloWorld

      for Java 1.5 and Java 1.6

        java -agentlib:jinsight-AAA-BB=cpuMode,configFile=example.cfg HelloWorld


  10.6 Sample Output
  ==================

  *note: The first line is wrapped in this README file.

  <transaction identity="helloWorldMain" cpuMicros="603" elapsedMicros="5390"
thread="main" seq="1" startTime"Thu Nov 26 12:16:46 2009">
      <fragment identity="printHello" cpuMicros="219" elapsedMicros"1012"
seq="2", startTime="Thu Nov 26 12:16:46 2009">
      </fragment>
   </transaction>


    Explanation:

      The first XML Element is <transaction> which measures the cost of the
main methods and any calls that it makes. It has 6 attributes; identity, which
is the name given in the config file, cpuMicros the CPU cost of this method and
everything it called, elapsedMicros which is the wallclock elapsed time of the
transaction, thread is the name of the thread is was called on, seq is a unique
incrementing ID number for this element, and startTime is the time the call was
started.

      Within the transaction a method declared in the config file as being a
fragment was called, it's costs were measured independently and a <fragment>
element written as a child of the <transaction> element. The <fragment> element
has the same attributes as transaction. It's costs are a component of the owning
transaction costs. So in this case the transaction took 603us of CPU of which
219us were the printHello fragment.

      It is possible to have fragment elements has children of other fragment
elements in a deep hierarchy depending of the config file and path of the code.


  10.7 Identities
  ===============

    In an Application Server such as WebSphere one may declare as a transaction
method in the config file a container method  that is called early in the
processing of the request and therefore report not only application code CPU
but also Application Server CPU. However, this would mean that all transactions
would have the generic name given to this transaction owning method in the
config file even though they may be performing different tasks.

    An "identity" declaration in the config file defines a method that when
executed within a transaction renames the transaction instance to something
that more accurately represents the function that this particular call is
performing.


    For example:

      #Application Server Config
      #
      transaction app/server/Container processRequest ()V HTTPRequest
      identity my/app/Business balance ()I BalanceInquiry
      identity my/app/Business deposit ()I DepositFunds
      fragment my/app/DB getData ()Ljava/lang/String; GetFromDB


    Example output: (details removed)

      <transaction identity="HTTPRequest" cpuMicros="10321">
      </transaction>
      <transaction identity="BalanceInquiry" cpuMicros="23540">
        <fragment identity="GetFromDB" cpuMicros="1022">
        </fragment>
      </transaction>
      <transaction identity="DepositFunds" cpuMicros="44023">
        <fragment identity="GetFromDB" cpuMicros="1119">
        </fragment>
      </transaction>
      <transaction identity="BalanceInquiry" cpuMicros="24332">
        <fragment identity="GetFromDB" cpuMicros="939">
        </fragment>
      </transaction>
      <transaction identity="HTTPRequest" cpuMicros="9985">
      </transaction>


11. Jinsight Agent Options
==========================

  The Jinsight agent takes various options on the Java command line, some are
single options and some are name=value pairs, all are comma separated, spaces
are not allowed.   


  11.1 Profiling Mode: (default)
  ==============================


    port=n

      Defines the port for jinsight to listen on, local mode is disabled.


    localFilename=/path/name

      Defines a local trace file, network is disabled, local control is enabled.


    localID=n

      Sets the local Id for this JVM if multiple JVM on the same machine are in
local control mode, mutually exclusive with profileImmediately.


    profileImmediately

      When tracing to a local file, profile straight away, do not wait to be
told to start, mutually exclusive with localID.


    configFile=/path/name

      Location of a file containing classes and method to skip when applying
bytecode instrumentation when using JVMTI (Java 1.6). See section 10.4


    maxBuffers=n

        Sets the maximum outstanding trace buffers to n, default 16. Jinsight
writes profiling events to native memory buffers before writing to disk or the
network. If the write speed is low then the defined buffers will fill as the
Java program executes until no buffer space is left at which point is will
suspend execution of the profiled application until buffers are freed by
writing. The size, number buffers and usage of the native buffers can put
pressure of limited native memory this option in conjunction with the
bufferLength option allows the native memory usage to be tuned. The default
number of buffers, 16, multiplied by their default size of 4M means that without
setting these options the agent could consume 64MB of native memory under
certain circumstances.

        In addition to these write buffers Jinsight uses a local buffer per
thread currently fixed to 128K. These thread buffers are only initialised if
the thread is/was active during profiling, thread buffers are not deallocated
unless the thread ends. Therefore during a long profiling session in which
every thread has been active at one point during profile data collection then
jinsight will consume an additional 128K * number of threads. With some
applications having over 1000 threads this can also put pressure on native
memory.

    bufferLength=n [M|m][K|k]

        Set the length of each trace buffer to n bytes, kilobytes megabytes,
default 4M. See above for explanation.


  11.2 CPU Measurement Mode (disables profiling)
  ==============================================


    cpuMode

      Enables CPU measurement mode, see Section 9.


    dumpClasses

      Dump the methods and signatures of all classes loaded to help create the
config File, no CPU information is collection even if configFile is specified.
This is a one off action to generate a file containing the signatures.


    configFile=/path/name

      Location of a file containing methods to measure. See Section 9.


    logBufferLength=n [M|m][K|k]

      Increase if the output XML is truncated, default 64K. For performance the
XML element containing the information for a transaction and it's fragments is
first rendered to a buffer before being written to the log. This buffer is
statically allocated and defaults to 64K. If the transaction has hundreds of
fragments then the XML logged may be truncated as the buffer was not large
enough to render all of the fragments to. In this case as much of the XML
element is logged followed by the string TRUNCATED... Increasing the size of
this buffer allows all of the fragment data for the transaction to be logged.


  11.3 Generic Options
  ====================


    logFile=/path/name

      Write all jinsight logging to the named file.


    appendLogFile

      Do not overwrite log file, but rather append to it.

    help

      Prints an overview of Jinsight options and usage and then exits.


  11.4 Profiling Config File
  ==========================


    When using JVMTI (Java 1.6) The agent must manipulate the bytecodes of
methods when loaded, this can very occasionally fail on certain Classes and
methods. To overcome this a config file and be written and supplied to the
agent instructing it no to instrument the failing classes and methods.

    The format of each line in the file is:

      instruction class method signature

    Where instruction can be "skipclass" or "skipmethod", skipclass only
requires the class name after it, skipmethod requires class method and
signature.

    Example:

      #Profiling Config File
      #
      skipclass org/nasty/NastyClass
      skipmethod org/evil/Evil badMethod (II)J

12. Dealing with Large Trace Files
==================================


  12.1 TraceSplitter
  ==================

    It is possible to create profile trace files that are too large to be
visualised given finite client memory. I these cases the trace file can be
split into multiple parts that can be loaded by the client individually.

    Within the JinsightLive jar file there is a utility to split trace files
whith the class name jinsight.util.TraceSplitter.

    To invoke TraceSplitter use the following command line

      java -cp JinsightLive.jar jinsight.util.TraceSplitter inputFileName numberOfParts

    Where input file name is the name of the file to be split and numberOfParts
is the number of files you wish to split into.

    e.g.

      java -cp JinsightLive.jar jinsight.util.TraceSplitter trace.trc 10

    The defines number of partial traces files are written with the name
inputFileName.N where N is a number from 1 to numberOfParts.

    e.g.

      java -cp JinsightLive.jar jinsight.util.TraceSplitter mytrace.trc 5

    will create the following files

      mytrace.trc.1
      mytrace.trc.2
      mytrace.trc.3
      mytrace.trc.4
      mytrace.trc.5


13. Programatically Controling Profiling
========================================

  Often a known problematic method may need to be profiled by itself without the
additional data from the rest of the application. It is possible to
programatically turn tracing on and off with calls to the static
native methods JinsightTraceControl.startProfiling() and Jinsight
TraceControl.stopProfiling().

  These calls only work in local mode (i.e. locaFileName= is set).

  To compile against the JinsightTraceControl class it be found in the
Clients jinsightLive.jar or alternatively one could create and compile the
minimal class from the source below.


    public class JinsightTraceControl {
      public static native void startProfiling();
      public static native void stopProfiling();
    }

  These is no need to deploy this class or include jinsightLive.jar on the
applicatyions classpath as the binary jinsight agent automatically dfines the
class when it starts-up.


  Calling the functions would look like this:


      public void myMethod() {

	.... some code

	JinsightTraceControl.startProfiling();

	.... the code you wish to profileImmediately

	JinsightTraceControl.stopProfiling();

	... more code

      }