Tracing Administrative Server components on AIX using named (FIFO) pipes

Technote (FAQ)
Problem
Use these instructions to trace long running processes that generate large amounts of trace data and that could produce files that are too large for the filesystem to accommodate. This enables you to capture the trace data generated when a failure occurs.
Solution
If you already contacted support, continue to the component-specific MustGather information. Otherwise, click: MustGather: Read First for all WebSphere Application Server Products.

Use this information to trace systems management, connection manager, object request broker (ORB), and/or security components that run as subcomponents and are part of the administrative server. This technique is especially helpful when multiple days of execution are required to recreate a problem. In such cases, the enabling of tracing using normal files within the admin.config file is inappropriate because of the volume of trace data that can be generated.

With this tracing technique, the trace files that are generated are cyclically reused, so the portion of the trace associated with a particular situation can be captured. This allows the last N trace records to be saved. You will need the attachments at the bottom of this page.

This has been tested on AIX. It should work on other Unix platforms as well, but has not been tested on other platforms.

  1. Create a named (FIFO) pipe somewhere in the file system. For example:
    mkfifo /usr/WebSphere/AppServer/wsFIFO

    If you use ls -lgo to examine this, you will see something like:
    -> ls -lgo /usr/WebSphere/AppServer/wsFIFO
    prw-r--r-- 1 0 Jul 26 12:11 /usr/WebSphere/AppServer/wsFIFO

  2. Edit admin.config and set the administrative server trace output to point to the FIFO:
    com.ibm.ejs.sm.adminServer.traceOutput=/usr/WebSphere/AppServer/wsFIFO

  3. Set the administrative server trace string. This is done in admin.config file by adding the following line:
    com.ibm.ejs.sm.adminServer.traceString=com.ibm.ejs.*=all=enabled

    In the above, ejs tracing is an example; you can add other trace strings for connection manager, security or ORB that are pertinent to your situation. You can use multiple trace strings separated by a colon ( : ).

  4. Stop all WebSphere-related processes.

  5. Open a terminal session, change to the directory where you copied the
    fifoReader program (attached), then start the fifoReader program as in:

    fifoReader base_namepath_to_FIFOnbr_output_filesmax_lines

Where:

base_name is used to create the output file names. For example, if you specified "/tmp/ajay", then the output files are named /tmp/ajay01, /tmp/ajay02, and so forth.

path_to_FIFO tells where (in the file system) the FIFO is located. This must correspond to what you entered in step 1.

nbr_output_files is the maximum number of output files to be created. When this limit is reached, the first file is reused.

max_lines is the maximum number of lines to write in each file before moving on to the next file

For example:

fifoReader /usr/WebSphere/AppServer/logs/fifoTrace /usr/WebSphere/AppServer/wsFIFO 8 100000

This creates files like:
/usr/WebSphere/AppServer/logs/fifoTrace01
/usr/WebSphere/AppServer/logs/fifoTrace02
/usr/WebSphere/AppServer/logs/fifoTrace03
/usr/WebSphere/AppServer/logs/fifoTrace04
/usr/WebSphere/AppServer/logs/fifoTrace05
/usr/WebSphere/AppServer/logs/fifoTrace06
/usr/WebSphere/AppServer/logs/fifoTrace07
/usr/WebSphere/AppServer/logs/fifoTrace08

and each file has no more than 100,000 lines. Note that you can choose to either raise the number of files or the number of lines; however, it is strongly recommended that you use no fewer than 4 files of 100,000 records each.

Start the WebSphere Administrative Server, and watch for heap dumps. At this point, save the output files so that they are not overwritten.











Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, PD tools (for example Log Analyzer)
Operating system(s): AIX
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Edition Independent
Reference #: 1063362
IBM Group: Software Group
Modified date: 2004-03-02