|
Problem(Abstract) |
Collecting data for when your WebSphere® Application
Server is not responding (hangs) on AIX® and the Kill -3 command does not
generate a javacore file. Gathering this information before calling IBM®
support will help familiarize you with the troubleshooting process and
save you time. |
|
|
|
Resolving the
problem |
If you have already contacted support, continue on to the
Hang/Performance degradation MustGather information. Otherwise, click: MustGather:
Read first for all WebSphere Application Server products.
No javacore generated specific MustGather information
The following lists what is needed to troubleshoot a hung Java™ virtual
machine (JVM) on an AIX operating system when the kill -3 command
does not generate a javacore.txt file.
Issuing a kill -3 against a Java process normally results in the
generation of a javacore.txt file. However, there are scenarios
when the JVM gets into a state in which javacores are not
generated as a result of the kill -3. Follow these instructions if
the kill -3 command normally gives you javacores, but when
the JVM becomes hung or degraded, a kill -3 command does not
produce javacores. To determine the status of the Java threads
during the hang, you will generate AIX system core files which can be
processed to provide the needed information. The instructions in the
Machine Setup section will insure that your server is configured to
generate a full core file.
Machine Setup:
- Run the following command:
export DISABLE_JAVADUMP=true |
|
This should set an environment variable in any script
that starts a Java process. |
|
- Use one of the following 2 methods to insure that AIX is configured to
produce a full core file:
- Use the System Management Interface Tool (SMIT) to
create full core dumps by starting SMIT and setting the following to
"TRUE":
System Environments > Change/Show Characteristics of Operating
System > Enable Full CORE dump
- Or, enter the following commands:
lsattr -Elsys0 | grep full |
|
To set full core to true: |
|
chdev -a fullcore=true
-lsys0 |
|
- Ensure that your ulimit -c and ulimit -f is set high
enough.
Note: Units are in 512-byte blocks. Set to unlimited to ensure
that you have enough free file system space in your
install_root/bin directory (at least 750 MB to 2
GB).
To check the current settings, run the following commands:
ulimit -c
ulimit -f |
|
To set to unlimited, run the following commands: |
|
ulimit -c unlimited
ulimit -f unlimited |
|
- Ensure that the WebSphere Application Server process can write to the
install_root\bin directory (default, unless the
workingDirectory variable is changed).
- Check the file, /etc/security/limits. The stanza for the
user that runs the process should have fsize = -1 and core =
-1. Setting these values to -1 changes the setting to unlimited; you
must ensure that you have enough free space to handle a large core file.
- Restart all Java processes.
When the Hang occurs:
Collect the documentation as detailed in the following steps:
- Run the following command:
netstat -an > netstat.out |
|
Note: If the Web server is remote, gather
netstat output on the Application Server machine and Web server
machine. |
|
- Run the following command:
- This step differs based upon the version of AIX:
- For AIX V5.2:
svmon -P <pid> -m -r -i
600 > svmon.out |
|
- For AIX V5.1:
svmon -P -m -r <pid> -i 600 >
svmon.out |
|
- For AIX V4.3.3:
svmon -P -m <pid> -i 600 >
svmon.out |
|
- Can you serve a static document from the Web server?
- Collect CPU usage statistics by issuing the following command:
tprof -k -s -e -x sleep 60 |
|
- Collect a process dump of the Java process by executing the
pdump.sh script (attached below):
- Collecting system core files:
- If you are running AIX version 5.2 or later,
execute the following steps to produce a series of three AIX core files:
- Run the following command:
gencore java_pid
core1.process_ID |
|
- Wait for 3 to 5 minutes to allow for core file generation.
- Repeat the preceding steps a. and b. to generate two additional core
files at 3 to 5 minute intervals. It is important to remember to use a
different core file name each time you run the command (for example:
core2.process_id, core3.process_id).
- If you are running AIX version 5.1 or earlier,
run the following command to generate an AIX operating system core file:
- Execute the getlib.sh script (attached below) to collect the
system libraries. Using the core files and the system libraries, IBM
support will be able to create Java thread stacks for the JVM. To run the
script:
./getlib.sh java_executable_fullpath
core_file_name |
|
This creates the getlib_archive.tar.Z file.
Note: You must have dbx installed to run this command. |
|
- If the Developer Kit is 131 SR5 (131-20030618) or later, run the
following command against each core file:
./jextract -o coreN.sdff
[CORE_PATH] |
|
Where,
jextract |
is found in the
install_root/java/jre/bin directory |
coreN.sdff |
is the output file and "N" is the number (for example: 1,
2, 3, and so on) |
[CORE_PATH] |
is the path and file name for the core file |
|
|
- Gather the following information:
- For WebSphere Application Server V6.0:
- Include all of the files from the following directory:
profile_root/logs/server_name |
|
Include the following Application Server log files if
they are located in a different location:
- Systemout
- Systemerr
- Native_stderr
- Native_stdout
|
|
- The server.xml file located in the following
directory:
profile_root/config/cells/cell_name/
nodes/node_name/servers/server_name |
|
- For WebSphere Application Server V5.1 and 5.0:
- Include all of the files from the following directory:
install_root/logs/server_name |
|
Include the following Application Server log files if
they are located in a different location:
- Systemout
- Systemerr
- Native_stderr
- Native_stdout
|
|
- The server.xml file located in the following
directory:
install_root/config/cells/cell_name/
nodes/node_name/servers/server_name |
|
- For all releases of WebSphere Application Server
- The pdump output file.
- The core files. Ensure you compress these files as they
can be very large.
- All standard data file format (*.SDFF)
files
- All getlib_archive.tar.Z files
- If the Web server is remote, send the
http_plugin.log file from the Web server
machine.
- All netstat.out files
- All vmstat.out files
- All smvon.out files
- Files generated by tprof (especially
_prof.all). These files are in the directory from which the
tprof command was run and begin with an underscore ( _ ). Some
versions of AIX might produce just a sleep.prof file.
- Follow instructions to submit
diagnostic information to IBM Technical Support for problem
determination.
For a listing of all technotes, downloads, and educational materials
specific to a hang or performance degradation, search the WebSphere
Application Server support site. |
|
|
|
|
|
Cross Reference information |
Segment |
Product |
Component |
Platform |
Version |
Edition |
Application Servers |
WebSphere Application Server - Express |
Hangs/performance degradation |
AIX |
6.0, 5.1, 5.0 |
|
Application Servers |
Runtimes for Java Technology |
Java SDK |
|
|
|
|
|
|