|
Problem(Abstract) |
MustGather for when your WebSphere® Application Server is
not responding (hangs) on the AIX® 5.2 platform and a "kill -3" command
does not produce javacore files. Gathering this information before calling
IBM® support will help familiarize you with the troubleshooting process
and save you time. |
|
|
|
Resolving the
problem |
If you already contacted support, continue to the
component-specific MustGather information. Otherwise, click on the
following link: MustGather:
Read first for all WebSphere Application Server products.
No javacore generated specific MustGather information
The following steps will ensure that AIX is set up to accept a full core:
- Go into SMIT and ensure that full core is enabled; or, from a
command line, enter the following:
lsattr -Elsys0 | grep full |
|
To set full core to true: |
|
chdev -a fullcore=true
-lsys0 |
|
- Ensure that your ulimit -c and ulimit -f is set high
enough.
Note: Units are in 512-byte blocks. Set to unlimited to
ensure that you have enough free file system space in your
install_root\bin directory (at least 750 MB to 2
GB).
To check the current settings run the following commands:
ulimit -c
ulimit -f |
|
To set to unlimited, run the following
commands: |
|
ulimit -c unlimited
ulimit -f unlimited |
|
- Ensure that the WebSphere Application Server process can write to the
install_root\bin directory (default, unless
workingDirectory variable is changed).
- Check the file, /etc/security/limits. The stanza for the user
that runs the process should have fsize = -1 and core =
-1. Setting these values to -1 changes the setting to
unlimited; you must ensure that you have enough free space to handle a
large core file.
- Enter the following command:
gencore
ProcessID FileName |
|
Example: To generate a core file, named
core.1095, for the process with process ID 1095, enter the
following command: |
|
gencore 1095 core.1095 |
|
The gencore command creates a core file of the
process specified by the ProcessID without terminating the process. |
|
- If you still do not get a core file after setting these parameters
correctly, do the following:
- Add the following lines to the beginning of the file,
startupServer.sh:
export DISABLE_JAVADUMP
export IBM_NOSIGHANDLER=true |
|
- Restart all WebSphere Application Server java processes.
- Enter the following command against the PID of the java process to
test getting a core file:
gencore
ProcessID Filename |
|
- If you can get a core from the test, wait for a hang to occur during
an error condition.
If you are using WebSphere MQ, it might trap the signal and not propagate
to the Java™ process.
Disable the WebSphere MQ signal handling by setting the following
environment variable:
MQS_NO_SYNC_SIGNAL_HANDLING |
|
Example: |
|
export
MQS_NO_SYNC_SIGNAL_HANDLING=true |
|
- When the Java process hangs, perform the following steps:
- Enter the following command:
gencore
ProcessID core1.ProcessID |
|
- Wait for 3 to 5 minutes to allow for core file generation.
- Enter the following command:
gencore
ProcessID core2.ProcessID |
|
- Wait for 3 to 5 minutes to allow for core file generation.
- Enter the following command:
gencore
ProcessID core3.ProcessID |
|
- Enter the following command:
./getlib.sh
java_executable_fullpath core_filename |
|
This creates getlib_archive.tar.Z.
Note: You must have dbx installed to run this
command. |
|
- Enter the following command:
tar cvf
core_file_name.tar core_file_name |
|
This creates core_file_name.tar. |
|
- If the Developer Kit is 131 SR5 (131-20030618) or later, run the
following command against each core file:
./jextract -o
coreN.sdff [CORE_PATH] |
|
Where,
jextract |
is found in
the install_root/java/jre/bin directory
coreN.sdff is the output file and "N" is
the number (for example: 1, 2, 3, and so on) |
[CORE_PATH] |
is the path
and file name for the core file |
|
|
- Collect the following diagnostic information:
- All dbxtrace.out files
- All standard data file format (SDFF) files
- errpt.out
- All getlib_archive.tar.Z files
- All core.tar files
- All WebSphere Application Server log files for V5.0 and
V5.1:
- Include all of the files from the following directory:
install_root\logs\server_name
Include the following Application Server log files if they are
located in a different location:
- SystemOut
- SystemErr
- native_stdout
- native_stderr
- Send the server.xml file located in the
following directory:
install_root\config\cells\cell_name\
nodes\node_name\servers\server_name
- All WebSphere Application Server log files for V6.0:
- Include all of the files from the following directory:
install_root\profiles\profile_name\logs\server_name
Include the following Application Server log files if they are located in
a different location:
- SystemOut
- SystemErr
- native_stdout
- native_stderr
- Send the server.xml file located in the
following directory:
install_root\profiles\profile_name\
configcells\cell_name\nodes\node_name\
servers\server_name
|
|
Note: See the attached scripts
dbxtrace_aix.sh and getlib.sh. To run the
dbxtrace_aix.sh file, you must have the dbx component of
AIX installed on the machine. |
|
- Follow instructions to send
diagnostic information to IBM support.
For a listing of all technotes, downloads, and educational materials
specific to the Hangs/Performance Degradation component, search the WebSphere
Application Server support site. |
|
|
|