|
Problem |
Solaris® 8 ships with two threading libraries:
many-to-many and one-to-one. The names refer to how the model maps Solaris
threads to lwps and kernel threads. By default, the many-to-many thread
library is used in Solaris 8. In certain circumstances, this leads to
thread starvation and even hangs; eventually, kill -3 (used to create a
thread dump) does not work. In our experience, Oracle® and WebSphere®
Application Server running on Solaris 8 seems particularly prone to these
problems. |
|
Cause |
Reasons for changing the threading library
By default, the many-to-many thread library is used on Solaris 8.
Switching to the one-to-one library reduces the possibility of thread
starvation and improves scalability. This improves the overall performance
of the system. One-to-one is the default implementation in Solaris 9 and
subsequent releases.
|
|
Solution |
Determining which library is currently in
use
Run pldd PID and check the output for the path to
libthread.so. The pldd command lists details of all dynamic libraries
linked into the process.
- usr/lib/libthread.so
the many-to-many threading library is currently in use
- usr/lib/lwp/libthread.so
the one-to-one alternate threading library is currently in use
Changing to the alternate threading library
There are two ways to change the threading library. The first method is
preferred.
- Change the LD_LIBRARY_PATH variable (instructions below)
or
- Set the JVM Arg -XX:UseBoundThreads.
Changing to the alternate threading library on V4.0 AE releases
- Edit startupServer.sh located in the WAS_HOME/bin
directory.
- Search on $LD_LIBRARY_PATH in the file.
- Prefix /usr/lib/lwp to $LD_LIBRARY_PATH
Example:
LD_LIBRARY_PATH="$WAS_LIBPATH":/usr/lib/lwp$LD_LIBRARY_PATH
export LD_LIBRARY_PATH ;;
- Restart the server instance.
Changing to the alternate threading library on V5.0 and V5.1
releases
LD_LIBRARY_PATH can be set at one place in setupCmdLine.sh.
- Edit the setupCmdLine.sh file located in the install_root/bin
directory.
- Search on WAS_LIBPATH in the file.
- Add :/usr/lib/lwp to the WAS_LIBPATH.
Example:
WAS_LIBPATH="$WAS_HOME"/bin:/opt/mqm/java/lib:/opt/wemps/lib:/usr/lib/lwp
- Restart the server instance.
Benefits of changing the threading library
If you run pstack against the hung state, there is at least one
thread stack with mutex_lock at the top and some form of signal
processing further down in the stack.
- Sample pstack:
----------------- lwp# 218 --------------------------------
ff29bcfc ___lwp_mutex_lock (ff37e000, 1, 0, 0, 0, 2) + c
ff359afc _park (c8e81e30, ff37e000, 0, c8e81d70, 24d54, c9e81d70) + 11c
ff3597bc _swtch (c8e81d70, c8e81d70, ff37e000, 5, 1000, 0) + 424
ff35b178 _mutex_adaptive_lock (ff3898ec, 4c00, 1000, fffeffff, 1, 4d58) +
160
...................................
__1cbCCompiledCodeSafepointHandlerbKhandle_illegal_instruction_exception6M_
- Sample pstack:
----------------- lwp# 335 / thread# 4144 --------------------
ff29ef88 lwp_mutex_lock (ff385908)
ff3680e0 sigacthandler (21, e1381d70, e137ec70, 0, cb5e58,
ff37e000) + 310
--- called from signal handler with signal 33 (SIGLWP) ---
..................................................
e53d55e4 Java_oracle_jdbc_oci8_OCIDBAccess_do_1execute (cd1744,
e138032c,
If you see the indicators above in a pstack of a hung JVM™, change to the
alternate thread library.
Full details of this fix are provided on the Sun support site:
http://java.sun.com/docs/hotspot/PerformanceFAQ.html#8
A full description of Solaris threading is provided here:
http://java.sun.com/docs/hotspot/threads/threads.html
|
|
|
|
|
|
|