Changing to the alternate, one-to-one threading library on Solaris 8 improves performance

Technote (FAQ)
Problem
Solaris® 8 ships with two threading libraries: many-to-many and one-to-one. The names refer to how the model maps Solaris threads to lwps and kernel threads. By default, the many-to-many thread library is used in Solaris 8. In certain circumstances, this leads to thread starvation and even hangs; eventually, kill -3 (used to create a thread dump) does not work. In our experience, Oracle® and WebSphere® Application Server running on Solaris 8 seems particularly prone to these problems.
Cause
Reasons for changing the threading library

By default, the many-to-many thread library is used on Solaris 8. Switching to the one-to-one library reduces the possibility of thread starvation and improves scalability. This improves the overall performance of the system. One-to-one is the default implementation in Solaris 9 and subsequent releases.

Solution

Determiningwhich library is currently in use

Run plddPID and check the output for the path to libthread.so. The pldd command lists details of all dynamic libraries linked into the process.
  • usr/lib/libthread.so
    the many-to-many threading library is currently in use
  • usr/lib/lwp/libthread.so
    the one-to-one alternate threading library is currently in use

Changing to the alternate threading library

There are two ways to change the threading library. The first method is preferred.

  1. Change the LD_LIBRARY_PATH variable (instructions below)

    or

  2. Set the JVM Arg -XX:UseBoundThreads.

Changing to the alternate threading library on V4.0 AE releases
  1. Edit startupServer.sh located in the WAS_HOME/bin directory.

  2. Search on $LD_LIBRARY_PATH in the file.

  3. Prefix /usr/lib/lwp to $LD_LIBRARY_PATH

    Example:
    LD_LIBRARY_PATH="$WAS_LIBPATH":/usr/lib/lwp$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH ;;

  4. Restart the server instance.

Changing to the alternate threading library on V5.0 and V5.1 releases

LD_LIBRARY_PATH can be set at one place in setupCmdLine.sh.
  1. Edit the setupCmdLine.sh file located in the install_root/bin directory.

  2. Search on WAS_LIBPATH in the file.

  3. Add :/usr/lib/lwp to the WAS_LIBPATH.

    Example:
    WAS_LIBPATH="$WAS_HOME"/bin:/opt/mqm/java/lib:/opt/wemps/lib:/usr/lib/lwp

  4. Restart the server instance.

Benefits of changing the threading library

If you run pstack against the hung state, there is at least one thread stack with mutex_lock at the top and some form of signal processing further down in the stack.
  1. Sample pstack:

    ----------------- lwp# 218 --------------------------------
    ff29bcfc ___lwp_mutex_lock (ff37e000, 1, 0, 0, 0, 2) + c
    ff359afc _park (c8e81e30, ff37e000, 0, c8e81d70, 24d54, c9e81d70) + 11c
    ff3597bc _swtch (c8e81d70, c8e81d70, ff37e000, 5, 1000, 0) + 424
    ff35b178 _mutex_adaptive_lock (ff3898ec, 4c00, 1000, fffeffff, 1, 4d58) + 160
    ................................... __1cbCCompiledCodeSafepointHandlerbKhandle_illegal_instruction_exception6M_

  2. Sample pstack:

    ----------------- lwp# 335 / thread# 4144 --------------------
    ff29ef88 lwp_mutex_lock(ff385908)
    ff3680e0 sigacthandler(21, e1381d70, e137ec70, 0, cb5e58, ff37e000) + 310
    --- called from signal handler with signal 33 (SIGLWP) ---
    ..................................................
    e53d55e4 Java_oracle_jdbc_oci8_OCIDBAccess_do_1execute (cd1744, e138032c,

If you see the indicators above in a pstack of a hung JVM™, change to the alternate thread library.

Full details of this fix are provided on the Sun support site:
http://java.sun.com/docs/hotspot/PerformanceFAQ.html#8

A full description of Solaris threading is provided here:
http://java.sun.com/docs/hotspot/threads/threads.html











Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server
Operating system(s): Solaris
Software version: 3.5, 4.0, 5.0, 5.1, 6.0
Software edition: Edition Independent
Reference #: 1107291
IBM Group: Software Group
Modified date: 2004-07-13