Exceptions similar to ones listed below will be written to standard output (stdout), standard error (stderr), or to an application log file, depending on the application's configuration:
Exception in thread "main" java.lang.UnsatisfiedLinkError: fontmanager
(Not found in com.ibm.oti.vm.bootstrap.library.path)
at
java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1279)
at
java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:124
5)
at java.lang.System.loadLibrary(System.java:540)
at
sun.font.FontManagerNativeLibrary$1.run(FontManagerNativeLibrary.java
:73)
at
java.security.AccessController.doPrivileged(AccessController.java:503
)
at
sun.font.FontManagerNativeLibrary.
ava:44)
..........
To look up the failing Native library, prefix the library name with "lib" and suffix with ".so".
Example:
For the failing native library "fontmanager" in the exception:
Exception in thread "main" java.lang.UnsatisfiedLinkError: fontmanager
..............
search for the library by prefixing with "lib" and suffixing with ".so" :
libfontmanager.so
Before proceeding with the setup and data collection instructions to collect complete diagnostic data for analysis, click Step 7. Upload Data for instructions to upload any existing generated logs and data collected.
To prepare for these data collection procedures, the process environment needs to be configured to collect complete data and save the additional debug information to a log file.
A. Set the user ulimits:
From a command prompt and while logged in as the root user, execute following commands to:
{ set file and data ulimit sizes to unlimited }
# chuser fsize=-1 data=-1 USERID
B. Redirect or save standard error (stderr) messages to a file
Commonly used application servers may already save standard out and standard error messages to a log file (e.g., SystemOut.log native_stdout.log, SystemErr.log, native_stderr.log) or to the application log file.
For custom applications, redirect the standard error messages by appending "2>&LOG_FILE" or to redirect both the stdout and stderr to a file append ">LOG_FILE 2>&1".
C. Perform the following actions in order for the changes to take effect:
- Stop the application
- Relogin as the "USERID" used in Step 1.A
- Confirm the new ulimits are in effect by executing the commands:
# ulimit -a
- Do not restart the application until Step 3. Configure has been completed.
A. Enabling debug options will result in additional data being stored in memory buffers and written to application logs. The process file and data sizes should be increased during the data collection to ensure the data is complete.
If there are multiple processes executed by multiple user ids experiencing the issue, then all preparation steps must be repeated for each id and process.
To confirm the process environment is configured correctly, login using the USERID specified in the steps, then run the command:
# ulimit -a
The values for "file" and "data" should show as:
file(blocks) unlimited
data(kbytes) unlimited
B. As an example, to save standard error message to the file /tmp/stderr.log, use a command line syntax similar to:
# java YOUR_APP 2>/tmp/stderr.log
to save both standard out and standard error messages to the file /tmp/out.log, specify:
# java YOUR_APP > /tmp/out.log 2>&1
To confirm the messages are being redirected to the log file, view the contents of the log file.
A. Locate the failing library:
Locate the failing native library in the install directory of the specific Java version that has the problem. To search, add the prefix of "lib" and suffix of ".so" to the library name in the exception :
# export LIBNAME= libLIBRARY.so
# export LIBNAME_PATH="$(find JAVA_PATH -name $LIBNAME -print)"
Confirm the library exists by making sure full path to the library along with the library name is listed ::
# echo $LIBNAME_PATH
B. Set environment variables
Make sure path to the native library is set :
# export PATH="$(dirname $LIBNAME_PATH)":$PATH
# export LIBPATH="$(dirname $LIBNAME_PATH)":$LIBPATH
C. Add java command line options:
Enable verbose logging of the class loading and debug output:
-verbose:class,dynload,debug
Enable manual generation of javacores and snap traces:
-Xdump:java+snap:events=user
Generate Javacore and Snap trace files when the exception is thrown:
-Xdump:java+snap:events=systhrow+throw,filter=EXCEPTION,range=1..1
Enable tracing of the entry and exit of all the static initializers of the failing library:
"-Xtrace:maximal=all,methods={*.*()},resumecount=1,trigger=method{NATIVE-LIBRARY.
To designate a directory for the generated logs, instead of the default directory, add:
-Xdump:directory=SPECIFIC_PATH
D. Restart the java application (e.g., node agent/manager) from the USERID new login session.
A. Locate the failing library:
For the failing native library "fontmanager" from Step 1, search for the library as below:
Example:
$ export LIBNAME=libfontmanager.so
$ echo $LIBNAME
libfontmanager.so
$ export LIBNAME_PATH=$(find /usr/java7 -name $LIBNAME -print)
$ echo $LIBNAME_PATH
/usr/java7/jre/lib/ppc/libfontmanager.so
C . Alternate ways of setting java command line options and non-default directory for generated log files:
Environment variable can be used to set Java command line options. However, options specified on java command line take precedence over environment variable settings:
# export IBM_JAVA_OPTIONS="-verbose:gc -Xverbosegclog:SPECIFIC_PATH/gc.log -Xdump:java+heap+snap:events=systhrow+throw,filter=java/lang/UnsatisfiedLinkError,range=1..3"
For exception in step1:
Exception in thread "main" java.lang.UnsatisfiedLinkError: fontmanager
......
specify EXCEPTION as :
-Xdump:java+snap:events=systhrow+throw,filter=java/lang/UnsatisfiedLinkError,range=1..1
To enable tracing for the native library in the stack trace in Step 1,:
......
sun.font.FontManagerNativeLibrary.
ava:44)
...........
specify:
"-Xtrace:maximal=all,methods={*.*()},resumecount=1,trigger=method{sun.font.FontManagerNativeLibrary./clinit/
To direct javacore and snap trace to specific directories instead of the default directory of START_PATH, set environment variables:
# export IBM_JAVACOREDIR=SPECIFIC_PATH
D. For application server environments such as WebSphere and WebLogic, both the node manager/agent and the application servers must be restarted after the relogin of "USERID".
A. Execute the following commands to collect the required diagnostic data.
# mkdir -p /TMP_PATH/MM-DD/UnsatisfiedLinkError/data
# cd /TMP_PATH/MM-DD/UnsatisfiedLinkError/data
Generated logs are written to the default directory of "START_PATH" or the user specified SPECIFIC_PATH.
B. Tar up the failing native library parent directory and its sub-directories to check the contents:
# tar -cvh -f ./native-library-dir.tar $(dirname $LIBNAME_PATH)
C. Check the dependents of the native library:
# ldd $LIBNAME_PATH 2>&1 | tee -a ldd.out
D. Dump the header entries of the native library.
Dump the loader section header :
for 32-bit Java library:
# dump -Hov $LIBNAME_PATH > dump.out
for 64-bit Java library:
# dump -X64 -Hov $LIBNAME_PATH > dump-64.out
E. Start java process with truss enabled :
# truss -aelfdo native-library-truss.out JAVA-COMPLETE-COMMAND-LINE
OR
# truss -aelfd JAVA-COMPLETE-COMMAND-LINE > native-library-truss.out
F. Collect output of AIX commands from the same AIX LPAR:
# errpt -a > errpt-a.out 2>&1
# oslevel -s > oslevel-s.out 2>&1
# prtconf > prtconf.out 2>&1
# lsps -a > lsps-a.out 2>&1
# lslpp -hac > lslpp-hac.out 2>&1
# ps avwwwg > ps-avwwwg.out 2>&1
G. Copy the files generated to the data directory "/TMP_PATH/MM-DD/UnsatisfiedLinkError/data" directory created in step 4.A:
# cp /START_PATH/javacore*txt ./
# cp /START_PATH/Snap*trc ./
# cp /START_PATH/j9snap.trc ./
Also copy standard error, standard output, SystemOut, SystemErr, gc.log, application logs and any other logs generated to the /TMP_PATH/MM-DD/UnsatisfiedLinkError/data directory.
A. Examples of commands to be executed:
# mkdir -p /large_fs/01-31UnsatisfiedLinkError/data
# cd /large_fs/01-31/UnsatisfiedLinkError/data
If "START_PATH" can not easily be determined, execute these commands to help determine the correct path:
Example:
# ps -ef | grep -i java
rt 3211380 1 0 May 31 - 1109:46 java -Dsrse_property=/rt/pmr/test/classloader/JvmTest-dir/j_test/etc/aib.ini....
to check where the generated javacore and other logs get written, excute:
# kill -3 3211380
# procwdx 3211380
3211380: /rt/pmr/test/
Check for the generated javacore and other logs in the the above directory:
# cd /rt/pmr/test
# ls -l *javacore*
-rw-r--r-- 1 rtstaff 234048 Aug 25 10:54 javacore.20150825.105402.3211380.0010.txt
C. Check native library dependencies :
Example:
# ldd /usr/java7/jre/lib/ppc/libfontmanager.so
/usr/java7/jre/lib/ppc/libfontmanager.so needs:
/lib/libC.a(shr.o)
/lib/libC.a(shr2.o)
/lib/libC.a(ansi_32.o)
Cannot find libawt.so
Cannot find libmawt.so
Cannot find libjava.so
/usr/java8/jre/bin/classic/libjvm.so
Cannot find libdbgwrapper70.so
/lib/libc.a(shr.o)
/lib/libC.a(shrcore.o)
/lib/libC.a(ansicore_32.o)
/lib/libc_r.a(shr.o)
/unix
/lib/libcrypt.a(shr.o)
E. Enable truss for the failing Java command line:
Example:
# truss -aelfdo native-library-truss.out
/usr/java7/bin/java -jar
-Xdump:system+java+snap:events=systhrow+throw,filter=java/lang/Unsatisfi edLinkError,range=1..1 "-Xtrace:maximal=all,methods={*.*()},resumecount=1,trigger=method{sun/fo nt/FontManagerNativeLibrary.
G. Confirm all files and directories have been saved to the data directory:
# cp /var/myapp/*.log ./
# ls javacore* *nap*
# ls *.log *.out *.txt System*
** MANDATORY **
Prior to packaging and uploading the data, confirm that the following files have been saved in the "/TMP_PATH/MM-DD/UnsatisfiedLinkError/data directory:
a. Javacore files
b. Snap traces
c. ldd and dump commands output
d. Tar file of the native library parent and sub-directories
e. Truss output
f. standard error, standard output, SystemOut, SystemErr, application logs and any other logs generated.
g. AIX commands output
After all of the files have been collected, from a command prompt, execute the following commands to package the requested files:
# cd /TMP_PATH/MM-DD/UnsatisfiedLinkError
# tar -cf - data | gzip -c > PMR.MM-DD.tgz
Examples of commands to be executed:
# cd /large_fs/01-31/UnsatisfiedLinkError
# tar -cf - data | gzip -c > 12345.678.000.01-31.tgz
Upload the packaged data to IBM secured servers using one of upload options provided on the "IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM" web page:
http://www-01.ibm.com/support/docview.wss?uid=isg3T1022619
If this step is reached by clicking "Step 7. Upload Data" in Step 2, at the end of the completion of the existing latest logs/data upload, click Step 2. Prepare Environment to continue with the data collection instructions.