Content manager OnDemand Content manager OnDemand ODWEK troubleshooting - Servlet This presentation covers troubleshooting the ODWEK servlet. ODWEK servlet troubleshooting topics ODWEK servlet troubleshooting topics Known installation issue Initialization and deployment errors Accessing the ODWEK servlet returns a 500 Internal Server Error Accessing the ODWEK servlet returns a 404 error Abends Hang This module covers an installation issue, initialization and deployment errors on the ODWEK servlet, servlet abends and hang situations. ODWEK known installation issue ODWEK known installation issue Running the ODWEK V8.3 InstallShield installer on AIX® might not update the ODM if using Java 1.4.1 Flash: http://www-1.ibm.com/support/docview.wss?rs=129&context=SSEPCD&context=SSEPC6&context=SS3U2Y&dc=D600&uid=swg21243190&loc=en_US&cs=utf-8&lang=en Ensure Java 1.4.2 is installed when running the ODWEK V8.3 InstallShield installer The InstallShield installer will look for Java 1.4.2 without the System Administrator having to specify environment variables/configuration to do so In V8.4, OnDemand/ODWEK InstallShield installers now ship with a built-in JVM™ for use during installation If ODWEK is installed on V8.3 of AIX the ODM might not be updated if using Java V1.4.1. The solution is to use Java V1.4.2 In V8.4, a JVM™ is built into the installer, preventing these issues from occurring in the future. ODWEK servlet returns a 500 internal server error ODWEK servlet returns a 500 internal server error Check the application server’s SystemOut.log for specific ODWEK initialization error messages SystemOut.log and other server logs are located in $WAS_HOME/profiles/$profile_name/logs/$WAS_server_name Ensure all steps were followed in the Copying Files section of the ODWEK Implementation Guide: http://publib.boulder.ibm.com/infocenter/cmod/v8r4m0/topic/com.ibm.ondemand.mp.doc/ars1y37152.htm ODWEK V7.1.2.7 and later, step two should use ArsSVTInterface.class instead of ArsWWWInterface.class ODWEK V8.4 and later, step three should reference libarswwwsl32 or arswwwsl32 native libraries Optionally: steps one and two specify these files/directories in the application server’s CLASSPATH instead For situations where the ODWEK servlet returns a 500 Internal server error: Check the application server’s SystemOut.log for specific ODWEK initialization error messages. Ensure all steps were followed within the copying files section of the ODWEK Implementation Guide. Within ODWEK V7.1.2.7 and later, step two should use the ArsSVTInterface.class instead of the ArsWWWInterface.class. For ODWEK V8.4 and later, step three should reference libarwwwsL32 or arswwwsL32 native libraries. Optionally steps one and two specify these files and directories in the application server’s CLASSPATH instead. ODWEK servlet returns a 500 internal server error ODWEK servlet returns a 500 internal server error Confirm that the ConfigDir parameter is set correctly in the web.xml Directory containing arswww.ini Ensure the ConfigDir parameter is named correctly Example: ‘Config Dir’ is incorrect ConfigDir parameter and value can be found: WebSphere Administration Console-> Enterprise Applications-> $ODWEK_app_name-> Web Modules-> $ODWEK_war-> View Deployment Descriptor web.xml is located: $WAS_HOME/profiles/$profile_name/config/cells/$node_name/applications/$ODWEK.ear/deployments/$ODWEK_app_name/$ODWEK.war//WEB-INF/web.xml Additionally confirm that the ConfigDir parameter within the web.xml is configured correctly. It must point to the directory containing the arswww.ini file. Also, check that the ConfigDir is named correctly. The parameter “ConfigDir” cannot contain spaces, dashes or underlines. The ConfigDir parameter and value can be found in the WebSphere Administration Console/Enterprise Applicatins/$ODWEK_app_name/Web Modules/$ODWEK.war/View Deployment Descriptor. The web.xml can be obtained from the path indicated. ODWEK servlet returns a 500 internal server error ODWEK servlet returns a 500 internal server error Ensure TemplateDir, CacheDir, TempDir, and TraceDir are valid directories in the arswww.ini Permissions for the UNIX® or Windows® account running the ODWEK application CacheDir, TempDir, TraceDir - read/write/execute permissions arswww.ini and TemplateDir - read access ODWEK program files - read and execute permissions Ensure that the TemplateDir, CacheDir, TempDir and TraceDir are valid directories in the arswww.ini file. Check the permissions for the UNIX/Windows account running the ODWEK application. CacheDir, TempDir and TraceDir must have read/write/execute permissions. The arswww.ini and TemplateDir must have read access and the ODWEK program files must have read and execute permissions. Accessing the ODWEK servlet returns a 404 error Accessing the ODWEK servlet returns a 404 error Can the ODWEK servlet be invoked directly using the application server’s built in HTTP server? To test Specify the host name HTTP server port of the application server default for WebSphere Application Server is 9080 When accessing the ODWEK servlet, it returns a 404 error. Can the ODWEK servlet be invoked directly using the application server’s built in HTTP server? To test, specify the host name, the HTTP server port of the application server, the servlet’s context-root, and the servlet mapping’s URL pattern that was specified during the assembly and deployment of the servlet. Accessing the ODWEK servlet returns a 404 error Accessing the ODWEK servlet returns a 404 error servlet’s context-root servlet mapping’s URL pattern that was specified during the assembly and deployment of the servlet Example: OnDemandWEKWeb is the context-root and /od/* is the servlet mapping’s URL pattern. Specify the URL below to invoke the ODWEK servlet directly: http://host name:9080/OnDemandWEKWeb/od The return output should be ‘Internet Connection Version X.X.X.X The argument '_function' was not specified’ X.X.X.X is the ODWEK version The default for WebSphere Application Server is 9080. For example, where OnDemandWEKWeb is the context-root and /od/* is the servlet mapping’s url-pattern, specify the URL below to invoke the ODWEK servlet directly: http://host name:9080/OnDemandWEKWeb/od The return output should be ‘Internet Connection Version X.X.X.X The argument '_function' was not specified’. Where X.X.X.X is the ODWEK version. Accessing the ODWEK servlet returns a 404 error Accessing the ODWEK servlet returns a 404 error WebSphere’s internal HTTP server port can be found at: WebSphere Administration Console-> Application Servers-> $WAS_server_name-> Web Container Settings-> Web container transport chains-> WcInboundDefault port Context-root can be found at: WebSphere Administration Console-> Enterprise Applications-> $ODWEK_app_name-> View Deployment Descriptor Servlet mapping’s URL pattern can be found at: WebSphere Administration Console-> Enterprise Applications-> $ODWEK_app_name-> Web Modules-> $ODWEK_war-> View Deployment Descriptor If the servlet can be invoked directly, configure and refresh the WebSphere HTTP server plug-in Open the external HTTP server’s configuration file $http_server_base/conf/httpd.conf, find and open the referenced plugin-cfg.xml WebSphere's internal HTTP server port, the Context-root and the servlet mapping’s URL pattern can be found by the paths specified. If the servlet can be invoked directly, the problem is in the WebSphere HTTP server plug-in. Accessing the ODWEK servlet returns a 404 error Accessing the ODWEK servlet returns a 404 error Make sure that the context-root of the ODWEK application is specified within the plugin-cfg.xml (see example below) Regenerate the plugin-cfg.xml through the WebSphere Administration Console if ODWEK’s context root is missing Restart the HTTP server for the new plugin-cfg.xml to be used Plugin-cfg.xml file Ensure that the context-root of the ODWEK application is specified in the plugin-cfg.xml The plugin-cfg.xml can be regenerated through the WebSphere Administration Console if ODWEK’s context root is missing. The HTTP server must be restarted for the new plugin-cfg.xml to be used. Accessing the ODWEK servlet returns a 404 error Accessing the ODWEK servlet returns a 404 error If the servlet cannot be invoked directly, the problem is in the deployment of the ODWEK servlet Ensure the servlet was deployed to the correct WebSphere server Confirm all default deployment options were used IBM Software Support can provide assembled working servlet for testing IBM Software Support might ask for your ODWEK servlet to test in their lab If the servlet can’t be invoked directly, the problem lies within the deployment of the ODWEK servlet. Ensure the servlet was deployed to the correct WebSphere server and all default deployment options were used. IBM Software Support can provide an assembled working servlet for testing. Your ODWEK servlet might be requested to test in an IBM lab environment. ODWEK servlet abend ODWEK servlet abend When ODWEK abends, the JVM and application server will also abend IBM Javacore (javacore*.txt) or Sun HotSpot error report file (hs_err*.log) is generated when JVM abends core dump file is generated as ODWEK has a native shared library It is within the ODWEK design to abend if it is unable to allocate sufficient native heap Due to the system being in an unstable state, attempting to avoid an abend will make troubleshooting more difficult When ODWEK abends, the JVM and application server will always abend. When JVM abends an IBM Javacores (javacore*.txt) or Sun HotSpot error report file (hs_err*.log) is generated. A corresponding core dump file is created as ODWEK has a native shared library. ODWEK can abend if it can’t allocate sufficient native heap. The system is considered in an unstable state, attempting to avoid an abend will make troubleshooting the unstable state more difficult. ODWEK servlet abend ODWEK servlet abend If the ODWEK servlet abends, the best source of diagnostic information is the native stack trace from the core dump and ODWEK trace=4 output The core file’s call stack provides the most information The Java current thread output is not useful Review the Java current thread output to verify the abend was in ODWEK Review the Java maximum and minimum heap settings to ensure sufficient native heap was configured If the ODWEK servlet abends, the best source of diagnostic information is the native stack trace from the core dump and ODWEK trace=4 output. Since the servlet does almost all processing in the native shared library, the core file’s call stack will provide the most information. The Java current thread output will almost always look the same and is not useful. Review the Java current thread output to verify the abend was in ODWEK. The Javacore is useful in this case to review the Java maximum and minimum heap settings, to ensure sufficient native heap was configured. ODWEK servlet abend ODWEK servlet abend Typical current thread stack trace from Javacore (below) core dump shows more than Javacore’s native stack Example below indicates abend is in the main method of ODWEK’s ArsSVTInterface class Native stack trace indicates abend is in the native getData method Thread stack trace from Javacore Following is a typical current thread stack trace from Javacore. Java stack traces almost always look the same, the native stack shows the most information. Core dumps will show more or the same information as the native stack. The example below shows the abend is in the main method of ODWEK’s ArsSVTInterface class. The native stack trace show more information and that the abend is in the native getData method. ODWEK servlet abend ODWEK servlet abend Most abends are caused by exhausted native memory (see Native Heap requirements slides) Javacore and core dump in this situation almost always show an abend in document retrieval Reasons for exhausted native memory: Very large documents being retrieved at one time You will find that most abends are caused by an exhaustion of the native memory. This information is covered in the Enabling Tracing/Logging and Requirements module. The Javacore and core dump created in this situation almost always show that the abend was caused during document retrieval. This situation is typically caused by very large documents being retrieved at one time, mis-configuration of the Java max heap or the volume of users has outgrown the system capacity. If a document is larger then 1MB (default specified by DocSize arswww.ini parameter), ODWEK will write the document data to a temp file in the TempDir, then read the whole document into native heap for processing. ODWEK servlet abend ODWEK servlet abend Example ODWEK trace=4 statements showing document file sizes: INFO,ArsWWW_retrieve,Document segment written to file [27629619] bytes INFO,ArsWWW_retrieve,Document data written to memory [850] bytes INFO,ArsWWW_retrieve,Compressed DocData Size is [2921] INFO,ArsWWW_retrieve,Uncompressed DocData Size is [6283] INFO,ArsWWW_retrieve,Compressed ResData Size is [11157] INFO,ArsWWW_retrieve,Uncompressed ResData Size is [35271] Native stack trace will show ODWEK is in a document retrieval method (retrieve, getData, output driver) Mis-configuration of Java max heap Volume of users outgrown system capacity Review ODWEK trace at the time of abend to confirm the ODWEK activity that caused the abend, and the size of document or resource being retrieved. To check the configuration of the Java max heap, check the Xmx setting in the Javacore to ensure there is sufficient native heap. ODWEK servlet abend ODWEK servlet abend Example native stack trace from an ODWEK core dump showing an abend in one of ODWEK’s document retrieval methods In this example of a native stack trace from an ODWEK core dump, it shows an abend during one of ODWEK’s document retrieval methods. ODWEK servlet abend ODWEK servlet abend Specific user operation causes a servlet abend Reproduce in test environment if issue is isolated to a specific operation Review the ODWEK trace=4 output to help in re-create Provide all diagnostic data for review Send information to IBM Support if further assistance is necessary If a specific user operation causes a servlet abend, attempt to re-create in a test environment if the issue can be isolated to a specific operation. Reviewing the ODWEK trace=4 output will help to re-create the problem. Refer to the Collecting Data for Servlet module and provide all diagnostic data for review. Send information collected to IBM Software Support if further investigation is necessary. ODWEK servlet hang/performance degradation ODWEK servlet hang/performance degradation ODWEK hang issues are typically caused by slow OnDemand server response ODWEK is a client to the OnDemand server and will wait indefinitely until a response or error is received from the OnDemand server No configurable method within ODWEK to timeout an operation Can the problem be re-created using the OnDemand Windows Client? If recreatable, eliminates ODWEK as the culprit Investigate the OnDemand server ODWEK hang issues are typically caused by slow OnDemand server response. ODWEK is a client to the OnDemand server and will wait indefinitely for a response or error from the OnDemand server. There isn’t a configurable method in which to timeout an operation within ODWEK, so ODWEK will wait indefinitely. Test to see if the hang or performance issue can be re-created using the OnDemand Windows Client. If the problem can be re-created, ODWEK is not the culprit and investigation of the OnDemand server should be performed in this situation. ODWEK servlet hang/performance degradation ODWEK servlet hang/performance degradation JVM reports hung threads after 10 minutes Thread can still be in a healthy running state Check if ODWEK threads complete eventually JVM reports when a reported hung thread has completed and length of time to complete If thread eventually completes, indication is not a hang, but slow execution/response from OnDemand server Provide all diagnostic data for review Send information to IBM Support if further assistance is necessary JVM reports hung threads after 10 minutes, although the JVM thread can still be in a healthy running state. Check to see if all ODWEK threads complete eventually. JVM reports when a hung thread has completed and the length of time it takes to complete. If the hung thread eventually completes, this is an indication that the servlet isn’t hung, but slow performance on the OnDemand server. Refer to Collecting Data for Servlet module and provide all diagnostic data for review. Send information to IBM Software Support if further investigation is necessary. ODWEK servlet additional information ODWEK servlet additional information ODWEK Servlet functions and parameters http://publib.boulder.ibm.com/infocenter/cmod/V8r4m0/topic/com.ibm.ondemand.mp.doc/ars1y371173.htm Running an ODWEK Java API application and the ODWEK servlet in the same WebSphere Application Server causes a JVM abend http://www-1.ibm.com/support/docview.wss?rs=0&q1=1303470&uid=swg21303470&loc=en_US&cs=utf-8&cc=us&lang=en For further information regarding ODWEK Servlet functions and parameters, refer to the link provided. There is additional information regarding a known issue when running an ODWEK Java API application and the ODWEK servlet in the same WebSphere Application Server in the second link. ODWEK additional information ODWEK additional information ODWEK Implementation Guide http://publib.boulder.ibm.com/infocenter/cmod/V8r4m0/topic/com.ibm.ondemand.mp.doc/ars1y371.htm ODWEK trace return codes See chapter 13.1.5 http://www.redbooks.ibm.com/abstracts/sg247646.html arswww.ini parameter descriptions http://publib.boulder.ibm.com/infocenter/cmod/V8r4m0//topic/com.ibm.ondemand.mp.doc/ars1y37158.htm The links provided here will give additional information regarding ODWEK implementation, trace return codes, arswww.ini parameter descriptions, and some known ODWEK problems. ODWEK additional information ODWEK additional information Scrolling or repeatedly opening an AFP document in the AFP Web Viewer Plug-in causes a browser abend http://www-1.ibm.com/support/docview.wss?rs=0&q1=1295357&uid=swg21295357&loc=en_US&cs=utf-8&cc=us&lang=en OnDemand Web Enablement Kit (ODWEK) and OnDemand server performance slows during high usage http://www-1.ibm.com/support/docview.wss?rs&uid=swg21223757 The links provided here gives you additional information regarding known ODWEK problems. Feedback Feedback Your feedback is valuable You can help improve the quality of IBM Education Assistant content to better meet your needs by providing feedback. Did you find this module useful? Did it help you solve a problem or answer a question? Do you have suggestions for improvements? Click to send e-mail feedback: mailto:iea@us.ibm.com?subject=Feedback_about_Troubleshooting_Servlet.ppt This module is also available in PDF format at: ../Troubleshooting_Servlet.pdf You can help improve the quality of IBM Education Assistant content by providing feedback. Trademarks