"ws_read_domain_link: read error 76" in a plug-in trace

Technote (FAQ)
Problem
The following is an example of an error that can show up in the WebSphere® Application Server plug-in trace when the application server stops communicating with the Web server.
Cause
The errors reported in the HTTP Server OSE Plug-in trace are broad and rarely identify the root cause of the problem. However, they do indicate on which side of the OSE connection the problem resides. In this particular example, the WebSphere® Application Server has stopped responding.
Solution
The following trace segment shows at which point the problem was experienced in the request and response process:

Note:
  • Each line begins with a date and time stamp (for example: Sat Dec 1 18:04:58 2001), followed by the process and thread identifier (for example: 0000302a 00004e4f).
  • For easier viewing, the date and time stamp, and the process and thread identifiers have been removed from the following:

---------------------------------------------------------
Warning - ws_read_domain_link: read error 76
Trace - callback_mode client_queue ibmoselink1 data type 102
Trace - ERROR: In do_service_time_request, enter_callback_mode failed
Error - mediate_service_to_clone: Error 1 from clone 0, uri /NotesTestSingleORBandIOR
Trace - router_service : Failed 1
Trace - ose_service() Failed: return code: 1
Trace - service_exit return_code: 3
Warning - ws_read_domain_link: read error 76
Trace - callback_mode client_queue ibmoselink1 data type 102
Trace - ERROR: In do_service_time_request, enter_callback_mode failed
Error - mediate_service_to_clone: Error 1 from clone 0, uri /NotesTestSingleORBandIOR
Trace - router_service : Failed 1
Trace - ose_service() Failed: return code: 1
Trace - service_exit return_code: 3
---------------------------------------------------------

The application server opens a listening socket for inbound connections. When the plug-in has all of the information associated with a request, it opens a socket, and a connection is established between the plug-in and the application server. The plug-in then sends the request, and waits for the reply. The above trace segment indicates that the connection was dropped on the application server side. A connection is closed by the application server when the servlet engine believes that the response has been completed. The application server calls a function to finish the connection.

The information from the trace segment above is associated with the part of the process where the plug-in is waiting for the response associated with the request that was sent. It has issued a "read", which failed. Any plug-in error messages that indicate a "read" or a "write" are referring to socket based actions on the connection between the plug-in and the application server. These routines report the TCP/IP error code thrown. In this case. the read action (an attempt to read the HTTP Response from the WebSphere Application Server) has failed with TCP/IP "error code 76" (the socket is not connected).

DB2® publishes a table of TCP/IP error codes in their Message Reference. Alternately, go to http://www.ibm.com/db2, select Library and search for "TCP/IP" documents. These codes are also documented on UNIX® systems in the/usr/include/sys/errno.h directory.

Reviewing a plug-in trace is simplified if you start by reviewing the Web server logs for errors. The Web server access and error logs provide timestamps, the URI used in requests and the return code (for example: 500). Matching these values with timestamps in the plug-in trace eliminates the need to review the entire trace.

It is also important to check the application server logs for errors. In some cases a servlet engine trace will be needed in order to match up the activity in the plug-in trace with the corresponding activity in the application server's servlet engine trace. In many cases, the errors in the plug-in trace are only symptoms and the root cause can be located in the application server. In these cases a servlet engine trace or thread dump of the application server will help identify root cause.

As is always the case, the OSE connection is controlled by the plug-in on one side and the WebSphere Application Server on the other. In this case, the question to ask is why did the application server close its side of the connection?

To examine the TCP/IP activity on the application server side, perform the following:
  1. Go to the administrative console.
  2. Select the servlet engine under the application server that you are using.
  3. Click on the Advanced tab.
  4. Click Settings.
  5. Check off all levels of trace and perform the test again.

A log will be produced for the TCP/IP activity on the application server side. This hopefully shows what is happening on the application server side.
Related information
DB2 Knowledge Base - TCP/IP Errors












Document Information

Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Plug-in
Operating system(s): AIX, HPUX, Linux, Multi-Platform, Solaris, Windows
Software version: 3.5
Software edition: All Editions
Reference #: 1046123
IBM Group: Software Group
Modified date: 2004-12-01