|
Problem(Abstract) |
This is a problem determination document to assist in the
analysis of synchronization problems with the WebSphere® Application
Server version 5.0, 5.1, and 6.0 product. |
|
|
|
Resolving the
problem |
 |
Trace/Exception analysis:
To investigate the synchronization process, the traces from the MustGather
are needed.
Review the following errors and links for solutions:
Review this technote SocketTimeoutException
Read timed out seen during SyncNode request if you the following error
within the syncNode.log file:
[6/20/06 8:36:03:489 GMT+01:00]
405b06ac SOAPConnector < reconnect
[SOAPException: faultCode=SOAP-ENV:Client;
msg=Read timed out;
targetException=java.net.SocketTimeoutException: Read timed out]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(Unknown
Source |
|
Using annotated trace comparison for analysis. Reviewing the following
trace, and then comparing against the trace gathered, greatly assists in
pinpointing the problem area.
The following trace illustrates a synchronization operation from start to
finish. The following trace level is set for both dmgr and node agent:
Admin=all=enabled:Sync=all=enabled |
All traces are taken from the node agent trace file, except the ones
starting with "(dmgr trace)".
Step 1. Initialization sync:
[8/22/04 21:27:04:686 EDT] 2e4c94c6 NodeSyncTask > initiate:
false
[8/22/04 21:27:04:686 EDT] 2e4c94c6 SyncResult >
initialize
[8/22/04 21:27:04:686 EDT] 2e4c94c6 SyncResult <
initialize
[8/22/04 21:27:04:686 EDT] 2e4c94c6 NodeSyncTask d setState:
1 <-- sync state: IDLE=0 COMPARE_FOLDERS=1 PROCESSING_UPDATE=2
PROCESSING_UPDATE_WITH_ERROR=3 VERIFY_SYNC_RESULT=4
[8/22/04 21:27:04:686 EDT] 2e4c94c6 RuntimeCollab >
sendNotification <-- send sync initiated
notification
websphere.nodesync.initiated[source=WebSphere:platform=common,cell=yiwen2Network,version=
5.0,name=nodeSync,mbeanIdentifier=nodeSync,type=NodeSync,node=yiwen2,process=nodeagent]
Step 2. Do sync:
[8/22/04 21:27:53:446 EDT] 2b3994ce NodeSyncTask d run
<-- a new thread is spawn (notice the thread ID 2b3994ce
is different from the sync init thread, 2e4c94c6)
[8/22/04 21:27:53:446 EDT] 2b3994ce NodeSyncTask
> doSync <-- start doSync method,
which does most of the work
[8/22/04 21:27:53:446 EDT] 2b3994ce NodeSync >
getCellRepositoryEpoch <-- compare dmgr's repository
epoch with node's repository epoch
(dmgr trace) [8/22/04 21:27:53:517 EDT] 55afd4cd SOAPConnectio >
handleRequest <-- dmgr receives the JMX call
......
(dmgr trace) [8/22/04 21:27:53:567 EDT] 55afd4cd SOAPConnector d
Parameter: getRepositoryEpoch(type: class java.lang.String)
......
(dmgr trace) [8/22/04 21:27:53:577 EDT] 55afd4cd SOAPConnectio <
handleRequest <-- dmgr finishes processing the JMX
call
Step 2.1 Make getModifiedFolders() JMX call:
[8/22/04 21:27:53:587 EDT] 2b3994ce NodeSync >
invokeGetModifiedFolders <-- method entry
(dmgr trace) [8/22/04 21:27:53:627 EDT] 478594c2 SOAPConnectio >
handleRequest <-- dmgr receives the JMX call
......
(dmgr trace) [8/22/04 21:27:53:647 EDT] 478594c2 SOAPConnector d
Parameter: getModifiedFolders(type: class java.lang.String)
......
(dmgr trace) [8/22/04 21:27:54:878 EDT] 478594c2 SOAPConnectio <
handleRequest <-- dmgr finishes processing the JMX
call
.....
[8/22/04 21:27:54:878 EDT] 2b3994ce NodeSync d
modifiedFolders <-- this is a list of folders changed in
master repository in dmgr
cells/yiwen2Network/applications/perfServletApp.ear added
cells/yiwen2Network/applications/perfServletApp.ear/deployments
added
cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp
added
cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp/META-INF
added
........
[8/22/04 21:27:54:878 EDT] 2b3994ce NodeSync <
invokeGetModifiedFolders <-- method exit
Step 2.2.2 Make getFolderSyncUpdate() JMX call:
[8/22/04 21:27:54:899 EDT] 2b3994ce NodeSync >
invokeGetFolderSyncUpdate <-- method entry
..........
(dmgr trace) [8/22/04 21:27:54:919 EDT] 478094c2 SOAPConnectio >
handleRequest <-- dmgr receives the JMX call
.........
(dmgr trace) [8/22/04 21:27:54:949 EDT] 478094c2 SOAPConnector d
Parameter: getFolderSyncUpdates(type: class java.lang.String)
..........
(dmgr trace) [8/22/04 21:27:55:179 EDT] 478094c2 SOAPConnectio <
handleRequest <-- dmgr finishes processing the JMX
call
..........
[8/22/04 21:27:55:229 EDT] 2b3994ce FileTransferC > downloadFile
(5) to InputStream.
srcPath=cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear53033.tmp,
options=com.ibm.ws.management.filetransfer.client.FileTransferOptionsImpl@551d94c8
<-- changed document is downloaded to node
..........
[8/22/04 21:27:56:030 EDT] 2b3994ce NodeSync <
invokeGetFolderSyncUpdate <-- method exit, a list
of changed documents is returned
cells/yiwen2Network/applications/perfServletApp.ear
cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear
created
cells/yiwen2Network/applications/perfServletApp.ear/deployments
cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp
........
Step 2.3 Process each document:
[8/22/04 21:27:56:030 EDT] 2b3994ce NodeSyncTask d
processFolderSyncUpdate:
cells/yiwen2Network/applications/perfServletApp.ear <--
processing perfServletApp.ear folder
......
[8/22/04 21:27:56:190 EDT] 2b3994ce FileRepositor A ADMR0009I:
Document
cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear was
created <-- create the file in
perfServletApp.ear folder
.......
[8/22/04 21:27:56:210 EDT] 2b3994ce NodeSyncTask d
processFolderSyncUpdate:
cells/yiwen2Network/applications/perfServletApp.ear/deployments
<-- processing deployments folder
.......
[8/22/04 21:27:56:230 EDT] 2b3994ce FileRepositor A ADMR0009I:
Document
cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp/deployment.xml
was created. <-- create the file in
deployments folder
.......
Step 3 Post sync operations:
[8/22/04 21:27:56:971 EDT] 2b3994ce AppBinaryProc d xxxxxxxxxxx pre:
-1 xxxxxxxxxx <--- app mgmt code gets invoked
......
[8/22/04 21:27:57:102 EDT] 2b3994ce AppBinaryProc < postProcess:
-1 <-- app mgmt code gets invoked (do not pay attention
to the -1, unless sync fails when around here, when AppBinaryProcessor is
called)
......
[8/22/04 21:27:57:102 EDT] 2b3514ce AppBinThread > run
<-- app mgmt spawn a new thread to expand the ear (note
different threads: 2b3514ce 2b3994ce)
[8/22/04 21:27:57:102 EDT] 2b3514ce AppBinThread d ----
Expand
cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear to
[D:\was5\base\installedApps\yiwen2Network\perfServletApp.ear]
........
[8/22/04 21:27:57:102 EDT] 2b3994ce RuntimeCollab >
sendNotification <-- sending out sync complete
notification
websphere.nodesync.complete[source=WebSphere:platform=common,cell=yiwen2Network,version=5.0,name=
nodeSync,mbeanIdentifier=nodeSync,type=NodeSync,node=yiwen2,process=nodeagent]
......
[8/22/04 21:27:57:242 EDT] 2b3994ce NodeSyncTask
< doSync <-- existing doSync
method, we are done |
The following ZIP file contains the node agent and dmgr traces used in
the preceding trace analysis:
For information on sending documentation to IBM click Submitting
Diagnostic Information to IBM.
For a listing of all technotes, search the WebSphere
Application Server support page. |
|
|
|
|
|