WebSphere Enterprise Service Bus for z/OS, Version 6.2.0 Operating Systems: z/OS


Troubleshooting version-to-version migration

Review this page for troubleshooting tips if you encounter problems while migrating from an older version of WebSphere® ESB

The following sections describe specific errors and exceptions that may occur in a version-to-version migration and provide steps you can follow to understand and resolve these problems.

Application installation error

If you select the option for the migration process to install the enterprise applications that exist in the version 6.0.x or 6.1.x configuration into the new version 6.2 configuration, you might encounter some error messages during the application-installation phase of migration.

The applications that exist in the version 6.0.x or 6.1.x configuration might have incorrect deployment information—usually, incorrect XML documents that were not validated sufficiently in previous WebSphere ESB runtimes. The runtime now has an improved application-installation validation process and will fail to install these malformed EAR files. This results in a failure during the application-installation phase of WBIPostUpgrade and produces an "E:" error message.

If the application installation fails in this way during migration, you can do one of the following:
  • Fix the problems in the version 6.0.x or 6.1.x applications, and then remigrate.
  • Proceed with the migration and ignore these errors.

    In this case, the migration process does not install the failing applications but does complete all of the other migration steps.

    Later, you can fix the problems in the applications and then manually install them in the new version 6.2 configuration using the administrative console or an install script.

Application server error

After you migrate a managed node to version 6.2, the application server might not start.

When you try to start the application server, you might see errors similar to those in the following example:
[5/11/06 15:41:23:190 CDT] 0000000a SystemErr  R  
    com.ibm.ws.exception.RuntimeError:
com.ibm.ws.exception.RuntimeError:  org.omg.CORBA.INTERNAL: 
   CREATE_LISTENER_FAILED_4
vmcid: 0x49421000  minor code: 56  completed: No
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at
com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:198)
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at
com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:139)
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at  
com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:460)
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at  
com.ibm.ws.runtime.WsServer.main(WsServer.java:59)
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at  
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[5/11/06 15:41:23:196 CDT] 0000000a SystemErr  R  at  
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
[5/11/06 15:41:23:197 CDT] 0000000a SystemErr  R  at  
sun.reflect.DelegatingMethodAccessorImpl.invoke
    (DelegatingMethodAccessorImpl.java:43)
Change the port number at which the managed node's server is listening. If the deployment manager is listening at port 9101 for ORB_LISTENER_ADDRESS, for example, the server of the managed node should not be listening at port 9101 for its ORB_LISTENER_ADDRESS. To resolve the problem in this example, perform the following steps:
  1. On the administrative console, click Application servers > server_name > Ports > ORB_LISTENER_ADDRESS.
  2. Change the ORB_LISTENER_ADDRESS port number to one that is not used.

Exceptions: database connectivity, loading, or missing class

Never change any WebSphere Application Server variables that are configured as a part of profile creation.

If you modify these values incorrectly in old profile, you might get database connectivity, loading, or other missing class exceptions, such as:

10/25/08 13:22:39:650 GMT+08:00] 0000002e J2CUtilityCla E J2CA0036E: An exception occurred while invoking method setDataSourceProperties on com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl used by resource jdbc/com.ibm.ws.sib/ewps6101.Messaging-BPC.cwfpcCell01.Bus : com.ibm.ws.exception.WsException: DSRA0023E: The DataSource implementation class "com.ibm.db2.jcc.DB2XADataSource" could not be found.DB2,

Derby, and SQL Embedded JDBC drivers are bundled with the WebSphere ESB product installation. If you need to change these drivers to any higher version, you must copy drivers on same location where they exists in the product installation, as follows:
  • Derby: %was.install.root%\derby\lib
  • DB2: %was.install.root%/universalDriver_wbi/lib
  • SQL: %was.install.root%lib
If you need a new JDBC provider and datasource for your application, you can create these resources by selecting a valid jdbcclasspath and setting the WebSphere Application Server variable accordingly. For example, if you need DB2 at cell level which doesn't exist earlier in your installation, you could use the following procedure.
  1. In the administrative console, navigate to: Resources > JDBC > JDBC Providers > DB2 Universal JDBC Driver Provider (XA).
  2. In the Class path box, set the following paths:
    • DB2UNIVERSAL_JDBC_DRIVER_PATH =%was.install.root%/universalDriver_wbi/lib
    • DB2UNIVERSAL_JDBC_DRIVER_NATIVEPATH=""
    If you need your own drivers, set the following path: DB2UNIVERSAL_JDBC_DRIVER_PATH=%myDriverLocation%

Out of memory error

If either the WBIPreUpgrade or WBIPostUpgrade command-line utility fail due to Out of Memory problems, you can increase the heap size to a number that takes into consideration the size and scope of the environment being migrated, as well as what the machine will allow.

For instructions on how to increase the heap size, use the procedure described in Solution 4 of the following technote: Handling certain Out of Memory conditions when migrating an earlier version of WebSphere Application Server to V6.0.2, V6.1, or 7.0.

Profile creation error

While you are using the version 6.2 migration wizard to create a profile when migrating a configuration, you might see the following profile-creation error messages.

profileName: profileName cannot be empty
profilePath: Insufficient disk space

These error messages might be displayed if you enter a profile name that contains an incorrect character such as a space. Rerun the migration wizard, and verify that there are no incorrect characters in the profile name such as a space, quotes, or any other special characters.

Synchronization error

If synchronization fails when you migrate a managed node to version 6.2, the server might not start.

You might receive messages similar to the following when you migrate a managed node to version 6.2:
ADMU0016I: Synchronizing configuration between node and cell.
ADMU0111E: Program exiting with error:
           com.ibm.websphere.management.exception.AdminException: ADMU0005E:
           Error synchronizing repositories
ADMU0211I: Error details may be seen in the file:
           /opt/WebSphere/62AppServer/profiles/AppSrv02/logs/syncNode.log
MIGR0350W: Synchronization with the deployment manager using the SOAP protocol 
 failed.
MIGR0307I: The restoration of the previous WebSphere Application Server 
 environment is complete.
MIGR0271W: Migration completed successfully, with one or more warnings.
These messages indicate the following:
  • Your deployment manager is at a version 6.2 configuration level.
  • The managed node that you are trying to migrate is at a version 6.2 configuration level on the deployment manager's repository (including applications).
  • The managed node itself is not quite complete given that you did not complete the syncNode operation.
Perform the following actions to resolve this issue:
  1. Rerun the syncNode command on the node to synchronize it with the deployment manager.

    See the syncNode command .

  2. Run the GenPluginCfg command.

    See the GenPluginCfg command .

WBIPreUpgrate and WBIPostUpgrade errors

If you encounter a problem when you are migrating from a previous version of WebSphere ESB to version 6.2, check your log files and other available information.

  • If the migration job fails before the WBIPostUpgrade step, re-run the migration job.
  • If the migration job fails in the WBIPostUpgrade step, the configuration of the new version 6.2 server is partially updated so re-create (or restore from backup) the new version 6.2 server before re-running the migration job.
  • Problems occur with a managed (federated) node migration.

    A federated node is the most complex node to migrate because it is essentially two migrations rolled into one. A federated node requires a migration of the node configuration information contained in the deployment manager's master repository as well as the configuration information contained in the federated node. Federated node migration requires an active connection to the deployment manager. If you have security enabled, it is essential that you follow the instructions that were generated when you created the migration jobs. The migration job must be submitted with a WebSphere Administrator's user ID that has been properly configured for obtaining secure connections.

    version 6.x node agents might display as not synchronized or not available when you change the deployment manager node name in a mixed cell during migration to the version 6.2 deployment manager. Version 6.x node agents maintain a link to the version 6.x deployment manager until they are restarted; therefore, they might fail to synchronize with the new deployment manager. The discovery problem, which prevents automatic synchronization, occurs because the node agent is not yet aware of the deployment manager name change that occurred during the migration. If you experience this problem, perform these steps on the node.
    1. Stop the node.
    2. Run the syncNode command.
    3. Restart the node.
  • Job fails during the application-installation phase of migration.

    If you select the option for the migration process to install the enterprise applications that exist in the version 6.0.x or 6.1.x configuration into the new version 6.2 configuration, you might encounter error messages during the application-installation phase of migration.

    The applications that exist in the version 6.0.x or 6.1.x configuration might have incorrect deployment information—usually, invalid XML documents that were not validated sufficiently in previous WebSphere ESB runtimes. The runtime now has an improved application-installation validation process and will fail to install these malformed EAR files. This results in a failure during the application-installation phase of WBIPostUpgrade and produces an "E:" error message.

    If the application installation fails in this way during migration, you can do one of the following:
    • Fix the problems in the version 6.0.x or 6.1.x applications, and then remigrate.
    • Proceed with the migration and ignore these errors.
      1. Restart the migration job in the FINISHUP step to allow the remaining migration functions to be performed.

        Do this by adding the RESTART=FINISHUP parameter to the job card and resubmitting the job.

      2. Later, fix the problems in the applications and then manually install them in the new version 6.2 configuration using the administrative console or an install script.
  • Out of space errors occur.

    The migration logs are located in temporary_directory_location/nnnnn, where temporary_directory_location is the value that you specified when you created the migration jobs (where the default is /tmp/migrate) and nnnnn is a unique number that was generated during the creation of your migration jobs. Normally, the space requirements for the migration logs are small. If you enable tracing, however, the log files can be quite large. The best practice is to enable tracing only after problems have been found. If tracing is required, try to only enable tracing related to the step in the process that is being debugged. This will help to reduce the space requirements.

    Tracing can be enabled when you create the migration jobs or by changing the variables in the migration JCL from disabled to enabled:
    TraceState=enabled
    profileTrace=disabled
    preUpGradeTrace=disabled
    postUpGradeTrace=enabled   

    During migration, a backup copy of your version 6.0.x or 6.1.x configuration is made. This backup becomes the source of the information being migrated. The default backup location is /tmp/migrate/nnnnn. This location can be changed when you create the migration jobs. Depending on the size of the node being migrated, this backup can be quite large. If your temporary space is inadequate, then you will need to relocate this backup.

  • Batch job time is exceeded.

    Each z/OS® installation is different with respect to job classes and time limitations. Make sure you have specified appropriate job classes and timeout values on your job card.

  • Failures occur during server startup after migration.

    Review the instructions that were generated when you created the migration jobs. Verify that the JCL procedures have been copied over correctly to your PROCLIB, the RACF® definitions have been created, the version 6.2 libraries have been authorized, and, if required, your STEPLIB statements to the version 6.2 libraries have been specified. Make sure that the daemon process associated with your cell is at the appropriate level. The daemon process must be at the highest WebSphere ESB for z/OS version level of all servers that it manages within the cell.

    After migrating to a version 6.2 cell that contains or interoperates with version 6.0.x or 6.1.x nodes that are not at version 6.0.1.3 or later, the cluster function might fail. When starting these version 6.0.x or 6.1.x application servers, you might see the following problems:
    • You might see a first failure data capture (FFDC) log that shows a ClassNotFoundException error message. This exception is thrown from the RuleEtiquette.runRules method and looks something like the following example:
      Exception = java.lang.ClassNotFoundException
      Source = com.ibm.ws.cluster.selection.SelectionAdvisor.<init>
      probeid = 133
      Stack Dump = java.lang.ClassNotFoundException: rule.local.server
      at java.net.URLClassLoader.findClass(URLClassLoader.java(Compiled Code))
      at com.ibm.ws.bootstrap.ExtClassLoader.findClass(ExtClassLoader.java:106)
      at java.lang.ClassLoader.loadClass(ClassLoader.java(Compiled Code))
      at java.lang.ClassLoader.loadClass(ClassLoader.java(Compiled Code))
      at java.lang.Class.forName1(Native Method)
      at java.lang.Class.forName(Class.java(Compiled Code))
      at com.ibm.ws.cluster.selection.rule.RuleEtiquette.runRules
      (RuleEtiquette.java:154)
      at com.ibm.ws.cluster.selection.SelectionAdvisor.handleNotification
      (SelectionAdvisor.java:153)
      at com.ibm.websphere.cluster.topography.DescriptionFactory$Notifier.run
      (DescriptionFactory.java:257)
      at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1462)
    • You might see a java.io.IOException that looks something like the following example:
      Exception = java.io.IOException
      Source = com.ibm.ws.cluster.topography.DescriptionManagerA. update probeid
       = 362
      Stack Dump = java.io.IOException
      at com.ibm.ws.cluster.topography.ClusterDescriptionImpl.importFromStream
      (ClusterDescriptionImpl.java:916)
      at com.ibm.ws.cluster.topography.DescriptionManagerA.update
      (DescriptionManagerA.java:360)
      Caused by: java.io.EOFException
      at java.io.DataInputStream.readFully(DataInputStream.java(Compiled Code))
      at java.io.DataInputStream.readUTF(DataInputStream.java(Compiled Code))
      at com.ibm.ws.cluster.topography.KeyRepositoryImpl.importFromStream
      (KeyRepositoryImpl.java:193)

    During migration, version 6.2 cluster information is distributed throughout the cell.version 6.0.x or 6.1.x nodes that are not at version 6.0.1.3 or later fail to read this information. To avoid this problem, upgrade all version 6.0.x or 6.1.x nodes that will be contained in or interoperating with a version 6.2 cell to version 6.0.1.3 or later before migrating your deployment managers to version 6.2.

After migration, carefully review the job output and log files for errors.

Note: WebSphere ESB provides an interactive problem control system (IPCS) verb exit to help you to format information from dumps of WebSphere ESB processes. This verb exit was named CBDATA, which was an alias of the real module name, in version 6.0.x or 6.1.x and earlier. In version 6.2, that alias was removed. In version 6.2 and later, therefore, you must use the real name of this verb exit, BBORDATA, instead of the alias.

If you migrate a node to version 6.2 then discover that you need to revert back to version 6.0.x or 6.1.x, see Rolling back your environment.

If none of these steps solves the problem, see Troubleshooting and support for WebSphere Enterprise Service Bus for additional troubleshooting resources, including information about how to contact IBM® Support.


reference Reference topic

Terms of use | Feedback


Timestamp icon Last updated: 21 June 2010


http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r2mx/topic//com.ibm.websphere.wesb620.zseries.doc/doc/tmig_vtv_troublesht.html
Copyright IBM Corporation 2005, 2010. All Rights Reserved.
This information center is powered by Eclipse technology (http://www.eclipse.org).