MustGather: Enterprise JavaBeans™ (EJB™) workload management

Technote (FAQ)
Problem
MustGather for problems with the WebSphere® Application Server enterprise bean workload management component. Gathering this information before calling IBM support helps familiarize you with the troubleshooting process and saves you time.
Solution
If you have already contacted support, continue to the component-specific MustGather information. Otherwise, click the following link: MustGather: Read First for all WebSphere Application Server Products.

EJB workload management- specific MustGather information

Two types of workload management exist for WebSphere Application Server:
  1. WebSphere plug-in workload management.
  2. EJB workload management.

The following content provides you with the MustGather information for EJB workload management and shows you how to enable tracing on workload management (WLM).

Collect the following information when you open a problem management record (PMR) for EJB workload management problems:

  1. Domain configuration detail
  2. WLM client information
  3. WLM and Object Request Broker (ORB) traces

    A. Domain configuration detail

    For WebSphere Application Server V3.5 and V4.0 (all releases):

    1. The admin.config file located in your $WAS_HOME/bin directory.
    2. The product.xml file located in your $WAS_HOME/properties/com/ibm/websphere directory.
    3. The XML export of your repository configuration located in your specified directory.

    Notes:
    1. If your problem involves a horizontal-cloned environment, send the admin.config, tracefile, nanny.trace, and product.xml file from each node.
    2. If you are using a stand-alone Java™ 2 Platform, Enterprise Edition (J2EE™) client, also collect your client-side product.xml file.

    For WebSphere Application Server V5.0 (all releases):

    1. Run the versionInfo command and collect all of the .txt files created. Run this command from each Base and Network Deployment (ND) instance:

      $WAS_Base_HOME/bin/versionInfo.sh/bat > versionof_Base_node1.txt

      $WAS_
      ND_HOME/bin/versionInfo.sh/bat > versionof_ND_node.txt

    2. Run the genVersionReport command and collect all of the .txt files created. Run this command from each Base and ND instance:

      $WAS_Base_HOME/bin/genVersionReport.bat/sh -format html -file myBaseVersionReport.html -efixes -ptfs -componentDetail

      $WAS_
      ND_HOME/bin/genVersionReport.bat/sh -format html -file myNDVersionReport.html -efixes -ptfs -componentDetail

    3. Collect the entire configuration directories for each Base and ND instance. Then create a separate .zip or .tar file for each instance.

      $WAS_Base_HOME/config

      $WAS_ND_HOME/config
        Note: If you are using a stand-alone J2EE client, collect client-side versionInfo as well.

      B. WLM client information

      Collect the following information:
      1. What .jar files are on your client machine?
      2. What are the locations of your .jar files?
      3. Did you install WebSphere Client or WebSphere Application Server on the client side?
      4. How are you launching the client? Provide step-by-step instructions.
      5. Do you launch your client with the launchClient utility from the $WAS_HOME/bin directory?
        1. If yes, then place the launchClient command in the startup file (.bat or .sh). If it already exists in the startup file, collect a copy of your startup file.
      6. Is the customer using a startup file (.bat or .sh)?
        1. If yes, collect your startup .bat file.
      7. Collect the environment settings. From where the client is running, run a set > environment.log command from the command line prompt.
      8. If there appears to be a problem with the wrong classes being loaded, or the wrong Java 2 SDK (J2SDK™)/Java Development Kit (JDK™):
        1. Try to unset the PATH and CLASSPATH before running the launchClient command. If this action solves your problem, correct the system PATH and CLASSPATH to use J2SDK/JDK and the other classes installed by WebSphere Application Server .
        2. If the WebSphere Application Server is running a different version of WebSphere Application Server than the client, use the fix pack installation script to upgrade your client and server. Do not manually copy files from any WebSphere Application Server directories.
        3. To understand which classes your client is loading, review the LaunchClient.bat and setupCmdLine.bat files. Also, the setupClient.bat is used for thin clients.
      C. WLM and ORB trace


      Server trace:

      1. Administrative server WLM, ORB and naming trace.

        AND

      2. Request of the client.
        Client trace:
        1. WLM, ORB and naming trace from the Application Server that makes the request

          OR

        2. Stand-alone enterprise bean /Java/J2EE
          Note: If you do not know how to set these traces, see Appendix A .

          Appendix A:

          Level of trace that you need to set in V3.5 (all releases)
          Administrative server trace:
          Gather the following information from all the nodes that participate in EJB workload management.

          1. Make sure that the default server and administrative server are not running.
          2. Make a backup copy of your admin.config file.
          3. Add the following lines to your admin.config file:

            com.ibm.CORBA.Debug=true
            com.ibm.CORBA.CommTrace=true

            #Note:Following trace string must be entered as one line and
            there is ":" between trace strings
            com.ibm.ejs.sm.adminServer.traceString=
            com.ibm.ejs.wlm.*=all=enabled:com.ibm.ws.naming.*=all=enabled:
            ORBRas=all=enabled
            #
            com.ibm.ejs.sm.adminServer.traceOutput=<WAS_Home>/logs/admin.trace
          4. Save your admin.config file.
          5. Recycle your administrative server.

          Application Server and clone trace:
          Gather the following information from all of your clones that participate in EJB workload management.
          1. Make sure that your administrative server is running and then start your administrative console.
          2. Enter the following line in the Command line arguments field of the Application Server:

            -Dcom.ibm.CORBA.Debug=true -Dcom.ibm.CORBA.CommTrace=true

          3. Click Apply and wait for the completion message.
          4. Enter the following information in the Trace specification field on the Advance tab of your Application Server:

            com.ibm.ejs.wlm.*=all=enabled:com.ibm.ws.naming.*=all=enabled:
            ORBRas=all=enabled

          5. Enter the following line in the Trace output file field on the Advance tab of your Application Server:

            $WAS_HOME\logs\MyClone.trace

          6. Click Apply and wait for the completion message.
          7. Recycle your Application Server to enable tracing.

          Client trace:
          1. For a stand-alone Java client
            1. In your $WAS_HOME\properties\TraceSettings.properties file, change the line at the bottom of your file:

              From:com.ibm.ejs.ras.*=all=enabled
              To: com.ibm.ejs.wlm.*=all=enabled:com.ibm.ws.naming.*=all=
              enabled:ORBRas=all=enabled

              Note: The previous trace string must be entered as one line with a colon (:) between trace strings.
            2. Start the client program with '-D' arguments to specify the trace settings:
              Example:
              java

              -Dcom.ibm.CORBA.Debug=true
              -Dcom.ibm.CORBA.CommTrace=true
              -DtraceSettingsFile=TraceSettings.properties<clientProgram>

          2. Application Server as a client
            Follow the same explanation in the procedure, ApplicationServer and clone trace.


          Level of trace that you need to set in V4.0 (all releases)
          Administrative server trace:
          1. Make sure that your Application Server and administrative server are not running.
          2. Make a backup copy of theadmin.config file located in the <WAS_HOME>/bin directory.
          3. Edit and append the following lines to your admin.config file:

            com.ibm.CORBA.Debug=true
            com.ibm.CORBA.CommTrace=true
          4. Edit the # Trace settings to look like the following example:

            #Note: Following trace string must be entered as one line and there is a colon (:) between trace strings
            com.ibm.ejs.sm.adminServer.traceString=com.ibm.ws.wlm.*=all=enabled:com.ibm.ws.naming.*=all=enabled:ORBRas=all=enabled
            #
            com.ibm.ejs.sm.adminServer.traceOutput=<WAS_Home>/logs/admin.trace
          5. Save your admin.config file.
          6. Recycle your administrative server to enable tracing.
            Application Server and clone trace:
            Collect the following information for all of your clones that participate in EJB workload management.
            1. Make sure that your administrative server is running and then start your administrative console.
            2. Highlight your Application Server, then select the Services tab.
            3. Select OBJECT REQUEST BROKER, then click Edit Properties.
            4. Select the Enable ORB Tracing check box.
            5. Click OK.
            6. Click Apply.
            7. Click the Services tab, then click Trace Service >Edit Properties. Place the following information in the Trace Specification field:

              com.ibm.ws.wlm.*=all=enabled:com.ibm.ws.naming.*=all=enabled:ORBRas=all=enabled

              #Note: The previous trace string must be entered as one line and there is colon (:) between trace strings.
            8. Enter the fully qualified name in the Trace Output field (that is, C:\WebSphere\AppServer\logs\MyAppServer.trace).
            9. Click Apply and wait for a completion message.
            10. Recycle the Application Server to enable tracing.
              Client trace:
              1. For a stand-alone Java client
                1. In the $WAS_HOME\properties\TraceSettings.properties file, change the line at the bottom of the file:

                  From: com.ibm.ejs.ras.*=all=enabled
                  To:com.ibm.ws.wlm.*=all=enabled:com.ibm.ws.naming.*=all=enabled:
                  ORBRas=all=enabled

                  #Note: The previous trace string must be entered as one line and there is a colon (:) between trace strings.
                2. Start your client program with '-D' arguments to specify the trace settings:
                  Example:
                  java
                  -Dcom.ibm.CORBA.Debug=true
                  -Dcom.ibm.CORBA.CommTrace=true
                  -DtraceSettingsFile=TraceSettings.properties<clientProgram>
              2. For a J2EE client
                Use the following steps to collect a J2EE client trace:
                1. Copy <ClientSide_$WAS_HOME>\bin\launchClient to a backup file.
                2. Edit the following two trace parameters to the following "$JAVA_HOME/bin/java" statement:
                  -Dcom.ibm.CORBA.Debug=true
                  -Dcom.ibm.CORBA.CommTrace=true

                  Example:
                  %JAVA_HOME%\bin\java %CONSOLE_ENCODING% %CLIENTSAS%-Dcom.ibm.CORBA.Debug=true
                  -Dcom.ibm.CORBA.CommTrace=true
                  -Dserver.root=%WAS_HOME% -Dws.ext.dirs=%WAS_EXT_DIRS%
                  -Dcom.ibm.CORBA.BootstrapHost=%COMPUTERNAME%
                  -Djava.naming.factory.initial=%NAMING_FACTORY% -classpath
                  %WAS_CLASSPATH% com.ibm.ws.bootstrap.WSLauncher
                  com.ibm.websphere.client.applicationclient.launchClient %*
                3. Start the launchClient with trace:

                  CCtrace=com.ibm.ws.wlm.*=all=enabled:com.ibm.ws.naming.*=all=enabled:
                  ORBRas=all=enabled

                  Example:
                  <ClientSide_$WAS_HOME>\bin> launchClient<customer.ear> -CCtrace=com.ibm.ws.wlm.*=all=enabled:com.ibm.ws.naming.*=all=enabled:ORBRas=all=enabled-CCtracefile=ClientSideTrace.txt
              3. Application server as a client

                Follow the explanation in the procedure, ApplicationServer and clone trace.


              Level of trace that you need to set for V5.0 (all releases)
              In all releases of WebSphere Application Server V5.0, you can perform the naming lookup at different levels. Based on the lookup you might need to set server-side traces.

              Using naminghttp://publib.boulder.ibm.com/infocenter/wasinfo/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/tnam_ovr.html

              Lookup names support in deployment descriptors and thin clientshttp://publib.boulder.ibm.com/infocenter/wasinfo/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/rnam_names.html

              Server trace:
              The following process applies to your dmgr, node agent and any application server.

              Part A: Setting the ORB Trace flag

              Through your administrative console
              1. Go to the Configuration page for the server to trace.

                For Application Server select
                Servers-->Application Servers-->MyServer-->Configuration Tab

                For NodeAgents select
                System Administration-->Node Agents-->MyNodeAgent-->Configuration Tab

                For DeploymentManager select
                System Administration -->DeploymentManager-->Configuration Tab
              2. Click ORB Service under Additional Properties.
              3. Select the check box for ORB tracing under General Properties.


              Through the server.xml file
              In a Network Deployment environment, always edit the file in the Network Deployment $WAS_HOME directory. Do not edit the file in the Base $WAS_HOME directory.
              1. Copy the server.xml file from the server that you are tracing to a backup file.

                Example: For a single-server installation, the file is located at:
                $WAS_HOME/config/cells/<cellName>/nodes/<nodeName>/servers/server1/server.xml
              2. Edit the server.xml file.
              3. Search for "orb:ObjectRequestBroker". The search points you to the ORB service section.
              4. Edit (or add) the 'commTraceEnabled' attribute and set it to true.

                Example:commTraceEnabled="true"
                <services xmi:type="orb:ObjectRequestBroker"xmi:id="ObjectRequestBroker_1" enable="true"
                requestTimeout="180" requestRetriesCount="1"
                requestRetriesDelay="0" connectionCacheMaximum="240"
                connectionCacheMinimum="100"
                commTraceEnabled="true"locateRequestTimeout="180" forceTunnel="never"
                noLocalCopies="false">
              5. Save your file.


              Part B: Setting ORB, WLM and naming trace strings

              Through your administrative console
              1. Start your Web console.
              2. Expand the following view:
                -Troubleshooting
                Logs and Trace
                Configuration Problems
                PMI Request Metrics
              3. Select Logs and Trace.
              4. Select the server you want to trace (In a Network Deployment environment you need to set the traces for dmgr, nodeagent and application server).
              5. Select Diagnostic Trace.
              6. Click the Configuration tab.
              7. Enter the following information in Trace specification:

                Naming=all=enabled:WLM=all=enabled:ORBRas=all=enabled
              8. Make sure to select File (radio button) in Trace Output and accept the default values.
              9. Click Apply.
              10. Save your changes and make sure to "Synchronize changes with Nodes".


              Through the server.xml file
              In a Network Deployment environment, always edit the file in your Network Deployment $WAS_HOME directory. Do not edit the file in your Base $WAS_HOME directory.
              1. Copy theserver.xml file from the server that you are tracing to a backup file.

                Example: For a single-server installation, the file is located at:
                $WAS_HOME/config/cells/<cellName>/nodes/<nodeName>/servers/server1/server.xml
              2. Edit theserver.xml file.
              3. Search for "traceservice:TraceService". The search points you to the Trace service section.
              4. Edit the 'startupTraceSpecification' attribute and set it to:

                Naming=all=enabled:WLM=all=enabled:ORBRas=all=enabled

                Example:
                <services xmi:type="traceservice:TraceService"
                xmi:id="TraceService_1" enable="true" startupTraceSpecification=
                "Naming=all=enabled:WLM=all=enabled:ORBRas=all=enabledtraceOutputType="SPECIFIED_FILE" traceFormat="BASIC"
                memoryBufferSize="8">
              5. Save your file.


              Client Trace:

              Multiple clients are supported; review the following information at:
              http://publib.boulder.ibm.com/infocenter/wasinfo/index.jsp
              Select your edition of WebSphere Application Server, then select
              - All topics by feature
              - Applications
              - Client modules
              - Using application clients


              1. For a stand-alone Java client
                1. In the file $WAS_HOME\properties\TraceSettings.properties file, change the line at the bottom of the file:

                  From: com.ibm.ejs.ras.*=all=enabled
                  To:com.ibm.ws.wlm.*=all=enabled:com.ibm.ws.naming.*=all=enabled:
                  ORBRas=all=enabled

                  #Note: Enter the previous trace string as one line, with a colon (:) between trace strings.
                2. Start your client program with '-D' arguments to specify the trace settings:

                  Example:
                  java
                  -Dcom.ibm.CORBA.Debug=true
                  -Dcom.ibm.CORBA.CommTrace=true
                  -DtraceSettingsFile=TraceSettings.properties<clientProgram>
              2. For a J2EE client
                Use the following steps to collect a J2EE client trace:
                1. Copy <ClientSide_$WAS_HOME>\bin\launchClient to a backup file.
                2. Edit the following two trace parameters to the following "$JAVA_HOME/bin/java" statement:
                  -Dcom.ibm.CORBA.Debug=true
                  -Dcom.ibm.CORBA.CommTrace=true

                  Example:
                  "%JAVA_HOME%\bin\java" %CONSOLE_ENCODING% "%CLIENTSAS%"
                  "%CLIENTSOAP%" %ORB_RAS_MGR% -Dwas.install.root="%WAS_HOME%"
                  -Dws.ext.dirs="%WAS_EXT_DIRS%" %JAAS_LOGIN_CONFIG%-Dcom.ibm.CORBA.Debug=true
                  -Dcom.ibm.CORBA.CommTrace=true
                  -Dcom.ibm.CORBA.BootstrapHost=%DEFAULTSERVERNAME%
                  -Dcom.ibm.CORBA.BootstrapPort=%SERVERPORTNUMBER%
                  -Djava.naming.factory.initial=%NAMING_FACTORY% %USER_INSTALL_PROP%
                  -Xbootclasspath/p:"%WAS_BOOTCLASSPATH%" -classpath
                  "%WAS_CLASSPATH%" com.ibm.ws.bootstrap.WSLauncher
                  com.ibm.websphere.client.applicationclient.launchClient %*
                3. Start your launchClient with trace:

                  CCtrace=com.ibm.ws.wlm.*=all=enabled:com.ibm.ws.naming.*=all=enabled:
                  ORBRas=all=enabled

                  Example:
                  <ClientSide_$WAS_HOME>\bin> launchClient<customer.ear> -CCtrace=com.ibm.ws.wlm.*=all=enabled:com.ibm.ws.naming.*=all=enabled:ORBRas=all=enabled-CCtracefile=ClientSideTrace.txt

                  LaunchClient toolhttp://publib.boulder.ibm.com/infocenter/wasinfo/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/rcli_javacmd.html

                  Application client troubleshooting tipshttp://publib.boulder.ibm.com/infocenter/wasinfo/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/rcli_troubleshoot.html

              3. Application server as a client

                Follow the explanation from the procedure, Server trace.


              For a listing of all Technotes, downloads, and educational materials specific to the Workload Management (WLM) component, search the WebSphere Application Server support site.











              Document Information

              Product categories: Software, Application Servers, Distributed Application & Web Servers, WebSphere Application Server, Workload Management (WLM)
              Operating system(s): Multi-Platform
              Software version: 3.5.x, 4.0.x, 5.0.x
              Software edition: Advanced
              Reference #: 1052165
              IBM Group: Software Group
              Modified date: 2004-01-29