Monitoring eXtreme Scale applications with CA Wily Introscope

CA Wily Introscope is a third-party management product that you can use to detect and diagnose performance problems in enterprise application environments. eXtreme Scale includes details on configuring CA Wily Introscope to introspect select portions of the eXtreme Scale run time to quickly view and validate eXtreme Scale applications. CA Wily Introscope works effectively for both stand-alone and WebSphere® Application Server deployments.

Overview

To monitor eXtreme Scale applications with CA Wily Introscope, you must put settings into the ProbeBuilderDirective (PBD) files that give you access to the monitoring information for eXtreme Scale.
Attention: The instrumentation points for Introscope might change with each fix pack or release. When you install a new fix pack or release, check the documentation for any changes in the instrumentation points.
You can configure CA Wily Introscope ProbeBuilderDirective (PBD) files to monitor your eXtreme Scale applications. CA Wily Introscope is an application management product with which you can proactively detect, triage and diagnose performance problems in your complex, composite and Web application environments.

PBD file settings for monitoring the catalog service

You can use one or more of the following settings in your PBD file to monitor the catalog service.
TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl 
		changeDefinedCompleted 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl 
		viewChangeCompleted 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl 
		viewAboutToChange 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeat 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeatCluster 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeatCurrentLeader 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeatDeadServer 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeatNewLeader 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeatNewServer 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.catalog.placement.
		PlacementServiceImpl 
importRouteInfo BlamePointTracerDifferentMethods 
		"OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.catalog.placement.
		PlacementServiceImpl heartbeat 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.catalog.placement.
		PlacementServiceImpl joinPlacementGroup 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
		TraceOneMethodOfClass: 
com.ibm.ws.objectgrid.catalog.placement.PlacementServiceImpl 
		classifyServer 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.catalog.placement.
		BalanceGridEventListener shardActivated 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.catalog.placement.
		BalanceGridEventListener shardDeactivate 
BlamePointTracerDifferentMethods "OGcatalog|{classname}|{method}"
Classes for monitoring the catalog service
HAControllerImpl
The HAControllerImpl class handles core group life cycle and feedback events. You can monitor this class to get an indication of the core group structure and changes.
ServerAgent
The ServerAgent class is responsible for communicating core group events with the catalog service. You can monitor the various heartbeat calls to spot major events.
PlacementServiceImpl
The PlacementServiceImpl class coordinates the containers. You can use the methods on this class to monitor server join and placement events.
BalanceGridEventListener
The BalanceGridEventListener class controls the catalog leadership. You can monitor this class to get an indication of which catalog service is currently acting as the leader.

PBD file settings for monitoring the containers

You can use one or more of the following settings in your PBD file to monitor the containers.
TraceOneMethodOfClass: com.ibm.ws.objectgrid.ShardImpl processMessage 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.plugins.
		CommittedLogSequenceListenerProxy applyCommitted 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.plugins.
		CommittedLogSequenceListenerProxy sendApplyCommitted 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.map.BaseMap evictMapEntries 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.checkpoint.
		CheckpointMapImpl$CheckpointIterator activateListener 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl 
		changeDefinedCompleted 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl 
		viewChangeCompleted 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.hamanager.HAControllerImpl 
		viewAboutToChange 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		batchProcess 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeat 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeatCluster 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeatCurrentLeader 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeatDeadServer 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeatNewLeader 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.container.ServerAgent 
		heartbeatNewServer 
BlamePointTracerDifferentMethods "OGcontainer|{classname}|{method}"
Classes for monitoring the containers
ShardImpl
The ShardImpl class has the processMessage method. The processMessage method is the method for client requests. With this method, you can get server side response time and request counts. By watching the counts across all the servers and monitoring heap utilization, you can determine if the grid is balanced.
CheckpointIterator
The CheckpointIterator class has the activateListener method call which puts primaries into peer mode. When the primaries are put into peer mode, the replica is up to date with the primary after the method completes. When a replica is regenerating from a full primary, this operation can take an extended period of time. The system is not fully recovered until this operation completes, so you can use this class to monitor the progress of the operation.
CommittedLogSequenceListenerProxy
The CommittedLogSequenceListenerProxy class has two methods of interest. The applyCommitted method runs for every transaction and the sendApplyCommitted runs as the replica is pulling information. The ratio of how often these two methods run can give you some indication of how well the replica is able to keep up with the primary.

PBD file settings for monitoring the clients

You can use one or more of the following settings in your PBD file to monitor the clients.
TraceOneMethodOfClass: com.ibm.ws.objectgrid.client.ORBClientCoreMessageHandler 
	sendMessage 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.corba.cluster.ClusterStore 
		bootstrap 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.corba.cluster.ClusterStore 
		epochChangeBootstrap 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.map.BaseMap evictMapEntries 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.cluster.orb.routing.
		SelectionServiceImpl routeFailed 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.cluster.orb.routing.
		SelectionServiceImpl routeFailed 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.SessionImpl getMap 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TraceOneMethodOfClass: com.ibm.ws.objectgrid.ObjectGridImpl getSession 
BlamePointTracerDifferentMethods "OGclient|{classname}|{method}" 
TurnOn: ObjectMap 
SetFlag: ObjectMap 
IdentifyClassAs: com.ibm.ws.objectgrid.ObjectMapImpl ObjectMap 
TraceComplexMethodsifFlagged: ObjectMap BlamePointTracerDifferentMethods 
"OGclient|{classname}|{method}"
Classes for monitoring the clients
ORBClientCoreMessageHandler
The ORBClientCoreMessageHandler class is responsible for sending application requests to the containers. You can monitor the sendMessage method for client response time and number of requests.
ClusterStore
The ClusterStore class holds the routing information on the client side.
BaseMap
The BaseMap class has the evictMapEntries method that is called when the evictor wants to remove entries from the map.
SelectionServiceImpl
The SelectionServiceImpl class makes the routing decisions. If the client is making failover decisions, you can use this class to see the actions that are completed from the decisions.
ObjectGridImpl
The ObjectGridImpl class has the getSession method that you can monitor to see the number of requests to this method.