Host blocking

Host blocking—a feature of application error handling—prevents Symphony from repeatedly trying to run a service on a host that does not have adequate hardware or software resources. You can configure host blocking to take effect on timeout or exit for each of your services, or when a service throws an exception or sends a specific return code.

About host blocking

When host blocking takes effect, Symphony creates a blocked host list for the application with which the service is associated. A host that appears on the blocked host list can no longer be used by the application until you intentionally unblock the host, or the application is re-registered or disabled and enabled again.

By default, host blocking is enabled for a version mismatch or communication timeout between the session manager and the service instance manager. You can also configure host blocking for a service instance error, a service instance exit, or a service instance method timeout. By default, host blocking is enabled for the following service instance methods:

Method

Event types

Register

  • Timeout

  • Exit

CreateService

  • Timeout

  • Exit

  • Failure exception

  • Fatal exception

SessionEnter

  • Timeout

  • Exit

SessionUpdate

  • Timeout

  • Exit


The following illustrations show the benefits of using the host blocking feature.

Without host blocking (feature disabled)



With host blocking enabled



Host blocking triggers

Host blocking triggers automatically when the session manager version on the management host does not match the service instance manager version on the comput host.

You can configure additional host blocking based on the requirements of your application so that Symphony triggers host blocking for any of the following reasons:
  • A service method times out, exits or crashes, throws an exception, or returns certain control codes.

  • The service instance manager does not communicate with the session manager before the configured timeout period expires (controlled by the startUpTimeout value).

  • The service instance does not communicate with the service instance manager before the configured timeout period expires (controlled by the setting for the Register method actionOnSI attribute).

Slot blocking for Symphony DE

Symphony DE blocks slots—not hosts—under the same conditions that trigger host blocking for a production grid. Symptoms of blocked slots include fewer resources than expected or no resources serving your application, more tasks in the PENDING state, a slower rate of workload completion, and clients that hang. You can check for blocked slots by looking in the ssm.hostname.app_name.log file and searching for WARN or ERROR messages about blocked hosts. If you see a blocked host message, one or more slots might be blocked. You can unblock slots by disabling and then enabling the application or by restarting the DE cluster.

Scope


Applicability

Details

Operating system

  • All host types that are supported by the Symphony system.

Limitations

  • For Symphony DE, only slots are blocked.


Configuration to enable host blocking

Host blocking is enabled in the application profile for each application. You can configure host blocking at the service instance manager level, the service instance level, or both.


Section

Attribute name and syntax

Behavior

SOAM > SIM

blockHostOnTimeout="true"

  • Enables host blocking for the application when the service instance manager times out while trying to communicate with the session manager.

  • Used with the startUpTimeout attribute.

startUpTimeout="seconds"

  • Number of seconds to wait for the service instance manager to communicate with the session manager. This attribute works in conjunction with blockHostOnTimeout.

  • When the process times out, the session manager requests a new host from EGO and tries to start a new service instance manager on the new host.

Service > Control > Method > Timeout

actionOnSI=blockHost

  • When a timeout is reached on the method, terminates the running service instance on this host and does not use this host to start any other service instance for the application.

  • Used with the duration attribute.

  • You can specify the blockHost option for the following methods:
    • Register

    • CreateService

    • SessionEnter

    • SessionUpdate

    • Invoke

    • SessionLeave

Service > Control > Method > Exit

actionOnSI=blockHost

  • When the service instance exits or crashes during execution of the method, the system does not use this host to start any other service instance for the application

  • You can specify the blockHost option for the following methods:
    • Register

    • CreateService

    • SessionEnter

    • SessionUpdate

    • Invoke

    • SessionLeave

Service > Control > Method > Return

actionOnSI=blockHost

  • When the method returns normally or with a specified control code, terminates the running service instance on this host and does not use this host to start any other service instance for the application.

  • You can specify the blockHost option for the following methods:
    • CreateService

    • SessionEnter

    • SessionUpdate

    • Invoke

    • SessionLeave

Service > Control > Method > Exception

actionOnSI=blockHost

  • When the specified exception (failure or fatal exception) occurs, terminates the running service instance on this host and does not use this host to start any other service instance for the application.

  • You can specify the blockHost option for the following methods:
    • CreateService

    • SessionEnter

    • SessionUpdate

    • Invoke

    • SessionLeave


Host blocking behavior

When host blocking is triggered, the system creates a blocked host list for the application. The following example illustrates the host blocking process triggered at the service instance level.

Example of the host blocking process

Configuration to modify host blocking behavior

Not applicable. There are no attributes that change the way that host blocking works other than those attributes configured in the application profile.

Host blocking actions

Actions to monitor

You can monitor host blocking through the Platform Management Console (PMC), the command line, and through the Symphony log files located in the logs directory of SOAM_HOME. You can also trap SNMP events to receive notifications when a service triggers the system to block a host.


User

Action

Description

  • Cluster administrator

From the Platform Management Console:

Symphony Workload > Monitor Workload > application_name > Blocked Hosts

  • Displays a list of blocked hosts for the selected application.

  • Cluster administrator

  • Consumer administrator

From the command line:

egosh alloc view

  • Displays detailed information about all allocations, including the allocation ID, current users, consumer, resource groups, resource requirements, minimum and maximum slots requested, whether it has exclusive use of the host, names of the allocated hosts, and any blocked hosts.


You can find information about host blocking in the following log file:

Log file

Location

Event

Description

Session manager log file: ssm.host_name.app_name.log

Linux/UNIX:

$SOAM_HOME/logs

Windows:

%SOAM_HOME%\logs

SOA_SERVICE_BLOCKED

Error level message that indicates that host blocking has occurred.


Actions to control

Typically, a cluster administrator removes a blocked host when the host has been modified—by means of a software or hardware upgrade, for example—to meet the requirements of the service. A host can be removed from the blocked host lists in one of two ways:
  • Directly from the Platform Management Console (PMC)

  • Indirectly, by disabling and re-enabling the application associated with the blocked host


User

Action

Behavior

  • Cluster administrator

From the Platform Management Console:

Symphony Workload > Monitor Workload > application_name > Blocked Hosts > host_name > Actions > Remove from Blocked Hosts

  • The system removes the host from the blocked host list

  • The application can start a service on the previously blocked host

  • Cluster administrator

From the Platform Management Console:

Symphony Workload > Configure Applications > consumer_name > application_name > Actions > Disable

  • Disables the application, which clears the blocked host list for the disabled application

  • No clients can be served by the disabled application

  • Cluster administrator

  • Consumer administrator

  • Consumer user

From the command line:

soamcontrol app disable application_name

  • Disables the application, which clears the blocked host list for the disabled application

  • No clients can be served by the disabled application

  • For information about how to use the soamcontrol command to disable and enable applications, see the Reference

  • Cluster administrator

  • Consumer administrator

From the Platform Management Console:

  • Symphony Workload > Configure Applications > application_name > Basic Configuration > Save

  • Symphony Workload > Configure Applications > application_name > Advanced Configuration > Save

  • The system first disables and then re-registers the application, which clears the blocked host list for the modified application


For Symphony DE, you can unblock slots by disabling and then enabling the application, or by restarting the DE cluster.

User

Action

Behavior

  • Developer

From the command line:

soamcontrol app disable application_name

  • Disables the application, which unblocks slots for the disabled application

  • No clients can be served by the disabled application

soamcontrol app enable application_name

  • Enables the application, which can start services on any previously blocked slot

  • Developer

Windows:

  • Right-click on the Symphony DE tray icon and choose Stop Symphony DE on all hosts. Once the DE cluster shuts down, right-click on the Symphony DE tray icon and choose Start Symphony DE on all hosts.

Linux/UNIX:

  • soamshutdown

  • soamstartup

  • Shuts down and then restarts Symphony DE

  • Unblocks slots for all applications running on the DE cluster


Actions to display configuration


User

Command

Behavior

  • Cluster administrator

  • Consumer administrator

From the Platform Management Console:

  • Symphony Workload > Configure Applications > application_name > Basic Configuration

  • Symphony Workload > Configure Applications > application_name > Advanced Configuration

  • Displays application profile settings for the selected application

  • Cluster administrator

  • Consumer administrator

  • Consumer user

From the command line:

  • soamview app app_name -p

  • Displays application profile settings for the selected application


You can also view an application profile using an XML editor.