Prioritizing sessions for time-critical workload

The session manager provides a priority scheduling policy where all resources are assigned to the highest priority session before lower priority sessions. If the highest priority session cannot make use of all of the resources, any unused resources are assigned to the next highest priority session. This cycle repeats to the next highest priority session, and so on. If there are still some slots left after all the sessions are satisfied, these slots are not assigned to any session.

If two sessions have the same priority, the session that is created earlier is treated as though it has higher priority.

Note:

A client application can update the session priority via a Symphony API after the session is created. The priority can also be updated by an administrator using the CLI or PMC.

Tasks within a session are dispatched in first come, first served (FCFS) order.

The following example demonstrates the behavior of the Priority Scheduling policy for sessions with different priorities. The higher priority session gets resources before lower priority sessions.

Preconditions:

  • The cluster only has one slot.

  • t1: a session starts with default session type, priority set to 1, and 100000 short tasks (one second task runtime).

  • t2: a second session starts with default session type, priority set to 10, and 10 long tasks (10 seconds task runtime) while the first session is still running.

  • t3: a third session starts with default session type, priority set to 100, and one long task (10 seconds tasks runtime) while the second session is still running.

Post conditions:

  1. When the first session is created and has outstanding workload at t1, the one resource is assigned to the first session. The first session continues to use it to run its tasks until a higher priority session requires the resource.

  2. When the second session is created and has outstanding workload at t2, the resource is assigned to the second session (since it has higher priority) after the current running task from the first session is done.

  3. When the third session is created and has outstanding workload at t3, the resource is assigned to the third session (since it has higher priority) after the current running task from the second session is done.

  4. The third session uses the slot to finish all of its tasks. Afterward, the second session receives the slot, since it now has the highest priority, and continues to use it until it finishes all of its tasks. Finally, the first session receives the slot and uses it to finish all of its tasks.

Preemption

If preemption is configured, Symphony takes back resources from lower priority sessions and gives them to higher priority sessions by terminating the currently running tasks of the lower priority sessions. If preemption is not configured, the highest priority session receives resources only after the currently running tasks finish in lower priority sessions.

You can specify session preemption via the PMC when you configure the application. Alternatively, a client application can specify session preemption via a Symphony API when the session is created. Session preemption cannot be updated once the session is created.

You can specify the preemptive option for each SessionType in the application profile. If not specified, the default is the session is not preemptive.

  • Any high priority session that is configured as preemptive preempts lower priority session(s) (from lowest to highest priority) if the high priority session has any unsatisfied workload. Note that the service instances assigned to lower priority sessions are terminated and restarted before they can be assigned to the next session.

  • Resources released as a result of preemption are distributed to the highest priority session before lower priority sessions. This could mean that the session that triggers preemption may not get the resources in the case that there are even higher priority sessions with unsatisfied workload.

The following example demonstrates the behavior of the R_PriorityScheduling policy when there is a mixture of preemptive and non-preemptive sessions with different priorities in the system.

Preconditions:

  • The cluster has five slots.

  • Sessions:


    Session

    Priority

    Preemptive flag

    First session

    100

    false

    Second session

    10

    true

    Third session

    1

    false


  • The application is already running.

  • There is no current workload for the first and second sessions. At this point, the third session has all five resources since the higher priority sessions do not need them.

The client application submits one task to the first session and four tasks to the second session before the next scheduling cycle.

Post conditions:

  1. At the next scheduling cycle:

    • The policy will not preempt on behalf of the first session, since the first session is configured as non-preemptive.

    • The policy will preempt four running tasks from the third session, to satisfy the second session's four pending tasks.

  2. After preemption occurs, one resource will go to the first session and three resources will go to the second session.

    Note:

    Even though the first session was not the trigger for preemption, it will be assigned the resource first, according to policy. It will benefit from the second session's preemptive behavior.

    Although the second session is not the recipient of those resources, the preemptive behavior is still helping the second session to get ahead. It allows higher priority sessions to get resources faster, which means that the higher priority sessions will finish using the resources faster, and the preemptive session's turn to receive those resources will come faster as well.

  3. At the next scheduling cycle, the policy will preempt the last running task from the third session to satisfy the second session, which still has unmet demand.

  4. After preemption occurs, the last resource will go to the second session.

  5. The third session will receive resources when the first and second session no longer need them.

Configure your application profile for priority scheduling

With this scheduling policy, you assign a priority to a session type relative to the priority of other sessions types; this establishes a rank of importance among the session types. The priority scheduling policy is configured in the application profile.

Note:

If you are editing the application profile outside the Platform Management Console, in the Consumer section, add the parameter policy="R_PriorityScheduling". In the session types section, add the parameter priority and register the application with the soamreg command.

  1. In the Platform Management Console, click Symphony Workload > Configure Applications.

    The Applications page displays.

  2. Select the application you want to modify.

    The Application profile page displays.

  3. Select SSM scheduling policy to expand it, then under Policy Name, select R_PriorityScheduling.
  4. Select Session Type Definition to expand it.
  5. Click Add to add a new session type or modify an existing session type.
  6. Define the Priority.
  7. Choose whether sessions of this type are preemptive, or not.
  8. Add new session type definitions or modify existing ones with different priorities, if necessary.
  9. Click Save to apply your changes.