Knowledge Center Contents Previous Next Index |
Advance Reservation
Contents
Understanding Advance Reservations
Advance reservations ensure access to specific hosts during specified times. During the time that an advance reservation is active only users or groups associated with the reservation have access to start new jobs on the reserved hosts.
Only LSF administrators or root can create or delete advance reservations. Any LSF user can view existing advance reservations.
Each reservation consists of the number of job slots to reserve, a list of hosts for the reservation, a start time, an end time, and an owner. You can also specify a resource requirement string instead of or in addition to a list of hosts.
Active reservations
When a reservation becomes active, LSF attempts to run all jobs associated with the reservation. By default jobs running before the reservation became active continue to run when the reservation becomes active. When a job associated with the reservation is pending because not enough job slots are available, LSF suspends
all
jobs not associated with the reservation that are running on the required hosts.During the time the reservation is active, only users or groups associated with the reservation have access to start new jobs on the reserved hosts. The reservation is active only within the time frame specified, and any given host may have several reservations in place, some of which may be active at the same time.
Jobs are suspended only if advance reservation jobs require the slots. Jobs using a reservation are subject to all job resource usage limits, but any resources freed by suspending non-advance reservation jobs are available for advance reservation jobs to use.
Closed and open reservations
Reservations are typically
closed
. When a closed reservation expires, LSF kills jobs running in the reservation and allows any jobs suspended when the reservation became active to run.
Open
advance reservations allow jobs to run even after the associated reservation expires. A job in the open advance reservation is only be treated as an advance reservation job during the reservation window, after which it becomes a normal job. This prevents the job from being killed and makes sure that LSF does not prevent any previously suspended jobs from running or interfere with any existing scheduling policies.Jobs running in a one-time open reservation are detached from the reservation and suspended when the reservation expires, allowing them to be scheduled as regular jobs. Jobs submitted before the reservation became active are still suspended when the reservation becomes active. These are only resumed after the open reservation jobs finish.
Jobs running is a closed
recurring
reservation are killed when the reservation expires.Jobs running in an open
recurring
reservation are suspended when the reservation expires, and remain pending until the reservation becomes active again to resume.If a non-advance reservation job is submitted while the open reservation is active, it remains pending until the reservation expires. Any advance reservation jobs that were suspended and became normal jobs when the reservation expired are resumed first before dispatching the non-advance reservation job submitted while the reservation was active.
Job scheduling in advance reservations
LSF treats advance reservation like other deadlines, such as dispatch windows or run windows; LSF does not schedule jobs that are likely to be suspended when a reservation becomes active. Jobs referencing the reservation are killed when the reservation expires.
note:
If IGNORE_DEADLINE=Y, there is no effect on advance reservations. Jobs are always prevented from starting if there is a chance that they could encounter an advance reservation.System reservations
Reservations can also be created for system maintenance. If a system reservation is active, no other jobs can use the reserved hosts, and LSF does not dispatch jobs to the specified hosts while the reservation is active.
Configure Advance Reservation
Enable advance reservation
- To enable advance reservation in your cluster, make sure the advance reservation scheduling plugin
schmod_advrsv
is configured inlsb.modules
.Begin PluginModule SCH_PLUGIN RB_PLUGIN SCH_DISABLE_PHASES schmod_default () () schmod_advrsv () () End PluginModuleAllow users to create advance reservations
By default, only LSF administrators or root can add or delete advance reservations. To allow other users to use
brsvadd
to create advance reservations andbrsvdel
to delete advance reservations, you need to configure advance reservation user policies.
- Use the ResourceReservation section of
lsb.resources
to configure advance reservation policies for users.A ResourceReservation section specifies:
- Users or user groups that can create reservations
- Hosts that can be used for the reservation
- Time window when reservations can be created.
Each advance reservation policy is defined in a separate ResourceReservation section, so it is normal to have multiple ResourceReservation sections in
lsb.resources
.Only
user1
anduser2
can make advance reservations onhostA
andhostB
. The reservation time window is between 8:00 a.m. and 6:00 p.m. every day:Begin ResourceReservation NAME = dayPolicy USERS = user1 user2 # optional HOSTS = hostA hostB # optional TIME_WINDOW = 8:00-18:00 # weekly recurring reservation End ResourceReservation
user1
can add the following reservation for useruser2
to use onhostA
every Friday between 9:00 a.m. and 11:00 a.m.:brsvadd -m "hostA" -n 1 -u "user2" -t "5:9:0-5:11:0"
Reservation "user2#2" is createdUsers can only delete reservations they created themselves. In the example, only user
user1
can delete the reservation;user2
cannot. Administrators can delete any reservations created by users.All users in user group
Begin ResourceReservation NAME = nightPolicy USERS = ugroup1 ~user1 HOSTS = hgroup1 ~hostB TIME_WINDOW = 20:00-8:00 End ResourceReservationugroup1
exceptuser1
can make advance reservations on any host in hgroup1, excepthostB
, between 10:00 p.m. and 6:00 a.m. every day
important:
The not operator (~) does not exclude LSF administrators from the policy.For example:
- Define a policy for user:
user1
:Policy Name: dayPolicy Users: user1 Hosts: hostA Time Window: 8:00-18:00- User
user1
creates a reservation matching the policy ( the creator isuser1
, the user isuser2
):brsvadd -n 1 -m hostA -u user2 -b 10:00 -e 12:00 user2#0 is created.- User
user1
modifies the policy to remove user1 from the users list:Policy Name: dayPolicy Users: user3 Hosts: hostA Time Window: 8:00-18:00- As the creator,
user1
can modify the reservation with thebrsvmod
optionsrmhost
,-u
,-o
,-on
, and-d
, butuser1
cannot add hosts or modify the time window of the reservation.USER_ADVANCE_RESERVATION is obsolete (lsb.params)
USER_ADVANCE_RESERVATION in
lsb.params
is obsolete in LSF Version 7. Use the ResourceReservation section configuration inlsb.resources
to configure advance reservation policies for your cluster.Using Advance Reservation
Advance reservation commands
Use the following commands to work with advance reservations:
brsvadd
Add a reservation
brsvdel
Delete a reservation
brsvmod
Modify a reservation
brsvs
View reservations
Add reservations
note:
By default, only LSF administrators or root can add or delete advance reservations.
- Run
brsvadd
to create new advance reservations.You must specify the following for the reservation:
- Number of job slots to reserve-This number should less than or equal to the actual number of slots for the hosts defined in the reservation.
- Hosts for the reservation
- Owners of the reservation
- Time period for the reservation-either:
- Begin time and end time for a one-time reservation, OR
- Time window for a recurring reservation
note:
Advance reservations should be 10 minutes or more in length. Advance reservations of less than 10 minutes may be rejected if they overlap other advance reservations in 10-minute time slots of the weekly planner.The
brsvadd
command returns a reservation ID that you use when you submit a job that uses the reserved hosts. Any single user or user group can have a maximum of 100 reservation IDs.Specify hosts for the reservation
- Use one or both of the following
brsvadd
options to specify hosts for which job slots are reserved:
- The
-m
option lists the hosts needed for the reservation. The hosts listed by the-m
option can be local to the cluster or hosts leased from remote clusters. At job submission, LSF considers the hosts in the specified order. If you also specify a resource requirement string with the-R
option,-m
is optional.- The
-R
option selects hosts for the reservation according to a resource requirements string. Only hosts that satisfy the resource requirement expression are reserved.-R
accepts any valid resource requirement string, but only the select string takes effect. If you also specify a host list with the-m
option,-R
is optional.- If LSF_STRICT_RESREQ=y in
lsf.conf
, the selection string must conform to the stricter resource requirement string syntax described in Chapter 19, "Specifying Resource Requirements". The strict resource requirement syntax only applies to theselect
section. It does not apply to the other resource requirement sections (order
,rusage
,same
,span
, orcu
).Add a one-time reservation
- Use the
-b
and-e
options ofbrsvadd
to specify the begin time and end time of a one-time advance reservation. One-time reservations are useful for dedicating hosts to a specific user or group for critical projects.The day and time are in the form:
[[[year
:
]month
:
]day
:
]hour:
minute
with the following ranges:
year
: any year after 1900 (YYYY)month
: 1-12 (MM)day of the month
: 1-31 (dd)hour
: 0-23 (hh)minute
: 0-59 (mm)You must specify at least hour
:
minute. Year, month, and day are optional. Three fields are assumed to be day:
hour:
minute, four fields are assumed to be month:
day:
hour:
minute, and five fields are year:
month:
day:
hour:
minute.If you do not specify a day, LSF assumes the current day. If you do not specify a month, LSF assumes the current month. If you specify a year, you must specify a month.
You must specify a begin and an end time. The time value for
-b
must use the same syntax as the time value for-e
. The begin time must be earlier than the time value for-e
. The begin time cannot be earlier than the current time.The following command creates a one-time advance reservation for 1024 job slots on host
hostA
for useruser1
between 6:00 a.m. and 8:00 a.m. today:brsvadd -n 1024 -m hostA -u user1 -b 6:0 -e 8:0
Reservation "user1#0" is createdThe hosts specified by
-m
can be local to the cluster or hosts leased from remote clusters.The following command creates a one-time advance reservation for 1024 job slots on a host of any type for user
user1
between 6:00 a.m. and 8:00 a.m. today:brsvadd -n 1024 -R "type==any" -u user1 -b 6:0 -e 8:0
Reservation "user1#1" is createdThe following command creates a one-time advance reservation that reserves 12 slots on
hostA
between 6:00 p.m. on 01 December 2003 and 6:00 a.m. on 31 January 2004:brsvadd -n 12 -m hostA -u user1 -b 2003:12:01:18:00 -e 2004:01:31:06:00
Reservation user1#2 is createdAdd a recurring reservation
- Use the
-t
option ofbrsvadd
to specify a recurring advance reservation. The-t
option specifies a time window for the reservation. Recurring reservations are useful for scheduling regular system maintenance jobs.The day and time are in the form:
[day
:
]hour[:
minute
]with the following ranges:
day of the week
: 0-6hour
: 0-23minute
: 0-59Specify a time window one of the following ways:
hour
-
hour
hour
:
minute
-
hour
:
minute
day
:
hour
:
minute
-
day
:
hour
:
minute
You must specify at least the hour. Day of the week and minute are optional. Both the start time and end time values must use the same syntax. If you do not specify a minute, LSF assumes the first minute of the hour (
:00
). If you do not specify a day, LSF assumes every day of the week. If you do specify the day, you must also specify the minute.If the current time when the reservation is created is within the time window of the reservation. the reservation becomes active immmediately.
When the job starts running, the termination time of the advance reservation job is determined by the minimum of the job run limit (if specified), the queue run limit (if specified), or the duration of the reservation time window.
The following command creates an advance reservation for 1024 job slots on two hosts
hostA
andhostB
for user groupgroupA
every Wednesday from 12:00 midnight to 3:00 a.m.:brsvadd -n 1024 -m "hostA hostB" -g groupA -t "3:0:0-3:3:0"
Reservation "groupA#0" is createdThe following command creates an advance reservation for 1024 job slots on
hostA
for useruser2
every weekday from 12:00 noon to 2:00 p.m.:brsvadd -n 1024 -m "hostA" -u user2 -t "12:0-14:0"
Reservation "user2#0" is createdThe following command creates a system reservation on
hostA
every Friday from 6:00 p.m. to 8:00 p.m.:brsvadd -n 1024 -m hostA -s -t "5:18:0-5:20:0"
Reservation "system#0" is createdWhile the system reservation is active, no other jobs can use the reserved hosts, and LSF does not dispatch jobs to the specified hosts.
The following command creates an advance reservation for 1024 job slots on hosts
hostA
andhostB
with more that 50 MB of swap space for useruser2
every weekday from 12:00 noon to 2:00 p.m.:brsvadd -n 1024 -R "swp > 50" -m "hostA hostB" -u user2 -t "12:0-14:0"
Reservation "user2#1" is createdAdd an open reservation
- Use the
-o
option ofbrsvadd
to create an open advance reservation. You must specify the same information as for normal advance reservations.The following command creates a one-time open advance reservation for 1024 job slots on a host of any type for user
user1
between 6:00 a.m. and 8:00 a.m. today:brsvadd -o -n 1024 -R "type==any" -u user1 -b 6:0 -e 8:0
Reservation "user1#1" is createdThe following command creates an open advance reservation for 1024 job slots on
hostB
for useruser3
every weekday from 12:00 noon to 2:00 p.m.:brsvadd -o -n 1024 -m "hostB" -u user3 -t "12:0-14:0"
Reservation "user2#0" is createdSpecify a reservation name
- Use the
-N
option ofbrsvadd
to specify a user-defined advance reservation name unique in an LSF cluster.The reservation name is a string of letters, numeric characters, underscores, and dashes beginning with a letter. The maximum length of the name is 39 characters.
If no user-defined advance reservation name is specified, LSF creates the reservation with a system assigned name with the form
user_name#sequenceFor example:
brsvadd -n 3 -M "hostA hostB" -u user2 -b 16:0 -e 17:0 -d "Production AR test" Reservation user2#0 (Production AR test) is created brsvadd -n 2 -N Production_AR -M hostA -u user2 -b 16:0 -e 17:0 -d "Production AR test" Reservation Production_AR (Production AR test) is createdIf a job already exists that references a reservation with the specified name, an error message is returned: The specified reservation name is referenced by a job.
Modify an advance reservation
- Use
brsvmod
to modify reservations. Specify the reservation ID for the reservation you want to modify. For example, run the following command to extend the duration from 6:00 a.m. to 9:00 a.m.:brsvmod -e "+60" user1#0
Reservation "user1#0" is modifiedAdministrators and root can modify any reservations. Users listed in the ResourceReservation section of
lsb.resources
, can only modify reservations they created themselves.Using brsvmod to modify advance reservations
Use
brsvmod
to make the following changes to an existing advance reservation:
- Modify start time (postpone or move closer)
- Modify the duration of the reservation window (and thus the end time)
- Modify the slot numbers required by the reservation (add or remove slots with hosts)
- Modify the host or host group list (add or remove hosts or host groups)
- Modify the user or user group
- Add hosts by resource requirement (
-R
)- Modify the reservation type (open or closed)
- Disable the specified occurrences of a recurring reservation
For example, assume an advance reservation is the box between the time t1 and t2, as shown in the following figure:
![]()
In this figure:
- The shadowed box shows the original reservation
- Time means the time window of the reservation
- t1 is the begin time of the reservation
- t2 is the end time of the reservation
- The reservation size means the resources that are reserved, such as hosts (slots) or host groups
Use
brsvmod
to shift, extend or reduce the time window horizontally; grow or shrink the size vertically.Extending the duration
The following command creates a one-time advance reservation for 1024 job slots on host
hostA
for useruser1
between 6:00 a.m. and 8:00 a.m. today:brsvadd -n 1024 -m hostA -u user1 -b "6:0" -e "8:0"
Reservation "user1#0" is createdRun the following command to extend the duration from 6:00 a.m. to 9:00 a.m.:
brsvmod -e "+60" user1#0
Reservation "user1#0" is modifiedAdding hosts to a reservation allocation
Use
brsvmod
to add hosts and slots on hosts into the original advance reservation allocation. The hosts can be local to the cluster or hosts leased from remote clusters.Adding a host without
-n
reserves all available slots on the host; that is, slots that are not already reserved by other reservations. You must specify-n
along with-m
or-R
. The-m
option can be used alone if there is no host group specified in the list. You cannot specify-R
without-n
.The specified slot number must be less than or equal to the available number of job slots for the host.
You can only add hosts (
-m
) to a system reservation. You cannot add slots (-n
) to a system reservation.For example:
- Reserve 2 more slots from
hostA
:brsvmod addhost -n2 -m "hostA"Reserve 4 slots in total from hostA
andhostB
:brsvmod addhost -n4 -m "hostA hostB"Reserve 4 more slots from any Linux hosts: brsvmod addhost -n4 -R"type==linux"Reserve 4 more slots from any Linux hosts in the host group hostgroup1
:brsvmod addhost -n4 -m "hostgroup1" -R "type==linux"Reserve all available slots from hostA
andhostB
:brsvmod addhost -m "hostA hostB"The following command creates an advance reservation for 1024 slots on two hosts
hostA
andhostB
for user groupgroupA
every Wednesday from 12:00 midnight to 3:00 a.m.:brsvadd -n 1024 -m "hostA hostB" -g groupA -t "3:0:0-3:3:0"
Reservation "groupA#0" is createdbrsvs
RSVID TYPE USER NCPUS RSV_HOSTS TIME_WINDOW groupA#0 user groupA 0/1024 hostA:0/256 3:3:0-3:3:0 * hostB:0/768The following commands reserve 512 slots from each host for the reservation:
brsvmod addhost -n 256 -m "hostA" groupA#0
Reservation "groupA#0" is modifiedbrsvmod rmhost -n 256 -m "hostB" groupA#0
Reservation "groupA#0" is modifiedRemoving hosts from a reservation allocation
Use
brsvmod rmhost
to remove hosts or slots on hosts from the original reservation allocation. You must specify either-n
or-m
. Use-n
to specify the number of slots to be released from the host. Removing a host without-n
releases all reserved slots on the host. The slot specification must be less than or equal to the actual reserved slot number of the host.For example:
- Remove 4 reserved slots from
hostA
brsvmod rmhost -n 4 -m "hostA"Remove 4 slots in total from hostA
andhostB
.brsvmod rmhost -n 4 -m "hostA hostB"Release reserved hostA
andhostB
.brsvmod rmhost -m "hostA hostB"Remove 4 slots from current reservation allocation. brsvmod rmhost -n 4You cannot remove slots from a system reservation. The following modification to the system reservation
System#1
is rejected:brsvmod rmhost -n 2 -m "hostA" system#1How many slots or hosts can be removed also depends on the number of slots free while the reservation is active.
brsvmod rmhost
cannot remove more slots than free amount on a host. For example:brsvs
RSVID TYPE USER NCPUS RSV_HOSTS TIME_WINDOW user1_1 user user1 3/4 hostA:2/2 1/24/12/2-1/24/13/0 hostB:1/2The following modifications are accepted, and one slot is removed from
hostB
:brsvmod rmhost -m hostB user1_1 brsvmod rmhost -n 1 -m hostB user1_1The following modifications are rejected:
brsvmod rmhost -n 2 user1_1 brsvmod rmhost -m hostA user1_1 brsvmod rmhost -n 1 -m hostA user1_1 brsvmod rmhost -n 2 -m hostB user1_1Modifying closed reservations
The following command creates an open advance reservation for 1024 job slots on host
hostA
for useruser1
between 6:00 a.m. and 8:00 a.m. today.brsvadd -o -n 1024 -m hostA -u user1 -b 6:0 -e 8:0 Reservation "user1#0" is createdRun the following command to close the reservation when it expires.
brsvmod -on user1#0 Reservation "user1#0" is modifiedDisable specified occurrences for recurring reservations
Use
brsvmod disable
to disable specified periods, orinstances
, of a recurring advance reservation.Recurring reservations may repeat either on a daily cycle or a weekly cycle. For daily reservations, the instances of the reservation that occur on disabled days will be inactive. Jobs using the reservation are not dispatched during on those disabled days. Other reservations are permitted to use slots of the reservation on those days. For overnight reservations (active from 11 p.m. to 9 a.m. daily), if the reservation is disabled on the starting day of an instance, the reservation is disabled for the whole of that instance.
For a weekly reservation, if the reservation is disabled on the start date of an instance of the reservation then the reservation is disabled for the entire instance. For example, for a weekly reservation with time window from 9 a.m. Wednesday to 10 p.m. Friday, in one particular week, the reservation is disabled on Thursday, then the instance of the reservation remains active for that week. However, if the same reservation is disabled for the Wednesday of the week, then the reservation is disabled for the week.
The following figure illustrates how the disable options apply to the weekly occurrences of a recurring advance reservation.
![]()
Once a reservation is disabled for a period, it cannot be enabled again; that is, the disabled periods remain fixed. Before a reservation is disabled, you are prompted to confirm whether to continue disabling the reservation. Use the
-f
option to silently force the command to run without prompting for confirmation, for example, to allow for automating disabling reservations from a script.For example, the following command creates a recurring advance reservation for 4 slots on host
hostA
for useruser1
between 6:00 a.m. and 8:00 a.m. every day.Reservation "user1#0" is created brsvadd -n 4 -m hostA -u user1 -t "6:0-8:0"Run the following command to disable the reservation instance that is active between Dec 1 to Dec 10, 2007.
brsvmod -disable -td "2007:12:1-2007:12:10" user1#0 Reservation "user1#0" is modifiedThen the administrator can use host
hostA
for other reservations during the durationbrsvadd -n 4 -m hostA -u user1 -b "2007:12:1:6:0" -e "2007:12:1:8:0" Reservation "user1#2" is createdChange users and user groups
Use
brsvmod -u
to change the user orbrsvmod -g
to change the user group that is able to submit jobs with the advance reservation.Jobs submitted by the original user or user group to the reservation still belong to the reservation and scheduled as advance reservation jobs, but new submitted jobs from the removed user or user group cannot use the reservation any longer.
brun
An advance reservationa job dispatched with
brun
is still subject to run windows and suspending conditions of the advance reservation for the job. The job must finish running before the time window of a closed reservation expires. Extending or shrinking a closed advance reservation duration prolongs or shortens lifetime of abrun
job.bslots
bslots
displays a snapshot of the slots currently not in use by parallel jobs or advance reservations. If the hosts or duration of an advance reservation is modified,bslots
recalculates and displays the available slots and available run time accordingly.How advance reservation modifications interact
The following table summarizes how advance reservation modification applies to various advance reservation instances.
Where: "Yes" means the modification is supported in the scenario; otherwise, "No" is marked. For example, all modifications are acceptable in the case that the advance reservation is inactive.
Reservation policy checking
The following table summarizes how advance reservation commands interpret reservation policy configurations in
lsb.resources
:
Reservation policies are checked when:
- Modifying the reservation time window
- Adding hosts to the reservation
Reservation policies are
not
checked when
- Running
brsvmod
to remove hosts- Changing the reservation type (open or closed)
- Changing users or user groups for the reservation
- Modifying the reservation description
Remove an advance reservation
- Use
brsvdel
to delete reservations. Specify the reservation ID for the reservation you want to delete.For example:
brsvdel user1#0
Reservation user1#0 is being deletedYou can delete more than one reservation at a time. Administrators can delete any reservation, but users may only delete their own reservations.
If the recurring reservation is deleted with
brsvdel
, jobs running in the reservation are detached from the reservation and scheduled as normal jobs.View reservations
- Use
brsvs
to show current reservations:brsvs
RSVID TYPE USER NCPUS RSV_HOSTS TIME_WINDOW user1#0 user user1 0/1024 hostA:0/1024 11/12/6/0-11/12/8/0 user2#0 user user2 0/1024 hostA:0/1024 12:0-14:0 * groupA#0 group groupA -/2048 hostA:-/1024 3:0:0-3:3:0 * hostB:0/1024 system#0 sys system 1024 hostA:0/1024 5:18:0-5:20:0 *In the TIME_WINDOW column:
- A one-time reservation displays fields separated by slashes (
month/day/hour/minute
). For example:11/12/14/0-11/12/18/0- A recurring reservation displays fields separated by colons (
day:hour:minute
). An asterisk (*) indicates a recurring reservation. For example:5:18:0-5:20:0 *In the NCPUS and RSV_HOSTS columns:
- Remote reservations do not display details. For example:
-/2048 hostA:-/1024Show a weekly planner
- Use
brsvs -p
to show a weekly planner for specified hosts using advance reservation. Theall
keyword shows the planner for all hosts with reservations.The output of
brsvs -p
is displayed in terms of weeks. The week starts on Sunday. The timeframe of a recurring reservation is not displayed, since it is unlimited. The timeframe of one-time reservation is displayed in terms of a week. If the reservation spans multiple weeks, these weeks are displayed separately. If a week contains a one-time reservation and a recurring reservation, the timeframe is displayed, since that is relevant for one-time reservation.
tip:
MAX indicates the configured maximum number of job slots for the host (MXJ defined inlsb.hosts
).brsvs -p all
RSVID TYPE USER NCPUS RSV_HOSTS TIME_WINDOW user1#0 user user1 0/1024 hostA:0/1024 11/12/6/0-11/12/8/0 user2#0 user user2 0/1024 hostA:0/1024 12:0-14:0 * groupA#0 group groupA 0/2048 hostA:0/1024 3:0:0-3:3:0 * hostB:0/1024 system#0 sys system 1024 hostA:0/1024 5:18:0-5:20:0 * HOST: hostA (MAX = 1024) Week: 11/11/2009
- 11/17/2009
Hour:Min Sun Mon Tue Wed Thu Fri Sat ------------------------------------------------------------------- 0:0 0 0 0 1024 0 0 0 0:10 0 0 0 1024 0 0 0 0:20 0 0 0 1024 0 0 0 ... 2:30 0 0 0 1024 0 0 0 2:40 0 0 0 1024 0 0 0 2:50 0 0 0 1024 0 0 0 3:0 0 0 0 0 0 0 0 3:10 0 0 0 0 0 0 0 3:20 0 0 0 0 0 0 0 ... 5:30 0 0 0 0 0 0 0 5:40 0 0 0 0 0 0 0 5:50 0 0 0 0 0 0 0 6:0 0 1024 0 0 0 0 0 6:10 0 1024 0 0 0 0 0 6:20 0 1024 0 0 0 0 0 ... 7:30 0 1024 0 0 0 0 0 7:40 0 1024 0 0 0 0 0 7:50 0 1024 0 0 0 0 0 8:0 0 0 0 0 0 0 0 8:10 0 0 0 0 0 0 0 8:20 0 0 0 0 0 0 0 ... 11:30 0 0 0 0 0 0 0 11:40 0 0 0 0 0 0 0 11:50 0 0 0 0 0 0 0 12:0 1024 1024 1024 1024 1024 1024 1024 12:10 1024 1024 1024 1024 1024 1024 1024 12:20 1024 1024 1024 1024 1024 1024 1024 ... 13:30 1024 1024 1024 1024 1024 1024 1024 13:40 1024 1024 1024 1024 1024 1024 1024 13:50 1024 1024 1024 1024 1024 1024 1024 14:0 0 0 0 0 0 0 0 14:10 0 0 0 0 0 0 0 14:20 0 0 0 0 0 0 0 ... 17:30 0 0 0 0 0 0 0 17:40 0 0 0 0 0 0 0 17:50 0 0 0 0 0 0 0 18:0 0 0 0 0 0 1024 0 18:10 0 0 0 0 0 1024 0 18:20 0 0 0 0 0 1024 0 ... 19:30 0 0 0 0 0 1024 0 19:40 0 0 0 0 0 1024 0 19:50 0 0 0 0 0 1024 0 20:0 0 0 0 0 0 0 0 20:10 0 0 0 0 0 0 0 20:20 0 0 0 0 0 0 0 ... 23:30 0 0 0 0 0 0 0 23:40 0 0 0 0 0 0 0 23:50 0 0 0 0 0 0 0 HOST: hostB (MAX = 1024) Week: 11/11/2009
- 11/17/2009
Hour:Min Sun Mon Tue Wed Thu Fri Sat ------------------------------------------------------------------- 0:0 0 0 0 1024 0 0 0 0:10 0 0 0 1024 0 0 0 0:20 0 0 0 1024 0 0 0 ... 2:30 0 0 0 1024 0 0 0 2:40 0 0 0 1024 0 0 0 2:50 0 0 0 1024 0 0 0 3:0 0 0 0 0 0 0 0 3:10 0 0 0 0 0 0 0 3:20 0 0 0 0 0 0 0 ... 23:30 0 0 0 0 0 0 0 23:40 0 0 0 0 0 0 0 23:50 0 0 0 0 0 0 0- Use
brsvs -z
instead ofbrsvs -p
to show only the weekly items that have reservation configurations. Lines that show all zero (0) are omitted.For example:
brsvs -z all
RSVID TYPE USER NCPUS RSV_HOSTS TIME_WINDOW user1_1 user user1 0/3 hostA:0/2 12/28/14/30-12/28/15/30 hostB:0/1 HOST: hostA (MAX = 2) Week: 12/23/2007 - 12/29/2007 Hour:Min Sun Mon Tue Wed Thu Fri Sat ---------------------------------------------------------------------- 14:30 0 0 0 0 0 1 0 14:40 0 0 0 0 0 1 0 14:50 0 0 0 0 0 1 0 15:0 0 0 0 0 0 1 0 15:10 0 0 0 0 0 1 0 15:20 0 0 0 0 0 1 0 HOST: hostB (MAX = 2) Week: 12/23/2007 - 12/29/2007 Hour:Min Sun Mon Tue Wed Thu Fri Sat ---------------------------------------------------------------------- 14:30 0 0 0 0 0 2 0 14:40 0 0 0 0 0 2 0 14:50 0 0 0 0 0 2 0 15:0 0 0 0 0 0 2 0 15:10 0 0 0 0 0 2 0 15:20 0 0 0 0 0 2 0Show reservation types and associated jobs
- Use the
-l
option ofbrsvs
to show each advance reservation in long format.The rows that follow the reservation information show the
- The status of the reservation
- Time when the next instance of recurring reservation is active
- Type of reservation (open or closed)
- The status by job ID of any job associated with the specified reservation (FINISHED, PEND, RUN, or SUSP)
brsvs -l
RSVID TYPE USER NCPUS RSV_HOSTS TIME_WINDOW user1_1#0 user user1_1 10/10 host1:4/4 8:00-22:00 * host2:4/4 host3:2/2 Reservation Status: Active Next Active Period: Sat Aug 22 08:00:00 2009 - Sat Aug 22 22:00:00 2009 Creator: user1_1 Reservation Type: CLOSED FINISHED Jobs: 203 204 205 206 207 208 209 210 211 212 PEND Jobs: 323 324 RUN Jobs: 313 314 316 318 319 320 321 322 SUSP Jobs: 315 317Show reservation ID
- Use
bjobs -l
to show the reservation ID used by a job:bjobs -l
Job <1152>, User <user1>, Project <default>, Status <PEND>, Queue <normal>, Reservation <user1#0>, Command <myjob> Mon Nov 12 5:13:21: Submitted from host <hostB>, CWD </home/user1/jobs>;View historical accounting information for advance reservations
- Use the
-U
option of thebacct
command to display accounting information about advance reservations.
bacct -U
summarizes all historical modification of the reservation and displays information similar to thebrsvs
command:
- The reservation ID specified on the -U option.
- The type of reservation:
user
orsystem
- The user names of users who used the
brsvadd
command to create the advance reservations- The user names of the users who can use the advance reservations (with
bsub -U
)- Number of slots reserved
- List of hosts for which job slots are reserved
- Time window for the reservation.
- A one-time reservation displays fields separated by slashes (
month/day/hour/minute
). For example:11/12/14/0-11/12/18/0- A recurring reservation displays fields separated by colons (
day:hour:minute
). For example:5:18:0 5:20:0For example, the following advance reservation has four time modifications during its life time. The original reservation has the scope of one user (
user1
) and one host (hostA
) with 1 slot. The various modifications change the user touser2
, then back touser1
, adds, then removes 1 slot from the reservation.bacct -U user1#1
Accounting about advanced reservations that are: - accounted on advanced reservation IDs user1#1, - accounted on advanced reservations created by user1, ---------------------------- SUMMARY ---------------------------- RSVID: user1#1 TYPE: user CREATOR: user1 Total number of jobs: 0 Total CPU time consumed: 0.0 second Maximum memory of a job: 0.0 MB Maximum swap of a job: 0.0 MB Total active time: 0 hour 6 minute 42 second ------------------------ Configuration 0 ------------------------ RSVID TYPE CREATOR USER NCPUS RSV_HOSTS user1#1 user user1 user1 1 hostA:1 Active time with this configuration: 0 hour 0 minute 16 second ------------------------ Configuration 1 ------------------------ RSVID TYPE CREATOR USER NCPUS RSV_HOSTS user1#1 user user1 user2 1 hostA:1 Active time with this configuration: 0 hour 0 minute 24 second ------------------------ Configuration 2 ------------------------ RSVID TYPE CREATOR USER NCPUS RSV_HOSTS user1#1 user user1 user2 1 hostA:1 Active time with this configuration: 0 hour 1 minute 58 second ------------------------ Configuration 3 ------------------------ RSVID TYPE CREATOR USER NCPUS RSV_HOSTS user1#1 user user1 user1 2 hostA:2 Active time with this configuration: 0 hour 1 minute 34 second ------------------------ Configuration 4 ------------------------ RSVID TYPE CREATOR USER NCPUS RSV_HOSTS user1#1 user user1 user1 1 hostA:2 Active time with this configuration: 0 hour 2 minute 30 secondThe following reservation (
user2#0
) has one time modification during its life time. The original one has the scope of one user (user2
) and one host (hostA
) with 1 slot; the modification changes the user touser3
.bacct -U user2#0
Accounting about advanced reservations that are: - accounted on all advanced reservation IDs: - accounted on advanced reservations created by all users: --------------------------- SUMMARY ------------------------- RSVID: user2#0 TYPE: user CREATOR: user2 Total number of jobs: 1 Total CPU time consumed: 5.0 second Maximum memory of a job: 1.7 MB Maximum swap of a job: 7.5 MB Total active time: 2 hour 0 minute 0 second ------------------------ Configuration 0 ------------------------ RSVID TYPE CREATOR USER NCPUS RSV_HOSTS user1#0 user user2 user2 1 hostA:1 Active time with this configuration: 1 hour 0 minute 0 second ------------------------ Configuration 1 ------------------------ RSVID TYPE CREATOR USER NCPUS RSV_HOSTS user1#0 user user2 user3 1 hostA:1 Active time with this configuration: 1 hour 0 minute 0 secondSubmit and modify jobs using advance reservations
- Use the
-U
option ofbsub
to submit jobs with a reservation ID. For example:bsub -U user1#0 myjob
The job can only use hosts reserved by the reservation
user1#0
. By default, LSF selects only hosts in the reservation. Use the-m
option to specify particular hosts within the list of hosts reserved by the reservation; you can only select from hosts that were included in the original reservation.If you do not specify hosts (
bsub -m
) or resource requirements (bsub -R
), the default resource requirement is to select hosts that are of any host type (LSF assumes"type==any"
instead of"type==local"
as the default select string).If you later delete the advance reservation while it is still active, any pending jobs still keep the
"type==any"
attribute.A job can only use one reservation. There is no restriction on the number of jobs that can be submitted to a reservation; however, the number of slots available on the hosts in the reservation may run out. For example, reservation
user2#0
reserves 1024 slots onhostA
. When all 1024 slots onhostA
are used by jobs referencinguser2#0
,hostA
is no longer available to other jobs using reservationuser2#0
. Any single user or user group can have a maximum of 100 reservation IDs.Jobs referencing the reservation are killed when the reservation expires.
Modify job reservation ID
Prerequisites: You must be an administrator to perform this task.
- Use the
-U
option ofbmod
to change a job to another reservation ID.For example:
bmod -U user1#0 1234
- To cancel the reservation, use the
-Un
option ofbmod
.For example:
bmod -Un 1234
Use
bmod -Un
to detach a running job from aninactive
open reservation. Once detached, the job is scheduled like a normal job.Job resource usage limits and job chunking
A job using a reservation is subject to all job resource usage limits. If a limit is reached on a particular host in a reservation, jobs using that reservation cannot start on that host.
An advance reservation job is dispatched to its reservation even if the run limit or estimated run time of the job exceeds the remaining active time of the reservation. For example, if a job has a runlimit of 1 hour, and a reservation has a remaining active time of 1 minute, the job is still dispatched to the reservation. If the reservation is closed, the job is terminated when the reservation expires.
Similarly, when using chunk job scheduling, advance reservation jobs are chunked together as usual when dispatched to a host of the reservation without regard to the expiry time of the reservation. This is true even when the jobs are given a run limit or estimated run time. If the reservation is closed, the jobs in WAIT state are terminated when the reservation expires.
Advance reservation preemption
Advance reservation preemption allows advance reservation jobs to use the slots reserved by the reservation. Slots occupied by non-advance jobs may be preempted when the reservation becomes active.
Without modification with
brsvmod
, advance reservation preemption is triggered at most once per reservation period (in the case of a non-recurring reservation, there is only one period) wheneverboth
of the following conditions are met:
- The reservation is active
- At least one job associated with the advance reservation is pending or suspended
If an advance reservation is modified, preemption is done for an active advance reservation after every modification of the reservation when there is at least one pending or suspended job associated with the reservation.
When slots are added to an advance reservation with
brsvmod
, LSF preempts running non-reservation jobs if necessary to provide slots for jobs belonging to the reservation. Preemption is triggered if there are pending or suspended jobs belonging to the reservation in the system.When preemption is triggered, non-advance reservation jobs are suspended and their slots given to the advance reservation on the hosts belonging to the reservation. On each host, enough non-advance reservation jobs are suspended so that all of slots required by the advance reservation are obtained. The number of slots obtained does not depend on the number of jobs submitted to the advance reservation. Non-advance reservation jobs on a host can only to use slots not assigned to the advance reservation.
When a job is preempted for an advance reservation, it can only resume on the host when either the advance reservation finishes, or some other non-advance reservation job finishes on the host.
For example, a single-host cluster has 10 slots, with 9 non-advance reservation jobs dispatched to the host (each requiring one slot). An advance reservation that uses 5 slots on the host is created, and a single job is submitted to the reservation. When the reservation becomes active, 4 of the non-advance reservation jobs are suspended, and the advance reservation job will start.
Forcing a job to run before a reservation is active
LSF administrators can use
brun
to force jobs to run before the reservation is active, but the job must finish running before the time window of the reservation expires.For example, if the administrator forces a job with a reservation to run one hour before the reservation is active, and the reservation period is 3 hours, a 4 hour run limit takes effect.
Host intersection and advance reservation
When
ENABLE_HOST_INTERSECTION
=y inlsb.params
, LSF finds any existing intersection with hosts specified in the queue and those specified at job submission bybsub -m
and/or hosts with advance reservation. When specifying keywords such asall
,allremote
, andothers
, LSF finds an existing intersection of hosts available and the job runs rather than being rejected.Advance reservations across clusters
You can create and use advance reservation for the MultiCluster job forwarding model. To enable this feature, you must upgrade all clusters to LSF Version 7 or later.
See the
Using Platform LSF MultiCluster
for more information.Resizable jobs and advance reservations
Like regular jobs, resizable jobs associated with an advance reservation can be dispatched only after the reservation becomes active, and the minimum processor request can be satisfied. The allocation request is treated like a regular advance reservation job, which relies on slots available to the reservation. If an advance reservation gets more resources by modification (
brsvmod addhost
), those resources can be used by pending allocation requests immediately.The following table summarizes the relationship of the AR lifecycle and resizable job requests:
By the time a reservation has expired or deleted, the status change of the resizable job to SSUSP blocks a resizable job allocation request from being scheduled.
Released slots from a resizable job can be reused by other jobs in the reservation.
Resizable advance reservation jobs can preempt non-advance reservation jobs that are consuming the slots that belong to the reservation. Higher priority advance reservation jobs can preempt low priority advance reservation jobs, regardless of whether both are resizable jobs.
Allocation requests of resizable AR jobs honor limits configuration. They cannot preempt any limit tokens from other jobs.
Compute units and advance reservations
Like regular jobs, jobs with compute unit resource requirements and an advance reservation can be dispatched only after the reservation becomes active, and the minimum processor request can be satisfied.
In the case of exclusive compute unit jobs (with the resource requirement
cu
[excl]
), the advance reservation can affect hosts outside the advance reservation but in the same compute unit as follows:
- An exclusive compute unit job dispatched to a host inside the advance reservation will lock the entire compute unit, including any hosts outside the advance reservation.
- An exclusive compute unit job dispatched to a host outside the advance reservation will lock the entire compute unit, including any hosts inside the advance reservation.
Ideally all hosts belonging to a compute unit should be inside or outside of an advance reservation.
Platform Computing Inc.
www.platform.com |
Knowledge Center Contents Previous Next Index |