Knowledge Center Contents Previous Next Index |
Specifying Resource Requirements
Contents
- About Resource Requirements
- Queue-level Resource Requirements
- Job-level Resource Requirements
- About Resource Requirement Strings
- Selection String
- Order String
- Usage String
- Span String
- Same String
- Compute Unit String
About Resource Requirements
Resource requirements define which hosts a job can run on. Each job has its resource requirements and hosts that match the resource requirements are the candidate hosts. When LSF schedules a job, it uses the load index values of all the candidate hosts. The load values for each host are compared to the scheduling conditions. Jobs are only dispatched to a host if all load values are within the scheduling thresholds.
By default, if a job has no resource requirements, LSF places it on a host of the same type as the submission host (i.e.,
type==local
). However, if a job has string or Boolean resource requirements specified and the host type has not been specified, LSF places the job on any host (i.e.,type==any
) that satisfies the resource requirements.To override the LSF defaults, specify resource requirements explicitly. Resource requirements can be set for queues, for application profiles, or for individual jobs.
To best place a job with optimized performance, resource requirements can be specified for each application. This way, you do not have to specify resource requirements every time you submit a job. The LSF administrator may have already configured the resource requirements for your jobs, or you can put your executable name together with its resource requirements into your personal remote task list.
The
bsub
command automatically uses the resource requirements of the job from the remote task lists.A resource requirement is an expression that contains resource names and operators.
Compound Resource Requirements
In some cases different resource requirements may apply to different parts of a parallel job. The first execution host, for example, may require more memory or a faster processor for optimal job scheduling. Compound resource requirements allow you to specify different requirements for some slots within a job in the queue-level, application-level, or job-level resource requirement string.
Compound resource requirement strings can be set by the application-level or queue-level
RES_REQ
parameter, or used withbsub -R
when a job is submitted.bmod -R
also accepts compound resource requirement strings for both pending and running jobs.Special rules take effect when compound resource requirements are merged with resource requirements defined at more than one level. If a compound resource requirement is used at any level (job, application, or queue) the compound multi-level resource requirement combinations described later in this chapter apply.
restriction:
Compound resource requirements cannot containcu
sections, multiple-R
options, or the || operator.Resizable jobs cannot have compound resource requirements.Resource allocation for parallel jobs using compound resources is done for each compound resource term in the order listed instead of considering all possible combinations. A host rejected for not satisfying one resource requirement term will not be reconsidered for subsequent resource requirement terms.Compound resource requirements were introduced in LSF Version 7 Update 5, and are not compatible with earlier versions of LSF.Resource requirements in application profiles
See Chapter 23, "Working with Application Profiles" for information about how resource requirements in application profiles are resolved with queue-level and job-level resource requirements.
Resizable jobs and resource requirements
In general, resize allocation requests for resizable jobs use the resource requirements of the running job. When the resource requirement string for a job is modified with
bmod -R
, the new string takes effects for a job resize request. The resource requirement of the allocation request is merged from resource requirements specified at the queue, job, and application levels.
restriction:
Autoresizable jobs cannot have compute unit resource requirements. Any autoresizable jobs switched to queues with compute unit resource requirements will no longer be autoresizable.Resizable jobs cannot have compound resource requirements.Queue-level Resource Requirements
Each queue can define resource requirements that apply to all the jobs in the queue.
When resource requirements are specified for a queue, and no job-level or application profile resource requirement is specified, the queue-level resource requirements become the default resource requirements for the job.
Resource requirements determined by the queue no longer apply to a running job after running
badmin reconfig
, For example, if you change the RES_REQ parameter in a queue and reconfigure the cluster, the previous queue-level resource requirements for running jobs are lost.Syntax
The condition for dispatching a job to a host can be specified through the queue-level
RES_REQ
parameter in the queue definition inlsb.queues
. Queue-levelRES_REQ
rusage values must be in the range set byRESRSV_LIMIT
(set inlsb.queues
), or the queue-levelRES_REQ
is ignored.Examples
RES_REQ=select[((type==LINUX2.4 && r1m < 2.0)||(type==AIX && r1m < 1.0))]This allows a queue, which contains LINUX2.4 and AIX hosts, to have different thresholds for different types of hosts.
RES_REQ=select[((hname==hostA && mem > 50)||(hname==hostB && mem > 100))]Using the
hname
resource in the resource requirement string allows you to set up different conditions for different hosts in the same queue.Load thresholds
Load thresholds can be configured by your LSF administrator to schedule jobs in queues. Load thresholds specify a load index value.
loadSched
The scheduling threshold that determines the load condition for dispatching pending jobs. If a host's load is beyond any defined
loadSched
, a job is not started on the host. This threshold is also used as the condition for resuming suspended jobs.loadStop
The suspending condition that determines when running jobs should be suspended.
Thresholds can be configured for each queue, for each host, or a combination of both. To schedule a job on a host, the load levels on that host must satisfy both the thresholds configured for that host and the thresholds for the queue from which the job is being dispatched.
The value of a load index may either increase or decrease with load, depending on the meaning of the specific load index. Therefore, when comparing the host load conditions with the threshold values, you need to use either greater than (>) or less than (<), depending on the load index.
See Chapter 37, "Load Thresholds" for information about suspending conditions and configuring load thresholds.
View queue-level resource requirements
- Use
bqueues -l
to view resource requirements (RES_REQ) defined for the queue:bqueues -l normal
QUEUE: normal -- No description provided. This is the default queue. ... RES_REQ: select[type==any] rusage[mem=10,dynamic_rsrc=10:duration=2:decay=1] ...
Job-level Resource Requirements
Each job can specify resource requirements. Job-level resource requirements override any resource requirements specified in the remote task list.
In some cases, the queue specification sets an upper or lower bound on a resource. If you attempt to exceed that bound, your job will be rejected.
Syntax
To specify resource requirements for your job, use
bsub -R
and specify the resource requirement string as usual. You can specify multiple-R
order
,same
,rusage
, andselect
sections.
tip:
Within esub, you can get resource requirements using the LSB_SUB_RES_REQ variable, which merges multiple -R from thebsub
command. If you want to modify the LSB_SUB_RES_REQ variable, you cannot use multiple -R format. Instead, use the && operator to merge them manually.MergedRES_REQ
rusage
values from the job and application levels must be in the range ofRESRSV_LIMIT
(set inlsb.queues
), or the job is rejected.Examples
bsub -R "swp > 15 && hpux order[ut]" myjob
or
bsub -R "select[swp > 15]" -R "select[hpux] order[ut]" myjob
This runs
myjob
on an HP-UX host that is lightly loaded (CPU utilization) and has at least 15 MB of swap memory available.
bsub -R "select[swp > 15]" -R "select[hpux] order[r15m]"
-R "order[r15m]" -R rusage[mem=100]" -R "order[ut]" -R "same[type] -R "rusage[tmp=50:duration=60]" -R "same[model]" myjobLSF merges the multiple
-R
options into one string and dispatches the job if all of the resource requirements can be met. By allowing multiple resource requirement strings and automatically merging them into one string, LSF simplifies the use of multiple layers of wrapper scripts.View job-level resource requirements
- Use
bjobs -l
to view resource requirements defined for the job:bsub -R "type==any" -q normal myjob
Job <2533> is submitted to queue <normal>.bjobs -l 2533
Job <2533>, User <user1>, Project <default>, Status <DONE>, Queue <normal>, Command <myjob> Fri May 10 17:21:26: Submitted from host <hostA>, CWD <$HOME>, Requested Resources <type==any>; Fri May 10 17:21:31: Started on <hostB>, Execution Home </home/user1>,Execution CWD </home/user1>; Fri May 10 17:21:47: Done successfully. The CPU time used is 0.3 seconds. ...- After a job is finished, use
bhist -l
to view resource requirements defined for the job:bhist -l 2533
Job <2533>, User <user1>, Project <default>, Command <myjob> Fri May 10 17:21:26: Submitted from host <hostA>, to Queue <normal>, CWD <$HOME>, Requested Resources <type==any>; Fri May 10 17:21:31: Dispatched to <hostB>; Fri May 10 17:21:32: Starting (Pid 1850232); Fri May 10 17:21:33: Running with execution home </home/user1>, Execution CWD </home/user1>, Execution Pid <1850232>; Fri May 10 17:21:45: Done successfully. The CPU time used is 0.3 seconds; ...
note:
If you submitted a job with multiple select strings using thebsub
-R
option,bjobs -l
andbhist -l
display a single, merged select string.About Resource Requirement Strings
Most LSF commands accept a
-R
res_req
argument to specify resource requirements. The exact behavior depends on the command. For example, specifying a resource requirement for thelsload
command displays the load levels for all hosts that have the requested resources.Specifying resource requirements for the
lsrun
command causes LSF to select the best host out of the set of hosts that have the requested resources.A resource requirement string describes the resources a job needs. LSF uses resource requirements to select hosts for remote execution and job execution.
Resource requirement strings can be simple (applying to the entire job) or compound (applying to the specified number of slots).
Resource requirement string sections
- A selection section (
select
). The selection section specifies the criteria for selecting hosts from the system.- An ordering section (
order
). The ordering section indicates how the hosts that meet the selection criteria should be sorted.- A resource usage section (
rusage
). The resource usage section specifies the expected resource consumption of the task.- A job spanning section (
span
). The job spanning section indicates if a parallel batch job should span across multiple hosts.- A same resource section (
same
). The same section indicates that all processes of a parallel job must run on the same type of host.- A compute unit resource section (
cu
). Thecu
section specifies how a job should be placed with respect to the underlying network architecture.Which sections apply
Depending on the command, one or more of these sections may apply. For example:
bsub
uses all sectionslshosts
only selects hosts, but does not order themlsload
selects and orders hostslsplace
uses the information inselect
,order
, andrusage
sections to select an appropriate host for a tasklsloadadj
uses therusage
section to determine how the load information should be adjusted on a hostSimple Syntax
select[
selection_string
] order[
order_string
] rusage[
usage_string
[,
usage_string
][||
usage_string
] ...] span[
span_string
] same[
same_string
]
cu[
cu_string
]
With the
bsub
andbmod
commands, and only with these commands, you can specify multiple-R
order, same, rusage, and select sections. Thebmod
command does not support the use of the||
operator.The section names are s
elect
,order
,rusage
,span
,same
, andcu
. Sections that do not apply for a command are ignored. Each section has a different syntax.The square brackets must be typed as shown for each section. A blank space must separate each resource requirement section.
You can omit the
select
keyword and the square brackets, but the selection string must be thefirst
string in the resource requirement string. If you do not give a section name, the first resource requirement string is treated as a selection string (select[
selection_string
]
).Each section has a different syntax.
By default, memory (
mem
) and swap (swp
) limits inselect[]
andrusage[]
sections are specified in MB. Use LSF_UNIT_FOR_LIMITS inlsf.conf
to specify a larger unit for the these limits (MB, GB, TB, PB, or EB).Compound Syntax
num1
*{
simple_string1
} +
num2
*{
simple_string2
} + ...
where
numx
is the number of slots affected andsimple_stringx
is a simple resource requirement string with the syntax:select[
selection_string
] order[
order_string
] rusage[
usage_string
[,
usage_string
]...] span[
span_string
]
Resource requirements applying to the first execution host (if used) should appear in the first compound term
num1
*{
simple_string1
}
.Place specific (harder to fill) requirements before general (easier to fill) requirements since compound resource requirement terms are considered in the order they appear. Resource allocation for parallel jobs using compound resources is done for each compound resource term independently instead of considering all possible combinations.
note:
A host rejected for not satisfying one resource requirement term will not be reconsidered for subsequent resource requirement terms.For jobs without the number of total slots specified using
bsub -n
, the finalnumx
can be omitted. The final resource requirement is then applied to the zero or more slots not yet accounted for using the default slot setting of the parameterPROCLIMIT
as follows:
- (final res_req number of slots) = MAX(0,(default number of job slots from
PROCLIMIT
)-(num1
+num2
+...))For jobs with the total number of slots specified using
bsub -n
num_slots
, the total number of slots must match the number of slots in the resource requirement as follows, and the finalnumx
can be omitted:
num_slots
=(num1
+num2
+num3
+...)For jobs with compound resource requirements and first execution host candidates specified using bsub -m, the first allocated host must satisfy the simple resource requirement string appearing first in the compound resource requirement. Thus the first execution host must satisfy the requirements in
simple_string1
for the following compound resource requirement:
num1
*{simple_string1
} +num2
*{simple_string2
} +num3
*{simple_string3
}Compound resource requirements do not support use of the
||
operator within the componentrusage
simple resource requirements, or use of thecu
section.How simple multi-level resource requirements are resolved
Simple resource requirements can be specified at the job, application, and queue levels. When none of the resource requirements are compound, requirements defined at different levels are resolved in the following ways:
- In a
select
string, a host must satisfyall
queue-level, application-level, and job-level requirements for the job to be dispatched.- In a
same
string, all queue-level, application-level, and job-level requirements are combined before the job is dispatched.order
,span
, andcu
sections defined at the job level overwrite those defined at the application level or queue level.order
,span
, andcu
sections defined at the application level overwrite those defined at the queue level. The defaultorder
string isr15s:pg
.- For usage strings, the
rusage
section defined for the job overrides therusage
section defined in the application. The tworusage
definitions are merged, with the job-levelrusage
taking precedence. Similarly,rusage
strings defined for the job or application are merged with queue-level strings, with the job and then application definitions taking precedence over the queue if there is any overlap.
For internal load indices and duration, jobs are rejected if the merged job-level and apllication-level resource reservation requirements exceed the requirements specified at the queue level.
note:
If a compound resource requirement is used at one or more levels, (job, application, or queue) the compound rules apply.How compound multi-level resource requirements are resolved
Compound resource requirements can be specified at the job, application, and queue levels. When one or more of the resource requirements is compound, requirements at different levels are resolved depending on where the compound resource requirement appears.
For internal load indices and duration, jobs are rejected if they specify resource reservation requirements that exceed the requirements specified at the application level or queue level.
note:
If a compound resource requirement is used at one or more levels, (job, application, or queue) the compound rules apply.Compound queue level
When a compound resource requirement is set for a queue it will be ignored unless it is the only resource requirement specified (no resource requirements are set at the job level or application level).
Compound application level
When a compound resource requirement is set at the application level, it will be ignored if any job-level resource requirements (simple or compound) are defined.
In the event no job-level resource requirements are set, the compound application-level requirements interact with queue-level resource requirement strings in the following ways:
- If no queue-level resource requirement is defined or a compound queue-level resource requirement is defined, the compound application-level requirement is used.
- If a simple queue-level requirement is defined, the application-level and queue-level requirements combine as follows:
Compound job level
When a compound resource requirement is set at the job level, any simple or compound application-level resource requirements are ignored, and any compound queue-level resource requirements are ignored.
In the event a simple queue-level requirement appears along with a compound job-level requirement, the requirements interact as follows:
Example 1
A compound job requirement and simple queue requirement.
job level:
2*{select[type==X86_64] rusage[licA=1] span[hosts=1]} + 8*{select[type==any]}
application level: not defined
queue level:
rusage[perslot=1]
The final job scheduling resource requirement merges the simple queue-level rusage section into each term of the compound job-level requirement, resulting in:
2*{select[type==X86_64] rusage[licA=1:perslot=1] span[hosts=1]} + 8*{select[type==any] rusage[perslot=1]}
Example 2
A compound job requirement and compound queue requirement.
job level:
2*{select[type==X86_64 && tmp>10000] rusage[mem=1000] span[hosts=1]} + 8*{select[type==X86_64]}
application level: not defined
queue level:
2*{select[type==X86_64] rusage[mem=1000] span[hosts=1]} +8*{select[type==X86_64]}
The final job scheduling resource requirement ignores the compound queue-level requirement, resulting in:
2*{select[type==X86_64 && tmp>10000] rusage[mem=1000] span[hosts=1]} + 8*{select[type==X86_64]}
Example 3
A compound job requirement and simple queue requirement where the queue requirement is a job-based resource.
job level:
2*{select[type==X86_64]} + 2*{select[mem>1000]}
application level: not defined
queue level:
rusage[licA=1]
wherelicA=1
is job-based.The queue-level requirement is added to the first term of the compound job-level requirement, resulting in:
2*{select[type==X86_64] rusage[licA=1]} + 2*{select[mem>1000]}
Example 4
Compound multi-phase job requirements and simple multi-phase queue requirements.
job level:
2*{rusage[mem=(400 350):duration=(10 15):decay=(0 1)]} + 2*{rusage[mem=300:duration=10:decay=1]}
application level: not defined
queue level:
rusage[mem=(500 300):duration=(20 10):decay=(0 1)]
The queue-level requirement is overridden by the first term of the compound job-level requirement, resulting in:
2*{rusage[mem=(400 350):duration=(10 15):decay=(0 1)]} + 2*{rusage[mem=300:duration=10:decay=1]}
Selection String
The selection string specifies the characteristics a host must have to match the resource requirement. It is a logical expression built from a set of resource names. The selection string is evaluated for each host; if the result is non-zero, then that host is selected. When used in conjunction with a
cu
string, hosts not belonging to compute unit are not considered.Syntax
The selection string can combine resource names with logical and arithmetic operators. Non-zero arithmetic values are treated as logical TRUE, and zero (0) as logical FALSE. Boolean resources (for example,
server
to denote LSF server hosts) have a value of one (1) if they are defined for a host, and zero (0) if they are not defined for the host.The resource names
swap
,idle
,login
, andcpu
are accepted as aliases forswp
,it
,ls
, andr1m
respectively.The
ut
index measures CPU utilization, which is the percentage of time spent running system and user code. A host with no processes running has aut
value of 0 percent; a host on which the CPU is completely loaded has aut
of 100 percent. You must specifyut
as a floating-point number between 0.0 and 1.0.For the string resources
type
andmodel
, the special valueany
selects any value andlocal
selects the same value as that of the local host. For example,type==local
selects hosts of the same type as the host submitting the job. If a job can run on any type of host, includetype==any
in the resource requirements.If no
type
is specified, the default depends on the command. Forbsub
,lsplace
,lsrun
, andlsgrun
the default istype==local
unless a string or Boolean resource is specified, in which case it istype==any
. Forlshosts
,lsload
,lsmon
andlslogin
the default istype==any
.
tip:
WhenPARALLEL_SCHED_BY_SLOT=Y
inlsb.params
, the resource requirement string keywordncpus
refers to the number of slots instead of the number of processors, howeverlshosts
output will continue to showncpus
as defined byEGO_DEFINE_NCPUS
inlsf.conf
.Specifying multiple -R options
bsub
accepts multiple-R
options for the select section in simple resource requirements.
restriction:
Compound resource requirements do not support multiple-R
options.You can specify multiple resource requirement strings instead of using the && operator. For example:
bsub -R "select[swp > 15]" -R "select[hpux]"
LSF merges the multiple
-R
options into one string and dispatches the job if all of the resource requirements can be met. By allowing multiple resource requirement strings and automatically merging them into one string, LSF simplifies the use of multiple layers of wrapper scripts.When LSF_STRICT_RESREQ=Y is configured in
lsf.conf
, you cannot specify more than one select section in the same-R
option. Use the logical and (&&
) operator to specify multiple selection strings in the same select section. For example, the following command submits a job calledmyjob
to run on a host that has more than 15 MB of swap space available, and maximum RAM larger than 100MB. The job is expected to reserve 100MB memory on the host:% bsub -R "select [swp > 15 && maxmem > 100] rusage[mem = 100] " myjobThe number of
-R
option sections is unlimited.Selecting shared string resources
You must use single quote characters (
'
) around string-type shared resources. For example, uselsload -s
to see the shared resources defined for the cluster:lsload -s
RESOURCE VALUE LOCATION os_version 4.2 pc36 os_version 4.0 pc34 os_version 4.1 devlinux4 cpu_type ia pc36 cpu_type ia pc34 cpu_type unknown devlinux4Use a select string in
lsload -R
to specify the shared resources you want to view, enclosing the shared resource values in single quotes. For example:lsload -R "select[os_version=='4.2' || cpu_type=='unknown']"
HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem pc36 ok 0.0 0.2 0.1 1% 3.4 3 0 895M 517M 123M devlinux4 ok 0.0 0.1 0.0 0% 2.8 4 0 6348M 504M 205M
note:
When reserving resources based on host status (bsub -R "status==ok"
), the host status must be the one displayed by runningbhosts
notlsload
.Operators
These operators can be used in selection strings. The operators are listed in order of decreasing precedence.
Examples
select[(swp > 50 && type == MIPS) || (swp > 35 && type == ALPHA)] select[((2*r15s + 3*r1m + r15m) / 6 < 1.0) && !fs && (cpuf > 4.0)]Specifying shared resources with the keyword "defined"
A shared resource may be used in the resource requirement string of any LSF command. For example, when submitting an LSF job that requires a certain amount of shared scratch space, you might submit the job as follows:
bsub -R "avail_scratch > 200 && swap > 50" myjob
The above assumes that all hosts in the cluster have access to the shared scratch space. The job is only scheduled if the value of the
"avail_scratch"
resource is more than 200 MB and goes to a host with at least 50 MB of available swap space.It is possible for a system to be configured so that only some hosts within the LSF cluster have access to the scratch space. To exclude hosts that cannot access a shared resource, the
defined(
resource_name
)
function must be specified in the resource requirement string.For example:
bsub -R "defined(avail_scratch) && avail_scratch > 100 && swap > 100" myjob
would exclude any hosts that cannot access the scratch resource. The LSF administrator configures which hosts do and do not have access to a particular shared resource.
Supported resource names in the defined function
Only the following resource names are accepted as the argument in the
defined(
resource_name
)
function:
- The following builtin resource names:
LSF_Base lsf_base LSF_Manager lsf_manager LSF_JobScheduler lsf_js LSF_Make LSF_parallel LSF_Analyzer lsf_analyzerResource names configured inlsf.shared
,except
dynamic NUMERIC resource names with INTERVAL fields defined.The following resource names are
not
accepted in thedefined(
resource_name
)
function:
- The following builtin resource names:
r15s r1m r15m ut pg io ls it tmp swp mem ncpus ndisks maxmem maxswp maxtmp cpuf type model status rexpri server and hnameDynamic NUMERIC resource names configured inlsf.shared
with INTERVAL fields defined. In the default configuration, these aremode
,cntrl
,it_t
.)Other non-builtin resource names not configured lsf.shared
.Specifying exclusive resources
An exclusive resource may be used in the resource requirement string of any placement or scheduling command, such as
bsub
,lsplace
,lsrun
, orlsgrun
. An exclusive resource is a special resource that is assignable to a host. This host will not receive a job unless that job explicitly requests the host. For example, use the following command to submit a job requiring the exclusive resourcebigmem
:bsub -R "bigmem" myjob
Jobs will not be submitted to the host with the bigmem resource unless the command uses the
-R
option to explicitly specify"bigmem"
.To configure an exclusive resource, first define a static Boolean resource in
lsf.shared
. For example:Begin Resource ... bigmem Boolean () () End ResourceAssign the resource to a host in the Host section of
lsf.cluster.
cluster_name
. Prefix the resource name with an exclamation mark (!) to indicate that the resource is exclusive to the host. For example:Begin Host HOSTNAME model type server r1m pg tmp RESOURCES RUNWINDOW ... hostE ! ! 1 3.5 () () (linux !bigmem) () ... End HostStrict syntax for resource requirement selection strings
When LSF_STRICT_RESREQ=Y is configured in
lsf.conf
, resource requirement strings in select sections must conform to a more strict syntax. The strict resource requirement syntax only applies to theselect
section. It does not apply to the other resource requirement sections (order
,rusage
,same
,span
, orcu
). When LSF_STRICT_RESREQ=Y inlsf.conf
, LSF rejects resource requirement strings where anrusage
section contains a non-consumable resource.Strict syntax in EBNF form:
<expression> ::= <relation1> { <logical or> <relation1>} <relation1> ::=<relation2> { <logical and > <relation2>} <relation2> ::= <simple expression> [ <relation op> <simple expression> ] <simple expression> ::= <term> { <adding op> <term> } <term> ::= <factor> { <multiple op> <factor> } <factor> ::= [<unary op>] <primary> <primary> ::= <numeric> | <string> | ( <expression> )| <name or function call> <logical or> ::= || <logical and>::=&& <relation op> ::= <= | >= | == | != | < | > | = <adding op> ::= + | - <unary op> ::= - | ! <multiple op> ::= * | / <name or function call> ::= <name> [( <argument list> )] <argument list> ::= <empty> | <argument> {, <argument> } <argument> ::= <expression> <name> ::= [a-zA-Z_][a-zA-Z_0-9]* <numeric> ::= <int> [. [0-9]*] <int> ::= [1-9][0-9]* | 0 <string> ::= <single quote> {<string chars>} <single quote> | <double quote> {<string chars>} <double quote> <string chars> ::= <printable ascii characters except single/double quote> <single quote> ::= ' <double quote> ::= " <empty>=Strict select string syntax usage notes
The strict syntax is case sensitive.
Operators '
=
' and '==
' are equivalent.Boolean variables, such as
fs
,hpux
,cs
, can only be computed with the following operators:&& || !String variables, such as
type
, can only be computed with the following operators:= == != < > <= >=For function calls, blanks between the parentheses "
( )
" and the resource name are not valid. For example, the following is not correct:defined( mg )Multiple logical NOT operators (
!
) are not valid. For example, the following is not correct:!!mgThe following resource requirement is valid:
!(!mg)At least one blank space must separate each section. For example, the following are correct:
type==any rusage[mem=1024] select[type==any] rusage[mem=1024] select[type==any]rusage[mem=1024]but the following is not correct:
type==anyrusage[mem=1024]Only a single select section is supported by the stricter syntax. The following is not supported in the same resource requirement string:
select[mem>0] select[maxmem>0]Escape characters (like '
\n
') are not supported in string literals.A colon (
:
) is not allowed inside the select string. For example,select[mg:bigmem]
is not correct.
inf
andnan
can be used as resource names or part of a resource name.Single or double quotes are only supported around the whole resource requirement string, not within the square brackets containing the selection string. For example, in
lsb.queues
,RES_REQ='swp>100'
andRES_REQ="swp>100"
are correct. NeitherRES_REQ=select['swp>100']
norRES_REQ=select["swp>100"]
are supported.The following are correct
bsub
command-level resource requirements:
bsub -R "'swp>100'"
bsub -R '"swp>100"'
The following are not correct:
bsub -R "select['swp>100']"
bsub -R 'select["swp>100"]'
Some incorrect resource requirements are no longer silently ignored. For example, when LSF_STRICT_RESREQ=Y is configured in
lsf.conf
, the following are rejected by the resource requirement parser:
microcs73
is rejected:linux rusage[mem=16000] microcs73select[AMD64]
is rejected:mem < 16384 && select[AMD64]linux
is rejected:rusage[mem=2000] linuxUsing a colon (:) to separate select conditions, such as linux:qscw
.The restricted syntax of resource requirement select strings described in the lsfintro
(1) man page is not supported.Explicit and implicit select sections
An explicit select section starts from the section keyword and ends at the begin of next section, for example: the select section is
select[selection_string]
. An implicit select section starts from the first letter of the resource requirement string and ends at the end of the string if there are no other resource requirement sections. If the resource requirement has other sections, the implicit select section ends before the first letter of the first section following the selection string.All explicit sections must begin with a section keywords (
select
,order
,span
rusage
, orsame
). The resource requirement content is contained by square brackets ([
) and (]
).An implicit select section must be the first resource requirement string in the whole resource requirement specification. Explicit select sections can appear after other sections. A resource requirement string can have only one select section (either an explicit select section or an implicit select section). A section with an incorrect keyword name is not a valid section.
An implicit select section must have the same format as the content of an explicit select section. For example, the following commands are correct:
bsub -R "select[swp>15] rusage[mem=100]" myjob
bsub -R "swp > 15 rusage[mem=100]" myjob
bsub -R "rusage[mem=100] select[swp >15]" myjob
Examples
The following examples illustrate some correct resource requirement select string syntax.
bsub -R "(r15s * 2 + r15m) < 3.0 && !(type == IBMAIX4) || fs" myjob
- If swap space is equal to 0, the following means TRUE; if swap space is not equal to 0, it means FALSE:
bsub -R "!swp" myjobSelect hosts of the same type as the host submitting the job: bsub -R "type == local" myjobSelect hosts that are not the same type as the host submitting the job: bsub -R "type != local" myjobbsub -R "r15s < 1.0 || model ==local && swp <= 10" myjob
Since &&
has a higher priority than||
, this example means:r15s < 1.0 || (model == local && swp <=10)This example has different meaning from the previous example: bsub -R "(r15s < 1.0 || model == local) && swp <= 10" myjobThis example means: (r15s < 1.0 || model == local) && swp <= 10Checking resource requirement syntax
Use the BSUB_CHK_RESREQ environment variable to check the compatibility of your existing resource requirement select strings against the stricter syntax enabled by LSF_STRICT_RESREQ=Y in
lsf.conf
.Set the BSUB_CHK_RESREQ environment variable to any value enable
bsub
to check the syntax of the resource requirement selection string without actually submitting the job for scheduling and dispatch. LSF_STRICT_RESREQ does not need to be set to check the resource requirement selection string syntax.
bsub
only checks the select section of the resource requirement. Other sections in the resource requirement string are not checked.If resource requirement checking detects syntax errors in the selection string,
bsub
returns and error message. For example:bsub -R "select[type==local] select[hname=abc]" sleep 10 Error near "select": duplicate section. Job not submitted. echo $? 255If no errors are found,
bsub
returns a successful message and exit code zero (0). For example:env | grep BSUB_CHK_RESREQ BSUB_CHK_RESREQ=1 bsub -R "select[type==local]" sleep 10 Resource requirement string is valid. echo $? 0If BSUB_CHK_RESREQ is set, but you do not specify
-R
, LSF treats it as empty resource requirement. For example:bsub sleep 120 Resource requirement string is valid. echo $? 0Resizable jobs
Resize allocation requests are scheduled using hosts as determined by the
select
expression of the merged resource requirement. For example, to run an autoresizable job on 1-100 slots, but only on hosts of type X86_64, the following job submission specifies this resource request:bsub -ar -n "1,100" -R "select[type == X86_64]" myjobEvery time the job grows in slots, slots are requested on hosts of the specified type.
note:
Resizable jobs cannot have compound resource requirements.Order String
The order string allows the selected hosts to be sorted according to the values of resources. The values of
r15s
,r1m
, andr15m
used for sorting are the normalized load indices returned bylsload -N
.The order string is used for host sorting and selection. The ordering begins with the rightmost index in the order string and proceeds from right to left. The hosts are sorted into order based on each load index, and if more hosts are available than were requested, the LIM drops the least desirable hosts according to that index. The remaining hosts are then sorted by the next index.
After the hosts are sorted by the leftmost index in the order string, the final phase of sorting orders the hosts according to their status, with hosts that are currently not available for load sharing (that is, not in the
ok
state) listed at the end.Because the hosts are sorted again for each load index, only the host status and the leftmost index in the order string actually affect the order in which hosts are listed. The other indices are only used to drop undesirable hosts from the list.
When sorting is done on each index, the direction in which the hosts are sorted (increasing vs. decreasing values) is determined by the default order returned by
lsinfo
for that index. This direction is chosen such that after sorting, by default, the hosts are ordered from best to worst on that index.When used with a
cu
string, the preferred compute unit order takes precedence. Within each compute unit hosts are ordered according to theorder
string requirements.Syntax
[-]resource_name
[:[-]resource_name
]...You can specify any built-in or external load index or static resource.
When an index name is preceded by a minus sign `-', the sorting order is reversed so that hosts are ordered from worst to best on that index.
Specifying multiple -R options
bsub
accepts multiple-R
options for the order section.
restriction:
Compound resource requirements do not support multiple-R
options.You can specify multiple resource requirement strings instead of using the && operator. For example:
bsub -R "order[r15m]" -R "order[ut]"
LSF merges the multiple
-R
options into one string and dispatches the job if all of the resource requirements can be met. By allowing multiple resource requirement strings and automatically merging them into one string, LSF simplifies the use of multiple layers of wrapper scripts. The number of-R
option sections is unlimited.Default
The default sorting order is
r15s:pg
(except forlslogin
(1):ls:r1m
).Example
swp:r1m:tmp:r15sResizable jobs
The order in which hosts are considered for resize allocation requests is determined by the
order
expression of the job. For example, to run an autoresizable job on 1-100 slots, preferring hosts with larger memory, the following job submission specifies this resource request:bsub -ar -n "1,100" -R "order[mem]" myjobWhen slots on multiple hosts become available simultaneously, hosts with larger available memory get preference when the job adds slots.
note:
Resizable jobs cannot have compound resource requirements.Usage String
This string defines the expected resource usage of the job. It is used to specify resource reservations for jobs, or for mapping jobs on to hosts and adjusting the load when running interactive jobs.
By default, no resources are reserved.
When LSF_STRICT_RESREQ=Y in
lsf.conf
, LSF rejects resource requirement strings where an rusage section contains a non-consumable resource.Multi-phase resources
Multiple phases within the
rusage
string allow different time periods to have different memory requirements (load indexmem
). The duration of all except the last phase must be specified, while decay rates are all optional and are assumed to be 0 if omitted. If the optional final duration is left blank, the final resource requirement applies until the job is finished.Multi-phase resource reservations cannot include increasing resources, but can specify constant or decreasing resource reservations over multiple periods of time.
Resource reservation limits
Resource requirement reservation limits can be set using the parameter
RESRSV_LIMIT
inlsb.queues
. Queue-levelRES_REQ
rusage values (set inlsb.queues
) must be in the range set byRESRSV_LIMIT
, or the queue-levelRES_REQ
is ignored. MergedRES_REQ
rusage values from the job and application levels must be in the range ofRESRSV_LIMIT
, or the job is rejected.When both the
RES_REQ
andRESRSV_LIMIT
are set inlsb.queues
for a consumable resource, the queue-levelRES_REQ
no longer acts as a hard limit for the mergedRES_REQ
rusage
values from the job and application levels. In this case only the limits set byRESRSV_LIMIT
must be satisfied, and the queue-level
RES_REQ
acts as a default value.
Batch jobs
The resource usage (
rusage
) section can be specified at the job level, with the queue configuration parameter RES_REQ, or with the application profile parameter RES_REQ.Basic syntax
rusage[usage_string
[,usage_string
][||usage_string
] ...]where
usage_string
is:load_index
=value
[:load_index
=value
]... [:duration=minutes
[m]| :duration=
hours
h | :duration=seconds
s [:decay=0 | :decay=1]]
Multi-phase memory syntax
rusage[multi_usage_string
[,usage_string
]...]where
multi_usage_string
is:mem=(v1
[v2
...vn
]):[duration=(t1
[t2
...tm
])][:decay=(d1
[d2
...dk
])]for
m = n|n-1
. For a single phase (n=1
), duration is not required.if
k
>m
,dm+1
todk
will be ignored; ifk
<m
,dk+1
=.. =dm
= 0.
usage_string
is the same as the basic syntax, for anyload_index
other thanmem
.Multi-phase syntax can be used with a single phase memory resource requirement as well as for multiple phases. For multi-phase slot-based resource reservation, use with
RESOURCE_RESERVE_PER_SLOT=Y
inlsb.params
.Multi-phase resource reservations cannot increase over time. A job submission with increasing resource reservations from one phase to the next will be rejected. For example:
bsub -R"rusage[mem=(200 300):duration=(2 3)]" myjob
specifies an increasing memory reservation from 200 MB to 300 MB. This job will be rejected.
tip:
When a multi-phasemem
resource requirement is being used,duration
can be specified separately for single-phase resources.Load index
Internal and external load indices are considered in the resource usage string. The resource value represents the initial reserved amount of the resource.
Duration
The duration is the time period within which the specified resources should be reserved. Specify a duration equal to or greater than the ELIM updating interval.
- If the value is followed by the letter
s
,m
, orh
, the specified time is measured in seconds, minutes, or hours respectively.- By default, duration is specified in minutes.
- For example, the following specify a duration of 1 hour for multi-phase syntax:
duration=(60)
duration=(1h)
duration=(3600s)
- For example, the following specify a duration of 1 hour for single-phase syntax:
duration=60
duration=1h
duration=3600s
tip:
Duration is not supported for static shared resources. If the shared resource is defined in an lsb.resources Limit section, then duration is not applied.Decay
The decay value indicates how the reserved amount should decrease over the duration.
- A value of 1 indicates that system should linearly decrease the amount reserved over the duration.
- A value of 0 causes the total amount to be reserved for the entire duration.
Values other than 0 or 1 are unsupported, and are taken as the default value of 0. If duration is not specified, decay value is ignored.
tip:
Decay is not supported for static shared resources. If the shared resource is defined in an lsb.resources Limit section, then decay is not applied.Default
If a resource or its value is not specified, the default is not to reserve that resource. If duration is not specified, the default is to reserve the total amount for the lifetime of the job. (The default decay value is 0.)
Example
rusage[mem=50:duration=100:decay=1]This example indicates that 50 MB memory should be reserved for the job. As the job runs, the amount reserved will decrease at approximately 0.5 MB per minute until the 100 minutes is up.
How simple queue-level and job-level rusage sections are resolved
Job-level rusage overrides the queue level specification:
- For internal load indices (
r15s
,r1m
,r15m
,ut
,pg
,io
,ls
,it
,tmp
,swp
, andmem
), the job-level value cannot be larger than the queue-level value (unless the limit parameterRESRSV_LIMIT
is being used as a maximum instead of the queue-level value).- For external load indices (e.g., licenses), the job-level rusage can be larger than the queue-level requirements.
- For duration, the job-level value of internal and external load indices cannot be larger than the queue-level value.
- For multi-phase simple rusage sections:
- For internal load indices (
r15s
,r1m
,r15m
,ut
,pg
,io
,ls
,it
,tmp
,swp
, andmem
), the first phase of the job-level value cannot be larger than the first phase of the queue-level value (unless the limit parameterRESRSV_LIMIT
is being used as a maximum instead of the queue-level value).- For duration and decay, if either job-level or queue-level is multi-phase, the job-level value will take precedence.
How simple queue-level and job-level rusage sections are merged
When both job-level and queue-level
rusage
sections are defined, therusage
section defined for the job overrides therusage
section defined in the queue. The tworusage
definitions are merged, with the job-levelrusage
taking precedence. For example:Example 1
Given a RES_REQ definition in a queue:
RES_REQ = rusage[mem=200:lic=1] ...and job submission:
bsub -R "rusage[mem=100]" ...The resulting requirement for the job is
rusage[mem=100:lic=1]where
mem=100
specified by the job overridesmem=200
specified by the queue. However,lic=1
from queue is kept, since job does not specify it.Example 2
For the following queue-level RES_REQ (decay and duration defined):
RES_REQ = rusage[mem=200:duration=20:decay=1] ...and job submission (no decay or duration):
bsub -R "rusage[mem=100]" ...The resulting requirement for the job is:
rusage[mem=100:duration=20:decay=1]Queue-level duration and decay are merged with the job-level specification, and
mem=100
for the job overridesmem=200
specified by the queue. However,duration=20
anddecay=1
from queue are kept, since job does not specify them.rusage in application profiles
See Chapter 23, "Working with Application Profiles" for information about how resource requirements in application profiles are resolved with queue-level and job-level resource requirements.
How simple queue-level rusage sections are merged with compound rusage sections
When simple queue-level and compound application-level or job-level
rusage
sections are defined, the tworusage
definitions are merged. If a job-level resource requirement (simple or compound) is defined, the application level is ignored and the job-level and queue-level sections merge. If no job-level resource requirement is defined, the application-level and queue-level merge.When a compound resource requirement merges with a simple resource requirement from the queue-level, the behavior depends on whether the queue-level requirements are job-based or not.
Example 1
Job-based simple queue-level requirements apply to the first term of the merged compound requirements. For example:
Given a
RES_REQ
definition for a queue which refers to a job-based resource:RES_REQ = rusage[lic=1] ...and job submission resource requirement:
bsub -R "2*{rusage[mem=100] ...} + 4*{[mem=200:duration=20:decay=1] ...}"The resulting requirement for the job is
bsub -R "2*{rusage[mem=100:lic=1] ...} + 4*{[mem=200:duration=20:decay=1] ...}"
The job-based resource
lic=1
from queue is added to the first term only, since it is job-based and wasn't included the job-level requirement.Example 2
Host-based or slot-based simple queue-level requirements apply to all terms of the merged compound requirements. For example:
For the following queue-level RES_REQ which does not include job-based resources:
RES_REQ = rusage[mem=200:duration=20:decay=1] ...and job submission:
bsub -R "2*{rusage[mem=100] ...} + 4*{rusage[lic=1] ...}"The resulting requirement for the job is:
2*{rusage[mem=100:duration=20:decay=1] ...} + 4*{rusage[lic=1:mem=200:duration=20:decay=1] ...}Where
duration=20
anddecay=1
from queue are kept, since job does not specify them in any term. In the first termmem=100
from the job is kept; in the second termmem=200
from the queue is used since it wasn't specified by the job resource requirement.Specifying multiple -R options
bsub
accepts multiple-R
options for the rusage section.
restriction:
Compound resource requirements do not support multiple-R
options. Multi-phase rusage strings do not support multiple-R
options.You can specify multiple resource requirement strings instead of using the && operator. For example:
bsub -R "rusage[mem=100]" -R "rusage[tmp=50:duration=60]"
LSF merges the multiple
-R
options into one string and dispatches the job if all of the resource requirements can be met. By allowing multiple resource requirement strings and automatically merging them into one string, LSF simplifies the use of multiple layers of wrapper scripts.Comma-separated multiple resource requirements within one
rusage
string is supported. For example:
bsub -R "rusage[mem=20, license=1:duration=2]"
-R "rusage[app_lic_v201=1||app_lic_v15=1]" myjobA given load index cannot appear more than once in the resource usage string.
Examples
- The following job requests 20 MB memory for the duration of the job, and 1 license to be reserved for 2 minutes:
bsub -R "rusage[mem=20, license=1:duration=2]" myjob
A queue with the same resource requirements could specify: RES_REQ = rusage[mem=20, license=1:duration=2]The following job requests 20 MB of memory and 50 MB of swap space for 1 hour, and 1 license to be reserved for 2 minutes: bsub -R "rusage[mem=20:swp=50:duration=1h, license=1:duration=2]" myjob
The following job requests 50 MB of swap space, linearly decreasing the amount reserved over a duration of 2 hours, and requests 1 license to be reserved for 2 minutes: bsub -R "rusage[swp=20:duration=2h:decay=1, license=1:duration=2]" myjob
The following job requests two resources with same duration but different decay: bsub -R "rusage[mem=20:duration=30:decay=1, lic=1:duration=30] myjob
Specifying alternative usage strings
If you use more than one version of an application, you can specify the version you prefer to use together with a legacy version you can use if the preferred version is not available. Use the
OR
(||
) expression to separate the different usage strings that define your alternative resources.Job-level resource requirement specifications that use the
||
operator are merged with otherrusage
requirements defined at the application and queue levels.
note:
Alternativerusage
strings cannot be submitted with compound resource requirements.Job-level examples
The following examples assume that you are running an application version 1.5 as a resource called
app_lic_v15
and the same application version 2.0.1 as a resource calledapp_lic_v201
. The license key for version 2.0.1 is backward compatible with version 1.5, but the license key for version 1.5 will not work with 2.0.1
- If you can only run your job using version 2.0.1 of the application, submit the job without specifying an alternate resource. To submit a job that will only use
app_lic_v201
:bsub -R "rusage[app_lic_v201=1]" myjob
If you can run your job using either version of the application, try to reserve version 2.0.1 of the application. If it is not available, you can use version 1.5. To submit a job that will try app_lic_v201
before tryingapp_lic_v15
:bsub -R "rusage[app_lic_v201=1||app_lic_v15=1]" myjob
If different versions of an application require different system resources, you can specify other resources in your rusage
strings. To submit a job that will use 20 MB of memory forapp_lic_v201
or 20 MB of memory and 50 MB of swap space forapp_lic_v15
:bsub -R "rusage[mem=20:app_lic_v201=1||mem=20:swp=50:app_lic_v15=1]" myjob
You can also specify alternative multi-phase memory requirements. To submit a job that will use 20 MB of memory for app_lic_v201
or 50 MB of swap space and 50 MB of memory for 10 minutes followed by 10 MB of memory for the remainder of the job forapp_lic_v15
:bsub -R "rusage[mem=20:app_lic_v201=1
||mem=(50 10):duration=(10),swp=50:app_lic_v15=1]" myjob
How LSF merges rusage strings that contain the || operator
The following examples show how LSF merges job-level and queue-level
rusage
strings that contain the||
operator.
note:
Alternativerusage
strings cannot be submitted with compound resource requirements.Non-batch environments
Resource reservation is only available for batch jobs. If you run jobs using only LSF Base, such as through
lsrun
, LIM uses resource usage to determine the placement of jobs. Resource usage requests are used to temporarily increase the load so that a host is not overloaded. When LIM makes a placement advice, external load indices are not considered in the resource usage string. In this case, the syntax of the resource usage string isres[=value]:res[=value]: ... :res[=value]
res
is one of the resources whose value is returned by the lsload command.rusage[r1m=0.5:mem=20:swp=40]The above example indicates that the task is expected to increase the 1-minute run queue length by 0.5, consume 20 MB of memory and 40 MB of swap space.
If no value is specified, the task is assumed to be intensive in using that resource. In this case no more than one task will be assigned to a host regardless of how many CPUs it has.
The default resource usage for a task is
r15s=1.0:r1m=1.0:r15m=1.0
. This indicates a CPU-intensive task which consumes few other resources.Resizable jobs
Unlike the other components of a resource requirement string that only pertain to adding additional slots to a running job,
rusage
resource requirement strings affect the resource usage when slots are removed from the job as well.When adding or removing slots from a running job:
- The amount of
slot-based
resources added to or removed from the job allocation is proportional to the change in the number of slots- The amount of
job-based
resources is not affected by a change in the number of slots- The amount of each
host-based
resource is proportional to the change in the number of hostsWhen using multi-phase resource reservation, the job allocation is based on the phase of the resource reservation.
note:
Resizable jobs cannot have compound resource requirements.Duration and decay of rusage
Duration and decay of resource usage and the || operator affect resource allocation.
Duration or decay of a resource in the
rusage
expression is ignored when scheduling the job for the additional slots.If a job has the following
rusage
string:rusage[mem=100:duration=300]
, the resize request of one additional slot is scheduled on a host only if there are 100 units of memory available on that host. In this case,mem
is a slot-based resource (RESOURCE_RESERVE_PER_SLOT=Y inlsb.params
).Once the resize operation is done, if the job has been running less than 300 seconds then additional memory will be reserved only until the job has run for 300 seconds. If the job has been running for more than 300 seconds when the job is resized, no additional memory is reserved. The behavior is similar for decay.
The || operator lets you specify multiple alternative rusage strings, one of which is used when dispatching the job. You cannot use
bmod
to change rusage to a new one with a || operator after the job has been dispatchedFor job resize, when the || operator is used, the resize request uses the
rusage
expression that was originally used to dispatch the job. If therusage
expression has been modified since the job started, the resize request is scheduled using the new singlerusage
expression.Example 1
You want to run an autoresizable job such that every slot occupied by the job reserves 100 MB of swap space In this case,
swp
is a slot-based resource (RESOURCE_RESERVE_PER_SLOT=Y inlsb.params
). The job also needs a separate license for each host on which it runs. Each additional slot allocated to the job should reserve additional swap space, and each new host should reserve an additional license. The following job submission specifies this resource request:bsub -ar -n "1,100" -R "rusage[swp=100,license=1]" myjobwhere
license
is a user-defined host-based resource.Similarly, if you want to release some of the slots from a running job, resources reserved by the job are decreased appropriately. For example, for the following job submission:
bsub -ar -n 100 -R "rusage[swp=50:license=1]" myjob Job <123> is submitted to default queue.you can run
bresize release
to release all the slots from the job on one host:bresize release "hostA" 123The swap space used by the job is reduced by the number of slots used on
hostA
times 50 MB, and one host-basedlicense
resource is released from the job.Example 2
You have a choice between two versions of an application, each version having different memory and swap space requirements on hosts and a different license (
app_lic_v15
andapp_lic_v201
). If you submit an autoresizable job with the || operator, once the job is started using one version of an application, slots added to a job during a resize operation reserve resources depending on which version of the application was originally run. For example, for the following job submission:bsub -n "1,100" -ar -R "rusage[mem=20:app_lic_v201=1 || mem=20:swp=50:app_lic_v15=1]" myjobIf the job starts with
app_lic_v15
, each additional slot added in a resize operation reserves 20 MB of memory and 50 MB of swap space.Span String
A
span
string specifies the locality of a parallel job. If span is omitted, LSF allocates the required processors for the job from the available set of processors.Syntax
The
span
string supports the following syntax:span[hosts=1]
Indicates that all the processors allocated to this job must be on the same host.
span[ptile=
value
]Indicates the number of processors on each host that should be allocated to the job, where
value
is one of the following:
- Default
ptile
value, specified byn
processors. In the following example, the job requests 4 processors on each available host, regardless of how many processors the host has:span[ptile=4]Predefined ptile
value, specified by '!'. The following example uses the predefined maximum job slot limitlsb.hosts
(MXJ per host type/model) as its value:span[ptile='!']
tip:
If the host or host type/model does not define MXJ, the default predefined ptile value is 1.
restriction:
Under bash 3.0, the exclamation mark (!) is not interpreted correctly by the shell. To use predefined ptile value (ptile='!'), use the +H option to disable '!' style history substitution in bash (sh +H).Predefined ptile
value with optional multipleptile
values, per host type or host model:
- For host type, you must specify
same[type]
in the resource requirement. In the following example, the job requests 8 processors on a host of typeHP
orSGI
, and 2 processors on a host of typeLINUX
, and the predefined maximum job slot limit inlsb.hosts
(MXJ) for other host types:span[ptile='!',HP:8,SGI:8,LINUX:2] same[type]For host model, you must specify same[model]
in the resource requirement. In the following example, the job requests 4 processors on hosts of modelPC1133
, and 2 processors on hosts of model PC233, and the predefined maximum job slot limit inlsb.hosts
(MXJ) for other host models:span[ptile='!',PC1133:4,PC233:2] same[model]span[hosts=-1]
Disables span setting in the queue. LSF allocates the required processors for the job from the available set of processors.
See Controlling Processor Allocation Across Hosts for more information about specifying
span
strings.Resizable jobs
For resource requirements with
span[hosts=1]
, a resize request is limited to slots on the first-execution host of the job. This behavior eliminates the ambiguities that arise when the span expression is modified from the time that the job was originally dispatched.For
span[ptile=n]
, the job will be allocated exactlyn
slots on some number of hosts, and a number between 1 andn
slots (inclusive) on one host. This is true even if a range of slots is requested. For example, for the following job submission:bsub -n "1,20" -R "span[ptile=2]" sleep 10000This special span behavior does not only apply to resize requests. It applies to resizable jobs only when the original allocation is made, and in making additional resize allocations.
If every host has only a single slot available, the job is allocated one slot.
Resize requests with partially filled hosts are handled so that LSF does not choose any slots on hosts already occupied by the job. For example, it is common to use the
ptile
feature withspan[ptile=1]
to schedule exclusive jobs. Another typical use isspan[ptile='!']
to make the job occupy all slots on each host it is allocated.For a resizable job (auto-resizable or otherwise) with a range of slots requested and
span[ptile=n]
, whenever the job is allocated slots, it will receive either of the following:
- The maximum number of slots requested, comprising
n
slots on each of a number of hosts, and between 0 andn
-1 (inclusive) slots on one hostn
slots on each of a number of hosts, summing to some value less than the maximumFor example, if a job requests between 1 and 14 additional slots, and
span[ptile=4]
is part of the job resource requirement string, when additional slots are allocated to the job, the job receives either of the following:
- 14 slots, with 2 slots on one host and 4 slots on each of 3 hosts
- 4, 8 or 12 slots, such that 4 slots are allocated per host of the allocation
note:
Resizable jobs cannot have compound resource requirements.Example
When running a parallel exclusive job, it is often desirable to specify
span[ptile=1]
so that the job is allocated at most one slot on each host. For an autoresizable job, new slots are allocated on hosts not already used by the job. The following job submission specifies this resource request:bsub -x -ar -n "1,100" -R "span[ptile=1]" myjobWhen additional slots are allocated to a running job, the slots will be on new hosts, not already occupied by the job.
Same String
tip:
You must have the parallel batch job scheduler plugin installed in order to use the same string.Parallel jobs run on multiple hosts. If your cluster has heterogeneous hosts, some processes from a parallel job may for example, run on Solaris and some on SGI IRIX. However, for performance reasons you may want all processes of a job to run on the same type of host instead of having some processes run on one type of host and others on another type of host.
The
same
string specifies that all processes of a parallel job must run on hosts with the same resource.You can specify the
same
string:
- At the job level in the resource requirement string of:
bsub
bmod
- At the queue level in
lsb.queues
in the RES_REQ parameter.When queue-level, application-level, and job-level
same
sections are defined, LSF combines requirements to allocate processors.Syntax
resource_name
[:resource_name
]...You can specify any static resource.
For example, if you specify
resource1:resource2
, if hosts always have both resources, the string is interpreted as allocate processors only on hosts that have the same value forresource1
and the same value forresource2
.If hosts do not always have both resources, it is interpreted as allocate processors either on hosts that have the same value for
resource1
, or on hosts that have the same value forresource2
, or on hosts that have the same value for bothresource1
andresource2
.Specifying multiple -R options
bsub
accepts multiple-R
options for the same section.
restriction:
Compound resource requirements do not support multiple-R
options.You can specify multiple resource requirement strings instead of using the && operator. For example:
bsub -R "same[type]" -R "same[model]"
LSF merges the multiple
-R
options into one string and dispatches the job if all of the resource requirements can be met. By allowing multiple resource requirement strings and automatically merging them into one string, LSF simplifies the use of multiple layers of wrapper scripts.Resizable jobs
The
same
expression ensures that the resize allocation request is dispatched to hosts that have the same resources as the first-execution host. For example, if the first execution host of a job is SOL7 and the resource requirement string containssame[type]
, additional slots are allocated to the job on hosts of type SOL7.Taking the same resource as the first-execution host avoids ambiguities that arise when the original job does not have a
same
expression defined, or has a differentsame
expression when the resize request is scheduled.For example, a parallel job may be required to have all slots on hosts of the same type or model for performance reasons. For an autoresizable job, any additional slots given to the job will be on hosts of the same type, model, or resource as those slots originally allocated to the job. The following command submits an autoresizable job such that all slots allocated in a resize operation are allocation on hosts with the same model as the original job:
bsub -ar -n "1,100" -R "same[model]" myjobExamples
bsub -n 4 -R"select[type==SGI6 || type==SOL7] same[type]" myjob
Run all parallel processes on the same host type. Allocate 4 processors on the same host type-either SGI IRIX, or Solaris 7, but not both.
bsub -n 6 -R"select[type==any] same[type:model]" myjob
Run all parallel processes on the same host type and model. Allocate 6 processors on any host type or model as long as all the processors are on the same host type and model.
Same string in application profiles
See Chapter 23, "Working with Application Profiles" for information about how resource requirements in application profiles are resolved with queue-level and job-level resource requirements.
Compute Unit String
A
cu
string specifies the network architecture-based requirements of parallel jobs.cu
sections are accepted bybsub -R
, and bybmod -R
for non-running jobs.Compute unit resource requirements are not supported in compound resource requirements.
For a complete description of compute units see Controlling Job Locality using Compute Units in Chapter 34, "Running Parallel Jobs".
Syntax
The
cu
string supports the following syntax:cu[type=
cu_type
]Indicates the type of compute units the job can run on. Types are defined by
COMPUTE_UNIT_TYPES
inlsb.params
. Iftype
is not specified, the default set byCOMPUTE_UNIT_TYPES
is assumed.cu[pref=maxavail | minavail | config]
Indicates the compute unit scheduling preference, grouping hosts by compute unit before applying a first-fit algorithm to the sorted hosts. For resource reservation, the default
pref
=config
is always used.Compute units are ordered as follows:
config
lists compute units in the order they appear in theComputeUnit
section oflsf.hosts
. Ifpref
is not specified,pref
=config
is assumed.maxavail
lists compute units with more free slots first. Should compute units have equal numbers of free slots, they appear in the order listed in theComputeUnit
section oflsf.hosts
.minavail
lists compute units with fewer free slots first. Should compute units have equal numbers of free slots, they appear in the order listed in theComputeUnit
section oflsf.hosts
.Free slots include all available slots not occupied by running jobs.
When
pref
is used with the keywordbalance
,balance
takes precedence.Hosts accept jobs separated by the time interval set by
JOB_ACCEPT_INTERVAL
inlsb.params
; jobs submitted closer together than this interval will run on different hosts regardless of thepref
setting.cu[maxcus=
number
]Indicates the maximum number of compute units a job can run over. Jobs may be placed over fewer compute units if possible.
When used with
bsub -n
min
,
max
a job is allocated the first combination satisfying bothmin
andmaxcus
, while withoutmaxcus
a job is allocated as close tomax
as possible.cu[usablecuslots=
number
]Specifies the minimum number of slots a job must use on each compute unit it occupies.
number
is a non-negative integer value.When more than one compute unit is used by a job, the final compute unit allocated can provide less than
number
slots if less are needed.usablecuslots
andbalance
cannot be used together.cu[balance]
Indicates that a job should be split evenly between compute units, with a difference in compute unit slot allocation of at most 1. A balanced allocation spans the fewest compute units possible.
When used with
bsub -n
min
,
max
the value ofmax
is disregarded.balance
andusablecuslots
cannot be used together.When
balance
andpref
are both used,balance
takes precedence. The keywordpref
is only considered if there are multiple balanced allocations spanning the same number of compute units. In this casepref
is considered when choosing the allocation.When
balance
is used withspan[ptile=X]
(forX>1
) a balanced allocation is one split evenly between compute units, with a difference in compute unit host allocation of at most 1.cu[excl]
Indicates that jobs must use compute units exclusively. Exclusivity applies to the compute unit granularity specified by
type
.Compute unit exclusivity must be enabled by
EXCLUSIVE=CU
[cu_type
] inlsb.queues
.Resizable jobs
Auto-resizable jobs cannot be submitted with compute unit resource requirements. In the event a
bswitch
call or queue reconfiguration results in an auto-resizable job running in a queue with compute unit resource requirements, the job will no longer be auto-resizable.
restriction:
Increasing resources allocated to resizable jobs with compute unit resource requirements is not supported.Examples
bsub -n 11,60 -R "cu[maxcus=2:type=enclosure]" myjob
Spans the fewest possible compute units for a total allocation of at least 11 slots using at most 2 compute units of type enclosure. In contrast, without maxcus:
bsub -n 11,60 myjob
In this case the job is allocated as close to 60 slots as possible, with a minimum of 11 slots.
bsub -n 64 -R "cu[balance:maxcus=4:type=enclosure]" myjob
Spans the fewest possible compute units for a balanced allocation of 64 slots using 4 or less compute units of type enclosure.Possible balanced allocations (in order of preference) are:
- 64 slots on 1 enclosure
- 32 slots on 2 enclosures
- 22 slots on 1 enclosure and 21 slots on 2 enclosures
- 16 slots on 4 enclosures
bsub -n 64 -R "cu[excl:maxcus=8:usablecuslots=10]" myjob
Allocates 64 slots over 8 or less compute units in groups of 10 or more slots per compute unit (with one compute unit possibly using less than 10 slots). The default compute unit type set in
COMPUTE_UNIT_TYPES
is used, and are used exclusively bymyjob
.
bsub -n 58 -R "cu[balance:type=rack:usablecuslots=20]" myjob
Provides a balanced allocation of 58 slots with at least 20 slots in each compute unit of type rack. Possible allocations are 58 slots in 1 rack or 29 slots in 2 racks.
Jobs submitted with
balance
requirements choose compute units based on thepref
keyword secondarily, as shown in the following examples wherecu1
has 5 available slots andcu2
has 19 available slots.
bsub -n 5 -R "cu[balance:pref=minavail]"
Runs the job on compute unit
cu1
where there are the fewest available slots.
bsub -n 5 -R "cu[balance:pref=maxavail]"
Runs the job on compute unit
cu2
where there are the most available slots. In both cases the job is balanced over the fewest possible compute units.Cu string in application profiles
See Chapter 23, "Working with Application Profiles" for information about how resource requirements in application profiles are resolved with queue-level and job-level resource requirements.
Platform Computing Inc.
www.platform.com |
Knowledge Center Contents Previous Next Index |