[ Platform Documentation ] [ Title ] [ Contents ] [ Previous ] [ Next ] [ Index ]
- About Platform LSF HPC for Linux/QsNet
- Configuring Platform LSF HPC for Linux/QsNet
- Operating Platform LSF HPC for Linux/QsNet
- Submitting and Monitoring Jobs
[ Top ]
About Platform LSF HPC for Linux/QsNet
- What Platform LSF HPC for Linux/QsNet does
- Assumptions and limitations
- Compatibility with earlier releases
What Platform LSF HPC for Linux/QsNet does
The Platform LSF HPC for Linux/QsNet combines the strengths of Platform LSF HPC, Quadrics Resource Management System (RMS), and Quadrics QsNet data network to provide a comprehensive Distributed Resource Management (DRM) solution on Linux.
LSF HPC acts primarily as the workload scheduler, providing policy and topology-based scheduling and fault tolerance. RMS acts as a parallel execution subsystem for CPU allocation and node selection.
Assumptions and limitations
- A single parallel LSF job must run within a single RMS partition.
- LSF uses its own access control, usage limits and accounting mechanism. You should not change the default RMS configuration for these features. Configuration changes may interfere with the correct operation of LSF. Do not use the commands or configure any of the following RMS features:
- If you use RMS_MCONT or RMS_SNODE allocation options, the
ptile
option in the span section of the resource requirement string (bsub -R "span[ptile=
n]"
) is not supported.You should use
-extsched "RMS[ptile=n]"
to define the locality of jobs instead of-R "span[ptile=n]"
.- Host preference (for example,
bsub -m hostA
) is only supported for RMS_SLOAD allocation. LSF host preference is not taken into account for RMS_SNODE and RMS_MCONT allocation.
Use RMS_SLOAD on smaller clusters, where the job placement decision should be influenced by host load, or where you want to keep a specific host preference.
- Using an exclamation point (
!
) in host selection to indicate mandatory first execution host (e.g.,bsub -m "hostA! hostB"
) is supported for RMS_SNODE and RMS_MCONT allocation. If you specify RMS_SLOAD with mandatory first execution host, LSF changes the allocation type to RMS_SNODE. Hosts are sorted by their position in the RMS partition; any host to the left of the first execution host is ignored.- Resizable jobs are not supported.
- Application-level checkpointing is supported.
- User-level checkpointing is not supported.
- Kernel-level checkpointing is not available on Linux/QsNet systems.
- When LSF selects RMS jobs to preempt, jobs to be preempted are selected from the list of preemptable candidates based on the topology-aware allocation algorithm.
Allocation always starts from the smallest numbered node on the LSF node and works from this node up. Some specialized preemption preferences, such as MINI_JOB and LEAST_RUN_TIME in the PREEMPT_FOR parameter in
lsb.params
, and others are ignored when slot preemption is required.- Preemptable queue preference is not supported.
- Jobs submitted to a chunk job queue are not chunked together, but run as a normal LSF job.
- User-level account mapping is not supported.
- By default, job start time is not accurately predicted for jobs reserving resources with topology requirements, so the forecast start time shown by
bjobs -l
is optimistic. LSF HPC may incorrectly indicate that the job can start at a certain time, when it actually cannot start until some time after the indicated time.For a more accuration start-time estimate, you should configure time-based slot reservation. With time-based reservation, a set of pending jobs will get future allocation and estimated start time.
See Administering Platform LSF for more information about time-based slot reservation.
- When a partition is blocked or down, the status of
prun
jobs becomes UNKNOWN, andbjobs
shows the jobs as still running.- The administrator must use
brun -c
to force a job to run on an RMS host. If the RMS allocation cannot be satisfied for any reason, the job will be dispatched, but will be requeued and returned to pending state. The administrator can usebrun -c
again to start the job.- In MultiCluster lease model, you should export entire LSF hosts.
Compatibility with earlier releases
In this version of Platform LSF HPC for Linux for Linux/QsNet:
- The topology scheduler options (
bsub -extsched
or queue-level DEFAULT_EXTSCHED and MANDATORY_EXTSCHED) no longer determine the host type for the job. Regular Platform LSF resource requirements (bsub -R
or queue-level RES_REQ) or host name selection (bsub -m
) dictate where the job will run.- The default RMS allocation type is RMS_SNODE.
- Node-level allocation parameters LSB_RLA_POLICY and LSB_RMS_NODESIZE in
lsf.conf
are no longer needed. Use theRMS[ptile=
cpus_per_node]
option to specify node-level allocation.- For topology scheduler options that use the obsolete syntax:
- Only the RMS allocation options (RMS_MCONT, RMS_SLOAD, RMS_SNODE) are honoured
- Options from the queue and from
bsub
are parsed separately before they are merged. For example:bsub -extsched "ptile=1" ...and
DEFAULT_EXTSCHED=RMS_SNODE
in the queueBefore Version 7, the net effect is:
RMS_SNODE;ptile=1In Version 7,
ptile=1
is not an RMS allocation option, so it is ignored and the net result is:RMS_SNODE
- The following
install.config
options are obsolete. You do not need to specify them when runninglsfinstall
:- For job control action configuration, the
batchid
in the RMSrcontrol
command must include the LSF cluster name. Thercontrol
command now has the form:rcontrol [suspend | kill | resume] batchid=cluster_name@$LSB_JOBID- By default, the LSF scheduler can contact the LSF topology adapter (RLA) running on any host for RMS allocation requests. Use LSB_RLA_HOST_LIST in
lsf.conf
to define a list of hosts to restrict which RLAs the LSF scheduler contacts.[ Top ]
Configuring Platform LSF HPC for Linux/QsNet
- Automatic configuration at installation
- Setting dedicated LSF partitions (recommended)
- Customizing job control actions (optional)
- Configuration notes
Automatic configuration at installation
For the default host,
lsfinstall
enables "!" in the MXJ column of the HOSTS section oflsb.hosts
. For example:Begin Host HOST_NAME MXJ r1m pg ls tmp DISPATCH_WINDOW # Keywords #hostA () 3.5/4.5 15/ 12/15 0 () # Example default ! () () () () () End HostDuring installation,
lsfinstall
adds theschmod_rms
external scheduler plugin module name to the PluginModule section oflsb.modules
to enable the RMS scheduler plugin module:Begin PluginModule SCH_PLUGIN RB_PLUGIN SCH_DISABLE_PHASES schmod_default () () schmod_fcfs () () schmod_fairshare () () schmod_limit () () schmod_preemption () () ... schmod_rms () () End PluginModule
The schmod_rms plugin name must be configured after the standard LSF plugin names in the PluginModule list.
See the Platform LSF Configuration Reference for more information about
lsb.modules
.During installation, LSF HPC defines a queue named
rms
inLSB_CONFDIR/lsb.queues
for RMS jobs running in LSF HPC.Begin Queue QUEUE_NAME = rms PJOB_LIMIT = 1 PRIORITY = 30 NICE = 20 STACKLIMIT = 5256 DEFAULT_EXTSCHED = RMS[RMS_SNODE] # LSF will using this scheduling policy if # -extsched is not defined. # MANDATORY_EXTSCHED = RMS[RMS_SNODE] # LSF enforces this scheduling policy RES_REQ = select[rms==1] DESCRIPTION = Run RMS jobs only on hosts that have resource 'rms' defined End Queue
To make therms
queue the default queue, set DEFAULT_QUEUE=rms inlsb.params
.
Use the
bqueues -l
command to view the queue configuration details. Before using LSF HPC, see the Platform LSF Configuration Reference to understand queue configuration parameters inlsb.queues
.During installation,
lsfinstall
sets the following parameters inlsf.conf
:
- LSF_ENABLE_EXTSCHEDULER=Y
LSF uses an external scheduler for RMS allocation.
- LSB_RLA_PORT=port_number
Where port_number is the TCP port used for communication between the LSF HPC topology adapter (RLA) and
sbatchd
.The default port number is 6883.
- LSB_SHORT_HOSTLIST=1
Displays an abbreviated list of hosts in
bjobs
andbhist
for a parallel job where multiple processes of a job are running on a host. Multiple processes are displayed in the following format:processes*hostADuring installation, the Boolean resource
rms
is defined inlsf.shared
:Begin Resource RESOURCENAME TYPE INTERVAL INCREASING DESCRIPTION ... rms Boolean () () (RMS) ... End Resource
You should add the rms resource name under the RESOURCES column of the Host section oflsf.cluster.
cluster_name. Hosts without the rms resource specified are not considered for scheduling RMS jobs.
lsf.cluster.cluster_name
For each RMS host,
hostsetup
adds therms
Boolean resource to the HOST section oflsf.cluster.
cluster_name.Setting dedicated LSF partitions (recommended)
You should use the RMS
rcontrol
command to preventprun
jobs from running directly on partitions dedicated to LSF.# rcontrol set partition=parallel configuration=day type=batchSee the RMS
rcontrol
command in the RMS Reference Manual for complete syntax and usage.Customizing job control actions (optional)
By default, LSF carries out job control actions by sending the appropriate signal to suspend, terminate, or resume a job. If your jobs need special job control actions, use the RMS
rcontrol
command in therms
queue configuration for RMS jobs to change the default job controls.Use the JOB_CONTROLS parameter in
lsb.queues
to configure suspend, terminate, or resume job controls for the queue:JOB_CONTROLS = SUSPEND[command] | RESUME[command] | TERMINATE[command]where command is an
rcontrol
command of the form:rcontrol [suspend | kill | resume] batchid=cluster_name@$LSB_JOBIDExample TERMINATE job control action
Begin Queue QUEUE_NAME=rms ... JOB_CONTROLS = TERMINATE[rcontrol kill batchid=cluster1@$LSB_JOBID] ... End Queue
- See Sample job termination script for queue job control for an example of using a script for job control
- See the Platform LSF Configuration Reference for more information about the JOB_CONTROLS parameter in
lsb.queues
- See the RMS
rcontrol
command in the RMS Reference Manual for complete syntax and usageConfiguration notes
Resource to determine RMS-enabled hosts
The
hostsetup
script configureslsf.cluster.
cluster_name to assign the Boolean resourcerms
to all LSF hosts that run on an RMS partition. Therms
resource is defined in the defaultlsf.shared
template file at installation.Maximum job slot limit (MXJ in lsb.hosts)
By default, the maximum job slot limit is set to the number of CPUs that LIM reports. This is specified by MXJ=! in the Host section of
LSB_CONFDIR/lsb.hosts
:Begin Host HOST_NAME MXJ r1m pg ls tmp DISPATCH_WINDOW # Keywords ... default ! () () () () () # Example ... End Host
Do not change this default.
Per-processor job slot limit (PJOB_LIMIT in lsb.queues)
By default, the per-processor job slot limit is 1 (PJOB_LIMIT=1 in the
rms
queue inlsb.queues
).
Do not change this default.
Maximum number of sbatchd connections (lsb.params)
If LSF operates on a large system (for example, a system with more than 32 hosts), you may need to configure the parameter MAX_SBD_CONNS in
lsb.params
. MAX_SBD_CONNS controls the maximum number of filesmbatchd
can have open and connected tosbatchd
.In a very busy cluster with many jobs being dispatched, running, finishing at the same time, you may see it takes a very long time for
mbatchd
to update the status change of a job, and to dispatch new jobs. If your cluster shows this behavior, you should set MAX_SBD_CONNS = (number_of_hosts) * 2 or 300, which ever is less. Setting MAX_SBD_CONNS too high may slow down the speed ofmbatchd
dispatching new jobs.
- About job limits and configuring hosts and queues, see Administering Platform LSF
- About the
bqueues
command and thelsb.hosts
,lsb.params
, andlsb.queues
files, see the Platform LSF Configuration ReferenceConfiguring default and mandatory topology scheduling options
Use the DEFAULT_EXTSCHED and MANDATORY_EXTSCHED queue paramters in
lsb.queues
to configure default and mandatory topology scheduling options.DEFAULT_EXTSCHED=RMS[[allocation_type][;topology][;flags]]
Specifies default topology scheduling options for the queue.
-extsched
options on thebsub
command are merged with DEFAULT_EXTSCHED options, and-extsched
options override any conflicting queue-level options set by DEFAULT_EXTSCHED.For example, if
DEFAULT_EXTSCHED=RMS[RMS_SNODE;rails=2]
and a job is submitted with-extsched "RMS[base=hostA;rails=1;ptile=2]"
, LSF uses the following resulting options for scheduling:RMS_SNODE;rails=1;base=hostA;ptile=2DEFAULT_EXTSCHED can be used in combination with MANDATORY_EXTSCHED in the same queue. For example:
-extsched "RMS[base=hostA;ptile=2]"
DEFAULT_EXTSCHED=RMS[rails=2]
MANDATORY_EXTSCHED=RMS[RMS_SNODE;ptile=4]
LSF uses the resulting options for scheduling:
RMS_SNODE;rails=2;base=hostA;ptile=4If topology options (
nodes
,ptile
, orbase
) or rail flags (rails
orrailmask
) are set in DEFAULT_EXTSCHED, and you do not want to specify values for these options, use the keyword with no value in the-extsched
option ofbsub
. For example, ifDEFAULT_EXTSCHED=RMS[nodes=2]
, and you do not want to specify any node option at all, use-extsched "RMS[RMS_SNODE;nodes=]"
.See bsub command for more information.
Syntax in the form DEFAULT_EXTSCHED=extsched_options is obsolete (for example, DEFAULT_EXTSCHED=RMS_SNODE). You should use the syntax of the form DEFAULT_EXTSCHED=RMS[extsched_options] (for example, DEFAULT_EXTSCHED=RMS[RMS_SNODE]).
The queue named
rms
defined during installation specifies DEFAULT_EXTSCHED=RMS[RMS_SNODE]. LSF uses the RMS_SNODE scheduling policy if no allocation type is specified at job submission.MANDATORY_EXTSCHED=RMS[[allocation_type][;topology][;flags]]
Specifies mandatory topology scheduling options for the queue.
-extsched
options on thebsub
command are merged with MANDATORY_EXTSCHED options, and MANDATORY_EXTSCHED options override any conflicting job-level options set by-extsched
.Overrides
-extsched
options on thebsub
command.For example, if
MANDATORY_EXTSCHED=RMS[RMS_SNODE;rails=2]
and a job is submitted with-extsched "RMS[base=hostA;rails=1;ptile=2]"
, LSF uses the following resulting options for scheduling:RMS_SNODE;rails=2;base=hostA;ptile=2MANDATORY_EXTSCHED can be used in combination with DEFAULT_EXTSCHED in the same queue. For example:
-extsched "RMS[base=hostA;ptile=2]"
DEFAULT_EXTSCHED=RMS[rails=2]
MANDATORY_EXTSCHED=RMS[RMS_SNODE;ptile=4]
LSF uses the following resulting options for scheduling:
RMS_SNODE;rails=2;base=hostA;ptile=4See bsub command for more information.
If you want to prevent users from setting the topology options (
nodes
,ptile
, orbase
) or flags (rails
orrailmask
) in the-extsched
option ofbsub
, use the keyword with no value. For example, if the job is submitted with-extsched "RMS[RMS_SNODE;nodes=4]"
, useMANDATORY_EXTSCHED=RMS[nodes=]
to override both of these settings.
Syntax in the form MANDATORY_EXTSCHED=extsched_options is obsolete (for example, MANDATORY_EXTSCHED=RMS_SNODE). You should use the syntax of the form MANDATORY_EXTSCHED=RMS[extsched_options] (for example, MANDATORY_EXTSCHED=RMS[RMS_SNODE]).
[ Top ]
Operating Platform LSF HPC for Linux/QsNet
- RMS hosts and RMS jobs
- Platform LSF HPC RMS topology support plugin
- LSF scheduling policies and RMS topology support
- LSF host preference and RMS allocation options
- RMS rail allocation options
RMS hosts and RMS jobs
An RMS host has the
rms
Boolean resource in the RESOURCES column of the host section inlsf.cluster.
cluster_name.An RMS job has appropriate external scheduler options at the command line (
bsub -extsched
) or queue level (DEFAULT_EXTSCHED or MANDATORY_EXTSCHED inrms
queue inlsb.queues
).RMS jobs only run on RMS hosts, and non-RMS jobs only run on non-RMS hosts.
Platform LSF HPC RMS topology support plugin
The Platform LSF HPC RMS external scheduler plugin runs on each LSF HPC host within an RMS partition. The RMS plugin is started by
mbschd
and handles all communication between the LSF HPC scheduler and RMS. It translates LSF HPC concepts (hosts and job slots) into RMS concepts (nodes, number of CPUs, allocation options, topology).The Platform LSF HPC topology adapter for RMS (RLA) is located on each LSF HPC host within an RMS partition. RLA is started by
sbatchd
and is the interface for the LSF RMS plugin and the RMS system.To schedule a job, the RMS external scheduler plugin calls RLA to:
- Report the number of free CPUs on every host requested by the job
- Allocate an RMS resource with the specified topology
- Deallocate RMS resources when the job finishes
LSF scheduling policies and RMS topology support
Supported RMS prun allocation options
LSF host preference and RMS allocation options
Host preference partition (for example,
bsub -m hostA
) is only supported for RMS_SLOAD allocation. LSF host preference is not taken into account for RMS_SNODE and RMS_MCONT allocation. All hosts in the preference list must be within the same host group or RMS partition.LSF sorts nodes according to RMS topology (numbering of nodes and domains), which takes precedence over LSF sorting order. LSF host preferences (for example,
bsub -m hostA
) are not taken into account.LSF sorts nodes based on host preference and load information, which take precedence over RMS topology (numbering of nodes and domains).
The allocation starts from the first host specified in the list of LSF hosts and continues until the allocation specification is satisfied.
Use RMS_SLOAD on smaller clusters, where the job placement decision should be influenced by host load, or where you want to keep a specific host preference.
LSF sorts nodes based on RMS topological order (numbering of nodes and domains); LSF host preferences are not taken into account.
RMS rail allocation options
Rails are the layers of a Quadrics switch network. In multirail systems, you can specify the following rail allocation options:
rails=
numberSpecifies the number of rails required, where 1 <= number <= 32
OR
railmask=
bitmask
Specifies which rails are required, using bitmask to select them
Specify either
rails
orrailmask
, but not both.To use the RMS rail options with LSF jobs, use the
-extsched
option ofbsub
:[rails=number | railmask=bitmask]For example, The following job uses 2 rails.
bsub
-n 4
-q rms
-extsched "RMS[RMS_MCONT; rails=2
]"
LSF checks the validity of rail options at job submission against the LSB_RMS_MAXNUMRAILS parameter if it is set in
lsf.conf
, which specifies a maximum value for therails
option. The default is 32. If incorrect rail option values pass this check, the job pends forever.About the
rails
andrailmask
options, see the RMSprun
command in the RMS Reference Manual[ Top ]
Submitting and Monitoring Jobs
- bsub command
- Running jobs on any host type
- Viewing nodes allocated to your job
- Example job submissions
bsub command
To submit a job, use the
bsub
command.bsub -ext
[sched
]"RMS[
[allocation_type][;
topology][;
flags]]"
job_nameSpecify topology scheduling policy options for RMS jobs either in the
-extsched
option, or with DEFAULT_EXTSCHED or MANDATORY_EXTSCHED in therms
queue definition inlsb.queues
.
You can abbreviate the -extsched option to -ext.
The options set by
-extsched
can be combined with the queue-level MANDATORY_EXTSCHED or DEFAULT_EXTSCHED parameters. If-extsched
and MANDATORY_EXTSCHED set the same option, the MANDATORY_EXTSCHED setting is used. If-extsched
and DEFAULT_EXTSCHED set the same options, the-extsched
setting is used.
The -extsched syntax in the form -extsched "extsched_options" is obsolete (for example, -extsched "RMS_SNODE"). You should use the syntax of the form -extsched "RMS[extsched_options]" (for example, -extsched "RMS[RMS_SNODE]").
Shell-specific command syntax is not supported when submitting jobs to run in RMS.
For example,bsub "cmd1|cmd2"
is not valid because of the pipe (|
).
If you use RMS_MCONT or RMS_SNODE allocation options, the
ptile
option in the span section of the resource requirement string (bsub -R "span[ptile=
n]"
) is not supported.You should use
extsched "RMS[ptile=n]"
to define the locality of jobs instead of-R "span[ptile=n]"
.RMS allocation and topology scheduling options
- allocation_type
Specifies the type of node allocation:
RMS_SNODE
|RMS_SLOAD
|RMS_MCONT
where
RMS_SNODE
Specifies sorted node allocation. Nodes do not need to be contiguous: gaps are allowed between the leftmost and rightmost nodes of the allocation map.
This is the default allocation policy for RMS jobs.
LSF sorts nodes according to RMS topology (numbering of nodes and domains), which takes precedence over LSF sorting order. LSF host preferences (for example,
bsub -m hostA
) are not taken into account.The allocation is more compact than in RMS_SLOAD and starts from the leftmost node allowed by the LSF host list and continues rightward until the allocation specification is satisfied.
Use RMS_SNODE on larger clusters where only the number of available job slots matters for job placement decisions. When RMS_SNODE is used with job preemption, LSF may suspend more jobs than necessary. You should use RMS_SLOAD if you use preemptive scheduling.
RMS_SLOAD
Specifies sorted load allocation. Nodes do not need to be contiguous: gaps are allowed between the leftmost and rightmost nodes of the allocation map.
LSF sorts nodes based on host preference and load information, which takes precedence over RMS topology (numbering of nodes and domains).
The allocation starts from the first host specified in the list of LSF hosts and continues until the allocation specification is satisfied.
Use RMS_SLOAD on smaller clusters, where the job placement decision should be influenced by host load, or where you want to keep a specific host preference.
RMS_MCONT
Specifies mandatory contiguous node allocation. The allocation must be contiguous: between the leftmost and rightmost nodes of the allocation map, each node must either have at least one CPU which belongs to this allocation OR this node must be configured out completely.
LSF sorts nodes based on RMS topological order (numbering of nodes and domains); LSF host preferences (for example
bsub -m hostA+
) are not taken into account.The allocation is more compact than in RMS_SNODE, so RMS_MCONT provides the best job scheduling performance, but it requires contiguous nodes. Allocation starts from the leftmost node that allows contiguous allocation. Nodes that are out of service are not considered as gaps.
If you specify a CPU range (
bsub -n
min_processors,
max_processors), LSF builds a CPU allocation on a first-fit basis. For example, if you specifybsub -n 1,20
, a partition with an area at the beginning of the partition with 10 contiguous CPUs, and another area at the end of the partition with 18 contiguous CPUs has two areas that can fit the job. LSF places the job in the first area that satisfies the minimum specification; in this case, LSF places the job at the beginning of the partition.- topology
Specifies the topology of the allocation:
nodes=
nodes |ptile=
cpus_per_nodeSpecifies the number of nodes the allocation requires or the number of CPUs per node.
The
ptile
topology option is different from the LSFptile
keyword used in thespan
section of the resource requirement string (bsub -R "span[ptile=
n]"
). If theptile
topology option is specified in-extsched
, the value ofbsub -n
must be an exact multiple of theptile
value.For example:
$ bsub -n 12 -extsched "RMS[ptile=4]" prun mt_appis correct because the value of
-n
is exactly divisible by the value ofptile
.To enforce a multi-threaded application to run within a node, use
-extsched "RMS[nodes=1]"
. For example:$ bsub -n 3 -extsched "RMS[RMS_SLOAD;nodes=1]" prun mt_appTo enforce that a job only takes nodes where there is no other job running on, use
ptile=
cpus_per_node. For example, on an ES40 machine:$ bsub -n 20 -extsched "RMS[RMS_SLOAD;ptile=4]" prun parallel_app
If you specify a CPU range (bsub -n
min_processors,
max_processors) with the RMS[nodes] option, only min_processors is considered; max_processors is ignored.
base=
base_node_nameIf specified with RMS_SNODE or RMS_MCONT allocation, the base node name is used as the starting node for the allocation instead of the leftmost node allowed by the LSF host list.
If
base
is specified with RMS_SLOAD allocation, RMS_SNODE allocation is used.- flags
Specifies other allocation options. The following flags are supported:
rails=
numberSpecifies the number of rails required, where 1 <= number <= 32
railmask=
bitmask
Specifies which rails are required, using bitmask to select them
Specify either rails or railmask, but not both.
See RMS rail allocation options for more information.
The topology options
nodes
andptile
, and therails
flag are limited by the values of the corresponding parameters inlsf.conf
:
nodes
must be between 1 and LSB_RMS_MAXNUMNODESptile
must be between 1 and LSB_RMS_MAXPTILErails
must be between 1 and LSB_RMS_MAXNUMRAILSIf topology options (
nodes
,ptile
, orbase
) or flags (rails
orrailmask
) are set in DEFAULT_EXTSCHED, and you do not want to specify values for these options, use the keyword with no value in the-extsched
option ofbsub
. For example, ifDEFAULT_EXTSCHED=RMS[nodes=2]
, and you do not want to specify any node option at all, use-extsched "RMS[RMS_SNODE;nodes=]"
.Running jobs on any host type
You can specify several types of topology scheduling options at job submission and LSF will schedule jobs appropriately. Jobs that do not specify RMS-related options can be dispatched to RMS hosts, and jobs with RMS-related options can be dispatched to non- RMS hosts.
Use the LSF resource requirements specification (
-R
option ofbsub
or RES_REQ in queue definition inlsb.queues
) to identify the host types required for your job.For example, HP pset hosts and Linux/QsNet hosts running RMS can exist in the same LSF cluster. Use the
-R
option to define resource requirements for either HP pset hosts or Linux/QsNet hosts. Your job will run on either host type, but not both.For example:
bsub n -8 -R "pset || rms" -ext "RMS[ptile=2];PSET[PTILE=2]" myjobRuns
myjob
on an HP pset host or an RMS host if one is available, but not both. If it runs on an RMS host, theRMS[ptile=2]
option is applied. If it runs on a pset host, the RMS option is ignored and thePSET[PTILE=2]
option is applied.Viewing nodes allocated to your job
Use
bjobs -l
to see RMS allocation information for a running job. For example, the following job allocates nodes on hostshostA
andhostB
, with resource ID 3759 in the partition namedparallel
:bjobs -l 723
Job <723>, User <user1>, Project <default>, Status <DONE>, Queue <rms>, Extsched <RMS[nodes=3;base=hostA]>, Command <hostname> Wed Aug 6 17:09:44: Submitted from host <hostA>, CWD <$HOME>, 7 Processors R equested; STACKLIMIT 5256 K Wed Aug 6 17:09:58: Started on 7 Hosts/Processors <3*hostA> <4*hostB>, Execution Home </home/user1>, Execution CWD </home/user1>; Wed Aug 6 17:09:58: rms_rid=parallel.3759;rms_alloc=3*hostA 2*hostB[2-3]; Wed Aug 6 17:10:01: Done successfully. The CPU time used is 0.1 seconds. SCHEDULING PARAMETERS: r15s r1m r15m ut pg io ls it tmp swp mem loadSched - - - - - - - - - - - loadStop - - - - - - - - - - - EXTERNAL MESSAGES: MSG_ID FROM POST_TIME MESSAGE ATTACHMENT 0 - - - - 1 user1 Aug 6 17:09 RMS[nodes=3;base=hostA] NFinished jobs (bhist -l)
Use
bhist -l
to see RMS allocation information for finished jobs. For example:bhist -l 723
Job <723>, User <user1>, Project <default>, Extsched <RMS[nodes=3;base=hostA ]>, Command <hostname> Wed Aug 6 17:09:44: Submitted from host <hostA>, to Queue <rms>, CWD <$HOME>, 7 Processors Requested; Wed Aug 6 17:09:58: Dispatched to 7 Hosts/Processors <3*hostA> <4*hostB>; Wed Aug 6 17:09:58: rms_rid=parallel.3759;rms_alloc=3*hostA 2*hostB[2-3]; Wed Aug 6 17:09:58: Starting (Pid 971318); Wed Aug 6 17:10:00: Running with execution home </home/user1>, Execution CWD </home/user1>, Execution Pid <971318>; Wed Aug 6 17:10:01: Done successfully. The CPU time used is 0.1 seconds; Wed Aug 6 17:10:01: Post job process done successfully; Summary of time in seconds spent in various states by Wed Aug 6 17:10:01 PEND PSUSP RUN USUSP SSUSP UNKWN TOTAL 14 0 3 0 0 0 17Job accounting information (bacct -l)
Use
bacct -l
to see RMS allocation information logged tolsb.acct
. For example:bacct -l 3088
Accounting information about jobs that are: - submitted by all users. - accounted on all projects. - completed normally or exited - executed on all hosts. - submitted to all queues. ------------------------------------------------------------------------------ Job <3088>, User <user1>, Project <default>, Status <DONE>, Queue <rms>, Command <prun hostname> Wed Aug 6 17:09:44: Submitted from host <hostS>, CWD <$HOME>; Wed Aug 6 17:09:58: Dispatched to 7 Hosts/Processors <3*hostA> <4*hostB>; Wed Aug 6 17:09:58: rms_rid=parallel.3759;rms_alloc=3*hostA 2*hostB[2-3]; Wed Aug 6 17:10:01: Completed <done>. Accounting information about this job: CPU_T WAIT TURNAROUND STATUS HOG_FACTOR MEM SWAP 0.43 8 43 exit 0.0100 1024K 0K ------------------------------------------------------------------------------ SUMMARY: ( time unit: second ) Total number of done jobs: 0 Total number of exited jobs: 1 Total CPU time consumed: 0.4 Average CPU time consumed: 0.4 Maximum CPU time of a job: 0.4 Minimum CPU time of a job: 0.4 Total wait time in queues: 8.0 Average wait time in queue: 8.0 Maximum wait time in queue: 8.0 Minimum wait time in queue: 8.0 Average turnaround time: 43 (seconds/job) Maximum turnaround time: 43 Minimum turnaround time: 43 Average hog factor of a job: 0.01 ( cpu time / turnaround time ) Maximum hog factor of a job: 0.01 Minimum hog factor of a job: 0.01Example job submissions
Example 1: Submitting a job with a script
The following script defines a job requiring 128 CPUs:
#!/bin/sh #myscript ./sequential_pre_processor prun -n 64 parallel1 & prun -n 64 parallel2 & wait ./sequential_prog prun -n 128 parallel3 ./write_resultsSubmit the job with the following command:
$bsub -n 128 ./myscript
Example 2: Submitting a job without a script
$bsub -n 128 prun parallel_app
Example 3: Submitting a job with prun specified in the queue
The following job assumes that
prun
is specified in the JOB_STARTER parameter of the default queue definition inlsb.queues
.$bsub -n 128 parallel_app
prun
with no arguments uses all CPUs allocated to it.Example 4: Using ptile and nodes topology options
To enforce a multi-threaded application to run within a node, use
-extsched "nodes=1"
. For example:$ bsub -n 3 -extsched "RMS[RMS_SLOAD;nodes=1]" prun mt_appTo enforce that a job only takes nodes where there is no other job running on, use RMS[
ptile=
cpus_per_node]
. For example, on an ES40 machine:$ bsub -n 20 -extsched "RMS[RMS_SLOAD;ptile=4]" prun parallel_appFor more information
- About MANDATORY_EXTSCHED and DEFAULT_EXTSCHED, see Configuring default and mandatory topology scheduling options
- About job operations, see Administering Platform LSF
- About job starters, see Administering Platform LSF
- About
bacct
,bhist
,bjobs
, andbsub
, see the Platform LSF Command Reference- About
lsb.queues
, andlsf.conf
see the Platform LSF Configuration Reference[ Top ]
[ Platform Documentation ] [ Title ] [ Contents ] [ Previous ] [ Next ] [ Index ]
Date Modified: August 20, 2009
Platform Computing: www.platform.com
Platform Support: support@platform.com
Platform Information Development: doc@platform.com
Copyright © 1994-2009 Platform Computing Corporation. All rights reserved.