Knowledge Center Contents Previous Next Index |
Runtime Resource Usage Limits
Contents
- About Resource Usage Limits
- Specifying Resource Usage Limits
- Supported Resource Usage Limits and Syntax
- CPU Time and Run Time Normalization
- PAM resource limits
About Resource Usage Limits
Resource usage limits control how much resource can be consumed by running jobs. Jobs that use more than the specified amount of a resource are signalled or have their priority lowered.
Limits can be specified by the LSF administrator:
- At the queue level in
lsb.queues
- In an application profile in
lsb.applications
- At the job level when you submit a job
For example, by defining a high-priority short queue, you can allow short jobs to be scheduled earlier than long jobs. To prevent some users from submitting long jobs to this short queue, you can set CPU limit for the queue so that no jobs submitted from the queue can run for longer than that limit.
Limits specified at the queue level are
hard
limits, while those specified with job submission or in an application profile aresoft
limits. The hard limit acts as a ceiling for the soft limit. Seesetrlimit(2)
man page for concepts of hard and soft limits.
note:
This chapter describes queue-level and job-level resource usage limits. Priority of limits is different if limits are also configured in an application profile. See Chapter 23, "Working with Application Profiles" for information about resource usage limits in application profiles.Resource usage limits and resource allocation limits
Resource usage limits are not the same as
resource allocation limits
, which are enforced during job scheduling and before jobs are dispatched. You set resource allocation limits to restrict the amount of a given resource that must be available during job scheduling for different classes of jobs to start, and which resource consumers the limits apply to.See Chapter 24, "Resource Allocation Limits" for more information.
Summary of resource usage limits
Priority of resource usage limits
If no limit is specified at job submission, then the following apply to all jobs submitted to the queue:
Incorrect resource usage limits
Incorrect limits are ignored, and a warning message is displayed when the cluster is reconfigured or restarted. A warning message is also logged to the
mbatchd
log file when LSF is started.If no limit is specified at job submission, then the following apply to all jobs submitted to the queue:
Resource usage limits specified at job submission must be less than the maximum specified in
lsb.queues
. The job submission is rejected if the user-specified limit is greater than the queue-level maximum, and the following message is issued:
Cannot exceed queue's hard limit(s). Job not submitted.
Enforce limits on chunk jobs
By default, resource usage limits are not enforced for chunk jobs because chunk jobs are typically too short to allow LSF to collect resource usage.
- To enforce resource limits for chunk jobs, define LSB_CHUNK_RUSAGE=Y in
lsf.conf
. Limits may not be enforced for chunk jobs that take less than a minute to run.Scaling the units for resource usage limits
The default unit for the following resource usage limits is KB:
- Core limit (
-C
and CORELIMIT)- Memory limit (
-M
and MEMLIMIT)- Stack limit (
-S
and STACKLIMIT)- Swap limit (
-v
and SWAPLIMIT)This default may be too small for some environments that make use of very large resource usage limits, for example, GB or TB.
LSF_UNIT_FOR_LIMITS in
lsf.conf
specifies larger units for the resource usage limits with default unit of KB.The unit for the resource usage limit can be one of:
- KB (kilobytes)
- MB (megabytes)
- GB (gigabytes)
- TB (terabytes)
- PB (petabytes)
- EB (exabytes)
LSF_UNIT_FOR_LIMITS applies cluster-wide to limts at the job-level (
bsub
), queue-level (lsb.queues
), and application level (lsb.applications
).The lmit unit specified by LSF_UNIT_FOR_LIMITS also applies to limts modified with
bmod
, and the display of resource usage limits in query commands (bacct
,bapp
,bhist
,bhosts
,bjobs
,bqueues
,lsload
, andlshosts
).
important:
Before changing the units of your resource usage limits, you should completely drain the cluster of all workload. There should be no running, pending, or finished jobs in the system.In a MultiCluster environment, you should configure the same unit for all clusters.
After changing LSF_UNIT_FOR_LIMITS, you must restart your cluster.
How limit unit changes affect jobs
When LSF_UNIT_FOR_LIMITS is specified, the defined unit is used for the following commands. In command output, the larger unit appears as T, G, P, or E, depending on the job rusage and the unit defined.
Example
A job is submitted with
bsub -M 100
and LSF_UNIT_FOR_LIMITS=MB; the memory limit for the job is 100 MB rather than the default 100 KB.Specifying Resource Usage Limits
Queues can enforce resource usage limits on running jobs. LSF supports most of the limits that the underlying operating system supports. In addition, LSF also supports a few limits that the underlying operating system does not support.
Specify queue-level resource usage limits using parameters in
lsb.
queues
.Specifying queue-level resource usage limits
Limits configured in
lsb.queues
apply to all jobs submitted to the queue. Job-level resource usage limits specified at job submission override the queue definitions.Maximum value only
Specify only a maximum value for the resource.
For example, to specify a maximum run limit, use one value for the RUNLIMIT parameter in
lsb.queues
:RUNLIMIT = 10The maximum run limit for the queue is 10 minutes. Jobs cannot run for more than 10 minutes. Jobs in the RUN state for longer than 10 minutes are killed by LSF.
If only one run limit is specified, jobs that are submitted with
bsub -W
with a run limit that exceeds the maximum run limit is not allowed to run. Jobs submitted withoutbsub -W
are allowed to run but are killed when they are in the RUN state for longer than the specified maximum run limit.For example, in
lsb.queues
:RUNLIMIT = 10The maximum run limit for the queue is 10 minutes. Jobs cannot run for more than 10 minutes.
Default and maximum values
If you specify two limits, the first one is the default limit for jobs in the queue and the second one is the maximum (hard) limit. Both the default and the maximum limits must be positive integers. The default limit must be less than the maximum limit. The default limit is ignored if it is greater than the maximum limit.
Use the default limit to avoid having to specify resource usage limits in the
bsub
command.For example, to specify a default and a maximum run limit, use two values for the RUNLIMIT parameter in
lsb.queues
:RUNLIMIT = 10 15
- The first number is the default run limit applied to all jobs in the queue that are submitted without a job-specific run limit (without
bsub -W
).- The second number is the maximum run limit applied to all jobs in the queue that are submitted with a job-specific run limit (with
bsub -W
). The default run limit must be less than the maximum run limit.You can specify both default and maximum values for the following resource usage limits in
lsb.queues
:
- CPULIMIT
- DATALIMIT
- MEMLIMIT
- PROCESSLIMIT
- RUNLIMIT
- THREADLIMIT
Host specification with two limits
If default and maximum limits are specified for CPU time limits or run time limits, only one host specification is permitted. For example, the following CPU limits are correct (and have an identical effect):
CPULIMIT = 400/hostA 600 CPULIMIT = 400 600/hostAThe following CPU limit is not correct:
CPULIMIT = 400/hostA 600/hostBThe following run limits are correct (and have an identical effect):
RUNLIMIT = 10/hostA 15 RUNLIMIT = 10 15/hostAThe following run limit is not correct:
RUNLIMIT = 10/hostA 15/hostBDefault run limits for backfill scheduling
Default run limits are used for backfill scheduling of parallel jobs.
For example, in
lsb.queues
, you enter: RUNLIMIT = 10 15
- The first number is the default run limit applied to all jobs in the queue that are submitted without a job-specific run limit (without
bsub -W
).- The second number is the maximum run limit applied to all jobs in the queue that are submitted with a job-specific run limit (with
bsub -W)
. The default run limit cannot exceed the maximum run limit.Automatically assigning a default run limit to all jobs in the queue means that backfill scheduling works efficiently.
For example, in
lsb.queues
, you enter:RUNLIMIT = 10 15The first number is the default run limit applied to all jobs in the queue that are submitted without a job-specific run limit. The second number is the maximum run limit.
If you submit a job to the queue without the
-W
option, the default run limit is used:bsub myjob
The job
myjob
cannot run for more than 10 minutes as specified with the default run limit.If you submit a job to the queue with the
-W
option, the maximum run limit is used:bsub -W 12 myjob
The job
myjob
is allowed to run on the queue because the specified run limit (12) is less than the maximum run limit for the queue (15).bsub -W 20 myjob
The job
myjob
is rejected from the queue because the specified run limit (20) is more than the maximum run limit for the queue (15).Specify job-level resource usage limits
- To specify resource usage limits at the job level, use one of the following
bsub
options:
-C
core_limit
-c
cpu_limit
-D
data_limit
-F
file_limit
-M
mem_limit
-p
process_limit
-W
run_limit
-S
stack_limit
-T
thread_limit
-v
swap_limit
Job-level resource usage limits specified at job submission override the queue definitions.
Supported Resource Usage Limits and Syntax
Core file size limit
Job syntax (bsub) Queue syntax (lsb.queues) Fomat/Default Units-C
core_limit
CORELIMIT=limit
integer
KB
Sets a per-process (soft) core file size limit for each process that belongs to this batch job.
By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in
lsf.conf
to specify a larger unit for the the limit (MB, GB, TB, PB, or EB).On some systems, no core file is produced if the image for the process is larger than the core limit. On other systems only the first
core_limit
KB of the image are dumped. The default is no soft limit.CPU time limit
Job syntax (bsub) Queue syntax (lsb.queues) Fomat/Default Units-c
cpu_limit
CPULIMIT=[default
]maximum
[hours
:]minutes
[/host_name
| /host_model
]
Sets the soft CPU time limit to
cpu_limit
for this batch job. The default is no limit. This option is useful for avoiding runaway jobs that use up too many resources. LSF keeps track of the CPU time used by all processes of the job.When the job accumulates the specified amount of CPU time, a SIGXCPU signal is sent to all processes belonging to the job. If the job has no signal handler for SIGXCPU, the job is killed immediately. If the SIGXCPU signal is handled, blocked, or ignored by the application, then after the grace period expires, LSF sends SIGINT, SIGTERM, and SIGKILL to the job to kill it.
You can define whether the CPU limit is a per-process limit enforced by the OS or a per-job limit enforced by LSF with LSB_JOB_CPULIMIT in
lsf.conf
.Jobs submitted to a chunk job queue are not chunked if the CPU limit is greater than 30 minutes.
Format
cpu_limit
is in the form [hour
:]minute
, whereminute
can be greater than 59. 3.5 hours can either be specified as 3:30 or 210.Normalized CPU time
The CPU time limit is
normalized
according to the CPU factor of the submission host and execution host. The CPU limit is scaled so that the job does approximately the same amount of processing for a given CPU limit, even if it is sent to a host with a faster or slower CPU.For example, if a job is submitted from a host with a CPU factor of 2 and executed on a host with a CPU factor of 3, the CPU time limit is multiplied by 2/3 because the execution host can do the same amount of work as the submission host in 2/3 of the time.
If the optional
host name or host model is not given, the CPU limit is scaled based on the DEFAULT_HOST_SPEC specified in the
lsb.params
file. (If DEFAULT_HOST_SPEC is not defined, the fastest batch host in the cluster is used as the default.) If host or host model is given, its CPU scaling factor is used to adjust the actual CPU time limit at the execution host.The following example specifies that
myjob
can run for 10 minutes on a DEC3000 host, or the corresponding time on any other host:bsub -c 10/DEC3000 myjob
See CPU Time and Run Time Normalization for more information.
Data segment size limit
Job syntax (bsub) Queue syntax (lsb.queues) Fomat/Default Units-D
data_limit
DATALIMIT=[default
]maximum
integer
KB
Sets a per-process (soft) data segment size limit in KB for each process that belongs to this batch job (see
getrlimit
(2)).This option affects calls to
sbrk()
andbrk()
. Ansbrk()
ormalloc()
call to extend the data segment beyond the data limit returns an error.
note:
Linux does not usesbrk()
andbrk()
within itscalloc()
andmalloc()
. Instead, it uses (mmap()
) to create memory. DATALIMIT cannot be enforced on Linux applications that callsbrk()
andmalloc()
.On AIX, if the XPG_SUS_ENV=ON environment variable is set in the user's environment before the process is executed and a process attempts to set the limit lower than current usage, the operation fails with errno set to EINVAL. If the XPG_SUS_ENV environment variable is not set, the operation fails with errno set to EFAULT.
The default is no soft limit.
File size limit
Job syntax (bsub) Queue syntax (lsb.queues) Fomat/Default Units-F
file_limit
FILELIMIT=limit
integer
KB
Sets a per-process (soft) file size limit in KB for each process that belongs to this batch job. If a process of this job attempts to write to a file such that the file size would increase beyond the file limit, the kernel sends that process a SIGXFSZ signal. This condition normally terminates the process, but may be caught. The default is no soft limit.
Memory limit
Job syntax (bsub) Queue syntax (lsb.queues) Fomat/Default Units-M
mem_limit
MEMLIMIT=[default
]maximum
integer
KB
Sets a per-process physical memory limit for all of the processes belonging to a job
By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in
lsf.conf
to specify a larger unit for the the limit (MB, GB, TB, PB, or EB).If LSB_MEMLIMIT_ENFORCE=Y or LSB_JOB_MEMLIMIT=Y are set in
lsf.conf
, LSF kills the job when it exceeds the memory limit. Otherwise, LSF passes the memory limit to the operating system. Some operating systems apply the memory limit to each process, and some do not enforce the memory limit at all.LSF memory limit enforcement
To enable LSF memory limit enforcement, set LSB_MEMLIMIT_ENFORCE in
lsf.conf
toy
. LSF memory limit enforcement explicitly sends a signal to kill a running process once it has allocated memory pastmem_limit
.You can also enable LSF memory limit enforcement by setting LSB_JOB_MEMLIMIT in
lsf.conf
toy.
The difference between LSB_JOB_MEMLIMIT set to y and LSB_MEMLIMIT_ENFORCE set to y is that with LSB_JOB_MEMLIMIT, only the per-job memory limit enforced by LSF is enabled. The per-process memory limit enforced by the OS is disabled. With LSB_MEMLIMIT_ENFORCE set to y, both the per-job memory limit enforced by LSF and the per-process memory limit enforced by the OS are enabled.LSB_JOB_MEMLIMIT disables per-process memory limit enforced by the OS and enables per-job memory limit enforced by LSF. When the total memory allocated to all processes in the job exceeds the memory limit, LSF sends the following signals to kill the job: SIGINT first, then SIGTERM, then SIGKILL.
On UNIX, the time interval between SIGINT, SIGKILL, SIGTERM can be configured with the parameter JOB_TERMINATE_INTERVAL in
lsb.params
.OS memory limit enforcement
OS enforcement usually allows the process to eventually run to completion. LSF passes
mem_limit
to the OS, which uses it as a guide for the system scheduler and memory allocator. The system may allocate more memory to a process if there is a surplus. When memory is low, the system takes memory from and lowers the scheduling priority (re-nice) of a process that has exceeded its declaredmem_limit
.OS memory limit enforcement is only available on systems that support
RLIMIT_RSS
forsetrlimit()
.The following operating systems do not support the memory limit at the OS level:
- Microsoft Windows
- Sun Solaris 2.x
Process limit
Job syntax (bsub) Queue syntax (lsb.queues) Fomat/Default Units-p
process_limit
PROCESSLIMIT=[default
]maximum
integer
Sets the limit of the number of processes to
process_limit
for the whole job. The default is no limit. Exceeding the limit causes the job to terminate.Limits the number of concurrent processes that can be part of a job.
If a default process limit is specified, jobs submitted to the queue without a job-level process limit are killed when the default process limit is reached.
If you specify only one limit, it is the maximum, or hard, process limit. If you specify two limits, the first one is the default, or soft, process limit, and the second one is the maximum process limit.
Run time limit
Job syntax (bsub) Queue syntax (lsb.queues) Fomat/Default Units-W
run_limit
RUNLIMIT=[default] maximum [hours
:]minutes
[
/host_name
|
/host_model
]
A run time limit is the maximum amount of time a job can run before it is terminated. It sets the run time limit of a job. The default is no limit. If the accumulated time the job has spent in the RUN state exceeds this limit, the job is sent a USR2 signal. If the job does not terminate within 10 minutes after being sent this signal, it is killed.
With deadline constraint scheduling configured, a run limit also specifies the amount of time a job is expected to take, and the minimum amount of time that must be available before a job can be started.
Jobs submitted to a chunk job queue are not chunked if the run limit is greater than 30 minutes.
Format
run_limit
is in the form [hour
:]minute
, whereminute
can be greater than 59. 3.5 hours can either be specified as 3:30 or 210.Normalized run time
The run time limit is
normalized
according to the CPU factor of the submission host and execution host. The run limit is scaled so that the job has approximately the same run time for a given run limit, even if it is sent to a host with a faster or slower CPU.For example, if a job is submitted from a host with a CPU factor of 2 and executed on a host with a CPU factor of 3, the run limit is multiplied by 2/3 because the execution host can do the same amount of work as the submission host in 2/3 of the time.
If the optional
host name or host model is not given, the run limit is scaled based on the DEFAULT_HOST_SPEC specified in the
lsb.params
file. (If DEFAULT_HOST_SPEC is not defined, the fastest batch host in the cluster is used as the default.) If host or host model is given, its CPU scaling factor is used to adjust the actual run limit at the execution host.The following example specifies that
myjob
can run for 10 minutes on a DEC3000 host, or the corresponding time on any other host:bsub -W 10/DEC3000 myjob
If ABS_RUNLIMIT=Y is defined in
lsb.params
, the run time limit isnot
normalized by the host CPU factor. Absolute wall-clock run time is used for all jobs submitted with a run limit.See CPU Time and Run Time Normalization for more information.
Platform MultiCluster
For MultiCluster jobs, if no other CPU time normalization host is defined and information about the submission host is not available, LSF uses the host with the largest CPU factor (the fastest host in the cluster). The ABS_RUNLIMIT parameter in
lsb.params
is is not supported in either MultiCluster model; run time limit is normalized by the CPU factor of the execution host.Thread limit
Job syntax (bsub) Queue syntax (lsb.queues) Fomat/Default Units-T
thread_limit
THREADLIMIT=[default
]maximum
integer
Sets the limit of the number of concurrent threads to
thread_limit
for the whole job. The default is no limit.Exceeding the limit causes the job to terminate. The system sends the following signals in sequence to all processes belongs to the job: SIGINT, SIGTERM, and SIGKILL.
If a default thread limit is specified, jobs submitted to the queue without a job-level thread limit are killed when the default thread limit is reached.
If you specify only one limit, it is the maximum, or hard, thread limit. If you specify two limits, the first one is the default, or soft, thread limit, and the second one is the maximum thread limit.
Stack segment size limit
Job syntax (bsub) Queue syntax (lsb.queues) Fomat/Default Units-S
stack_limit
STACKLIMIT=limit
integer
KB
Sets a per-process (soft) stack segment size limit for all of the processes belonging to a job.
By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in
lsf.conf
to specify a larger unit for the the limit (MB, GB, TB, PB, or EB).An
sbrk()
call to extend the stack segment beyond the stack limit causes the process to be terminated. The default is no soft limit.Virtual memory (swap) limit
Job syntax (bsub) Queue syntax (lsb.queues) Fomat/Default Units-v
swap_limit
SWAPLIMIT=limit
integer
KB
Sets a total process virtual memory limit for the whole job. The default is no limit. Exceeding the limit causes the job to terminate.
By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in
lsf.conf
to specify a larger unit for the the limit (MB, GB, TB, PB, or EB).This limit applies to the whole job, no matter how many processes the job may contain.
Examples
Queue-level limits
CPULIMIT = 20/hostA 15
The first number is the default CPU limit. The second number is the maximum CPU limit.
However, the default CPU limit is ignored because it is a higher value than the maximum CPU limit.
CPULIMIT = 10/hostA
In this example, the lack of a second number specifies that there is no default CPU limit. The specified number is considered as the default and maximum CPU limit.
RUNLIMIT = 10/hostA 15
The first number is the default run limit. The second number is the maximum run limit.
The first number specifies that the default run limit is to be used for jobs that are submitted without a specified run limit (without the
-W
option ofbsub
).
RUNLIMIT = 10/hostA
No default run limit is specified. The specified number is considered as the default and maximum run limit.
THREADLIMIT=6
No default thread limit is specified. The value 6 is the default and maximum thread limit.
THREADLIMIT=6 8
The first value (6) is the default thread limit. The second value (8) is the maximum thread limit.
Job-level limits
bsub -M 5000 myjob
Submits
myjob
with a memory limit of 5000 KB.
bsub -W 14 myjob
myjob
is expected to run for 14 minutes. If the run limit specified withbsub -W
exceeds the value for the queue, the job is rejected.
bsub -T 4 myjob
Submits
myjob
with a maximum number of concurrent threads of 4.CPU Time and Run Time Normalization
To set the CPU time limit and run time limit for jobs in a platform-independent way, LSF scales the limits by the CPU factor of the hosts involved. When a job is dispatched to a host for execution, the limits are then normalized according to the CPU factor of the execution host.
Whenever a normalized CPU time or run time is given, the actual time on the execution host is the specified time multiplied by the CPU factor of the normalization host then divided by the CPU factor of the execution host.
If ABS_RUNLIMIT=Y is defined in
lsb.params
or inlsb.applications
for the application associated with your job, the run time limit and run time estimate are not normalized by the host CPU factor. Absolute wall-clock run time is used for all jobs submitted with a run time limit or a run time estimate.Normalization host
If no host or host model is given with the CPU time or run time, LSF uses the default CPU time normalization host defined at the queue level (DEFAULT_HOST_SPEC in
lsb.queues
) if it has been configured, otherwise uses the default CPU time normalization host defined at the cluster level (DEFAULT_HOST_SPEC inlsb.params
) if it has been configured, otherwise uses the submission host.Example
CPULIMIT=10/hostA
If
hostA
has a CPU factor of 2, andhostB
has a CPU factor of 1 (hostB
is slower thanhostA
), this specifies an actual time limit of 10 minutes onhostA
, or on any other host that has a CPU factor of 2. However, ifhostB
is the execution host, the actual time limit onhostB
is 20 minutes (10 * 2 / 1).Normalization hosts for default CPU and run time limits
The first valid CPU factor encountered is used for both CPU limit and run time limit. To be valid, a host specification must be a valid host name that is a member of the LSF cluster. The CPU factor is used even if the specified limit is not valid.
If the CPU and run limit have different host specifications, the CPU limit host specification is enforced.
If no host or host model is given with the CPU or run time limits, LSF determines the default normalization host according to the following priority:
- DEFAULT_HOST_SPEC is configured in
lsb.queues
- DEFAULT_HOST_SPEC is configured in
lsb.params
- If DEFAULT_HOST_SPEC is not configured in
lsb.queues
orlsb.params
, host with the largest CPU factor is used.CPU time display (bacct, bhist, bqueues)
Normalized CPU time is displayed in the output of
bqueues
. CPU time isnot
normalized in the output ifbacct
andbhist
.PAM resource limits
PAM limits are system resource limits defined in
limits.conf
.
- Windows: Not applicable
- Linux:
/etc/pam.d/lsf
When
USE_PAM_CREDS
is enabled inlsb.queues
orlsb.applications
, applies PAM limits to an application or queue when its job is dispatched to a Linux host using PAM. The job will fail if the execution host does not have PAM configured.Configure a PAM file
When
USE_PAM_CREDS
is enabled inlsb.queues
orlsb.applications
, the limits specified in the PAM configuration file are applied to an application or queue when its job is dispatched to a Linux host using PAM. The job will fail if the execution host does not have PAM configured.
- Create a PAM configuration file on each execution host you want.
/etc/pam.d/lsf/limits.conf
- In the first two lines, specify the authentication and authorization you need to successfully run PAM limits. For example:
auth required pam_localuser.so
account required pam_unix.so
- Specify any resource limits. For example:
session required pam_limits.so
On hosts that have a PAM configuration file with resource limits specified and when
USE_PAM_CREDS=y
inlsb.queues
orlsb.applications
, applies resource limits on jobs running on the execution host.For more information about configuring a PAM file, check Linux documentation.
Platform Computing Inc.
www.platform.com |
Knowledge Center Contents Previous Next Index |