Knowledge Center Contents Previous Next Index |
External Job Submission and Execution Controls
This document describes the use of external job submission and execution controls called
esub
andeexec
. These site-specific user-written executables are used to validate, modify, and reject job submissions, pass data to and modify job execution environments.Contents
Understanding External Executables
About esub and eexec
LSF provides the ability to validate, modify, or reject job submissions, modify execution environments, and pass data from the submission host directly to the execution host through the use of the
esub
andeexec
executables. Both are site-specific and user written and must be located in LSF_SERVERDIR.Validate, modify, or reject a job
To validate, modify, or reject a job, an
esub
needs to be written. See Using esubModifying execution environments
To modify the execution environment on the execution host, an
eexec
needs to be written. See Working with eexecPassing data
To pass data directly to the execution host, an
esub
andeexec
need to be written. See Using esub and eexec to pass data to execution environmentsInteractive remote execution
Interactive remote execution also runs
esub
andeexec
if they are found in LSF_SERVERDIR. For example,lsrun
invokesesub
, and RES runseexec
before starting the task.esub
is invoked at the time of thels_connect
(3) call, and RES invokeseexec
each time a remote task is executed. RES runseexec
only at task startup time.DCE credentials and AFS tokens
esub
andeexec
are also used for processing DCE credentials and AFS tokens. See the following documents on the Platform Web site for more information:
- "Installing LSF on AFS"
- "Installing LSF on DCE/DFS"
Using esub
About esub
An
esub
, short forexternal submission
, is a user-written executable (binary or script) that can be used to validate, modify, or reject jobs. Theesub
is put into LSF_SERVERDIR (defined inlsf.conf
) where LSF checks for its existence when a job is submitted, restarted, and modified. If LSF finds anesub
, it is run by LSF. Whether the job is submitted, modified, or rejected depends on the logic built into theesub
.Any messages that need to be provided to the user should be directed to the standard error (
stderr
) stream and not the standard output (stdout
) stream.In this section
- Environment variables to bridge esub and LSF
- General esub logic
- Rejecting jobs
- Validating job submission parameters
- Modifying job submission parameters
- Using bmod and brestart commands with mesub
- Use multiple esub (mesub)
- How master esub invokes application-specific esubs
- Configure master esub and your application-specific esub
Environment variables to bridge esub and LSF
LSF provides the following environment variables in the
esub
execution environment:LSB_SUB_PARM_FILE
This variable points to a temporary file containing the job parameters that
esub
reads when the job is submitted. The submission parameters are a set of name-value pairs on separate lines in the format "option_name
=value
".The following option names are supported:
Example submission parameter file
If a user submits the following job:
bsub -q normal -x -P my_project -R "r1m rusage[dummy=1]" -n 90 sleep 10
The contents of the LSB_SUB_PARM_FILE will be:
LSB_SUB_QUEUE="normal" LSB_SUB_EXCLUSIVE=Y LSB_SUB_RES_REQ="r1m rusage[dummy=1]" LSB_SUB_PROJECT_NAME="my_project" LSB_SUB_COMMAND_LINE="sleep 10" LSB_SUB_NUM_PROCESSORS=90 LSB_SUB_MAX_NUM_PROCESSORS=90LSB_SUB_ABORT_VALUE
This variable indicates the value
esub
should exit with if LSF is to reject the job submission.LSB_SUB_MODIFY_ENVFILE
The file in which
esub
should write any changes to the job environment variables.
esub
writes the variables to be modified to this file in the same format used in LSB_SUB_PARM_FILE. The order of the variables does not matter.After
esub
runs, LSF checks LSB_SUB_MODIFY_ENVFILE for changes and if found, LSF will apply them to the job environment variables.LSB_SUB_MODIFY_FILE
The file in which
esub
should write any submission parameter changes.
esub
writes the job options to be modified to this file in the same format used in LSB_SUB_PARM_FILE. The order of the options does not matter. Afteresub
runs, LSF checks LSB_SUB_MODIFY_FILE for changes and if found LSF will apply them to the job.
tip:
LSB_SUB_ADDITIONAL cannot be changed in or added to LSB_SUB_MODIFY_FILE.LSF_INVOKE_CMD
Indicates the name of the last LSF command that invoked an external executable (for example,
esub
).External executables get called by several LSF commands (
bsub
,bmod
,lsrun
). This variable contains the name of the last LSF command to call the executable.General esub logic
After
esub
runs, LSF checks:
- Is the
esub
exit value LSB_SUB_ABORT_VALUE?
- Yes, step 2
- No, step 4
- Reject the job
- Go to step 5
- Does LSB_SUB_MODIFY_FILE or LSB_SUB_MODIFY_ENVFILE exist?
- Apply changes
- Done
Rejecting jobs
Depending on your policies you may choose to reject a job. To do so, have
esub
exit with LSB_SUB_ABORT_VALUE.If
esub
rejects the job, it should not write to either LSB_SUB_MODIFY_FILE or LSB_SUB_MODIFY_ENVFILE.Example
The following Bourne shell
esub
rejects all job submissions by exiting with LSB_SUB_ABORT_VALUE:#!/bin/sh # Redirect stderr to stdout so echo can be used for # error messages exec 1>&2 # Reject the submission echo "LSF is Rejecting your job submission..." exit $LSB_SUB_ABORT_VALUEValidating job submission parameters
One use of validation is to support project-based accounting. The user can request that the resources used by a job be charged to a particular project. Projects are associated with a job at job submission time, so LSF will accept any arbitrary string for a project name. In order to ensure that only valid projects are entered and the user is eligible to charge to that project, an
esub
can be written.Example
The following Bourne shell
esub
validates job submission parameters:#!/bin/sh . $LSB_SUB_PARM_FILE # Redirect stdout to stderr so echo can be used for error messages exec 1>&2 # Check valid projects if [ $LSB_SUB_PROJECT_NAME != "proj1" -o $LSB_SUB_PROJECT_NAME != "proj2" ]; then echo "Incorrect project name specified" exit $LSB_SUB_ABORT_VALUE fi USER=`whoami` if [ $LSB_SUB_PROJECT_NAME = "proj1" ]; then # Only user1 and user2 can charge to proj1 if [$USER != "user1" -a $USER != "user2" ]; then echo "You are not allowed to charge to this project" exit $LSB_SUB_ABORT_VALUE fi fiModifying job submission parameters
esub
can be used to modify submission parameters and the job environment before the job is actually submitted.The following example writes modifications to LSB_SUB_MODIFY_FILE for the following parameters:
- LSB_SUB_QUEUE
- USER
- SHELL
In the example, user
userA
can only submit jobs to queuequeueA
. UseruserB
must use Bourne shell (/bin/sh
), and useruserC
should never be able to submit a job.#!/bin/sh . $LSB_SUB_PARM_FILE # Redirect stderr to stdout so echo can be used for error messages exec 1>&2 USER=`whoami` # Ensure userA is using the right queue queueA if [ $USER="userA" -a $LSB_SUB_QUEUE != "queueA" ]; then echo "userA has submitted a job to an incorrect queue" echo "...submitting to queueA" echo 'LSB_SUB_QUEUE="queueA"' > $LSB_SUB_MODIFY_FILE fi # Ensure userB is using the right shell (/bin/sh) if [ $USER="userB" -a $SHELL != "/bin/sh" ]; then echo "userB has submitted a job using $SHELL" echo "...using /bin/sh instead" echo 'SHELL="/bin/sh"' > $LSB_SUB_MODIFY_ENVFILE fi # Deny userC the ability to submit a job if [ $USER="userC" ]; then echo "You are not permitted to submit a job." exit $LSB_SUB_ABORT_VALUE fiUsing bmod and brestart commands with mesub
You can use the
bmod
command to modify job submission parameters, andbrestart
to restart checkpointed jobs. Likebsub
,bmod
andbrestart
also callmesub
, which in turn invoke any existingesub
executables in LSF_SERVERDIR.bmod
andbrestart
cannot make changes to the job environment throughmesub
andesub
. Environment changes only occur whenmesub
is called by the original job submission withbsub
.Use multiple esub (mesub)
LSF provides a master
esub
(LSF_SERVERDIR/mesub
) to handle the invocation of individual application-specificesub
executables and the job submission requirements of your applications.
- Use the
-a
option ofbsub
to specify the application you are running through LSF.For example, to submit a FLUENT job:
bsub -a fluent
bsub_options
fluent_command
The method name
fluent
, uses theesub
for FLUENT jobs (LSF_SERVERDIR/esub.fluent
), which sets the checkpointing methodLSB_ECHKPNT_METHOD="fluent"
to use theechkpnt.fluent
anderestart.fluent
.LSB_ESUB_METHOD (lsf.conf)
To specify a mandatory
esub
method that applies to all job submissions, you can configure LSB_ESUB_METHOD inlsf.conf
.LSB_ESUB_METHOD specifies the name of the
esub
method used in addition to any methods specified in thebsub
-a
option.For example,
LSB_ESUB_METHOD="dce fluent"
defines DCE as the mandatory security system, and FLUENT as the mandatory application used on all jobs.Compatibility note
restriction:
After LSF version 5.1, the value of -a and LSB_ESUB_METHOD must correspond to an actual esub file in LSF_SERVERDIR. For example, to use bsub -a fluent, the file esub.fluent must exist in LSF_SERVERDIR.How master esub invokes application-specific esubs
bsub
invokesmesub
at job submission, which callsesub
programs in this order:
- Mandatory
esub
programs defined by LSB_ESUB_METHOD- Any existing executable named
LSF_SERVERDIR/esub
- Application-specific
esub
programs in the order specified in thebsub -a
optionExample
In this example:
esub.dce
is defined as the only mandatoryesub
- An executable named
esub
already exists in LSF_SERVERDIR- Executables named
esub.fluent
andesub.license
exist in LSF_SERVERDIRbsub -a fluent license
submits the job as a FLUENT job, andmesub
invokes the following esub executables in LSF_SERVERDIR in this order:
esub.dce
esub
esub.fluent
esub.license
bsub
without the-a
option submits the job, andmesub
invokes only the mandatoryesub.dce
and the existingesub
in LSF_SERVERDIR, not the application-specificesub
programs.Configure master esub and your application-specific esub
The master
esub
is installed asLSF_SERVERDIR/mesub
. After installation:
- Create your own application-specific
esub
.- Optional. Configure LSB_ESUB_METHOD in
lsf.conf
to specify a mandatoryesub
for all job submissions.Name your esub
- Use the following naming conventions:
- On UNIX,
LSF_SERVERDIR/esub.
application
- On Windows,
LSF_SERVERDIR\esub.
application
.[exe |bat]
For FLUENT jobs, for example:
- UNIX: esub.fluent
- Windows:
esub.fluent.exe
The name of the
esub
program must be a valid file name. It can contain only alphanumeric characters, underscore (_
) and hyphen (-
).
caution:
The file name esub.user is reserved for backward compatibility. Do not use the name esub.user for your application-specific esub.Existing esub
Your existing esub does not need to follow this convention and does not need to be renamed. However, since mesub invokes any esub that follows this convention, you should move any backup copies of your esubs out of LSF_SERVERDIR or choose a name that does not follow the convention (for example, use esub_bak instead of esub.bak).
Working with eexec
About eexec
The
eexec
program runs on the execution host at job start-up and completion time and when checkpointing is initiated. It is run as the user after the job environment variables have been set. The environment variable LS_EXEC_T is set to START, END, and CHKPNT, respectively, to indicate wheneexec
is invoked.If you need to run
eexec
as a different user, such as root, you must properly define LSF_EEXEC_USER in the file/etc/lsf.sudoers
. See thePlatform LSF Configuration Reference
for information about thelsf.sudoers
file.
eexec
is expected to finish running because the parent job process waits foreexec
to finish running before proceeding. The environment variable LS_JOBPID stores the process ID of the process that invokedeexec
. Ifeexec
is intended to monitor the execution of the job,eexec
must fork a child and then have the parenteexec
process exit. Theeexec
child should periodically test that the job process is still alive using the LS_JOBPID variable.Using esub and eexec to pass data to execution environments
If
esub
needs to pass some data toeexec
, it can write the data to its standard output foreexec
to read from its standard input (stdin
). LSF effectively acts as the pipe betweenesub
andeexec
(e.g.,esub
|eexec
).Standard output (
stdout
) from anyesub
is automatically sent toeexec
.Limitation
Since
eexec
cannot handle more than one standard output stream, only oneesub
can use standard output to generate data as standard input toeexec
.For example, the
esub
for AFS (esub.afs
) sends its authentication tokens as standard output toeexec
. If you use AFS, no otheresub
can use standard output.
Platform Computing Inc.
www.platform.com |
Knowledge Center Contents Previous Next Index |