Memory preemption example

Configuration

This example uses pre_mem as the name of the preemption resource.

lsf.shared

Add the resource to the Resource section.
Begin Resource
RESOURCENAME TYPE    INTERVAL INCREASING      DESCRIPTION
pre_mem      Numeric 60       N    (external memory usage reporter)
...
End Resource

lsf.cluster.cluster_name

Add the resource to the "ResourceMap" section.
Begin ResourceMap
RESOURCENAME LOCATION
pre_mem      ([hostA] [hostB] ... [hostX]) 
#List the hosts where you want memory preemption to occur.
... End ResourceMap

lsb.params

Add the resource to the list of preemption resources.
...
PREEMPTABLE_RESOURCES=pre_mem
...

lsb.queues

Define a higher-priority queue to be the PREEMPTIVE queue by adding one line in the queue definition.
Begin Queue
QUEUE_NAME=high
PRIORITY=40
...
PREEMPTION=PREEMPTIVE
DESCRIPTION=preempt jobs in lower-priority queues
...
End Queue
Configure a job control action in a lower-priority queue, and let SIGTSTP be sent when the SUSPEND action is called. This assumes your application can catch the signal SIGTSTP and release (free) the resource (memory) it used, then suspend itself. You should also make sure that your application can catch the signal SIGCONT while it is suspended, and consume the resource (memory) again.
Begin Queue
QUEUE_NAME=low
PRIORITY=20
...
JOB_CONTROLS=SUSPEND[SIGTSTP] RESUME[SIGCONT] TERMINATE[SIGTERM]
DESCRIPTION=jobs may be preempted by jobs in higher-priority queues
...
End Queue

ELIM

This is an example of an ELIM that reports the current value of pre_mem. This ELIM starts on all the hosts that have the pre_mem resource.
#!/bin/sh
host=`hostname`
while :
do
lsload > /dev/null 2>&1
if [ $? != 0 ] ; then exit 1
fi
memStr=`lsload -I mem -w $host|grep $host|awk '{print $3}'|sed 's/M//'` reportStr="1 ""pre_mem ""$memStr"
echo "$reportStr \c"
sleep 60
done

Operation

Check how many pre_mem resources are available

Check the number of pre_mem existing on hostA by using bhosts -s pre_mem to display how much memory is available. In this example, 110 MB of memory is available on hostA.
bhosts -s pre_mem 
RESOURCE  TOTAL RESERVED LOCATION
pre_mem   110   0.0      hostA
pre_mem   50    0.0      hostB
...

Use up some pre_mem resources

Submit 1 job to a low-priority queue to consume 100 MB pre_mem. Assume the application mem_app consumes 100 MB memory after it starts.
bsub -J first -q low -R "rusage[pre_mem=100:duration=2]" mem_app
After a while, the first job is running and the pre_mem is reduced.
bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
301   you  RUN  low   hostx     hostA     /first   Aug 23 16:42
bhosts -s pre_mem 
RESOURCE TOTAL RESERVED LOCATION
pre_mem  10    100.0      hostA
pre_mem  50      0.0      hostB
...

Preempt the job for pre_mem resources

Submit a job to a high-priority queue to preempt a job from low-priority queue to get the resource pre_mem.
bsub -J second -q high -R "rusage[pre_mem=100:duration=2]" mem_app
After a while, the second job is running and the first job was suspended.
bjobs
JOBID USER STAT  QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
302   you  RUN   high  hostx     hostA     /second  Aug 23 16:48
301   you  SSUSP low   hostx     hostA     /first   Aug 23 16:42
bhosts -s pre_mem 
RESOURCE TOTAL RESERVED LOCATION
pre_mem  10    100.0      hostA
pre_mem  50      0.0      hostB
...