Learn more about Platform products at http://www.platform.com

[ Platform Documentation ] [ Title ] [ Contents ] [ Previous ] [ Next ] [ Index ]



Using Platform LSF HPC with Intel MPI


Contents

[ Top ]


About Platform LSF HPC and the Intel® MPI Library

The Intel® MPI Library ("Intel MPI") is a high-performance message-passing library for developing applications that can run on multiple cluster interconnects chosen by the user at runtime. It supports TCP, shared memory, and high-speed interconnects like InfiniBand and Myrinet.

Intel MPI supports all MPI-1 features and many MPI-2 features, including file I/O, generalized requests, and preliminary thread support. it is based on the MPICH2 specification.

The LSF HPC Intel® MPI integration is based on the LSF HPC generic PJL framework. It supports the LSF HPC task geometry feature.

Requirements

Assumptions and limitations

Glossary

MPD

Multi-Purpose Daemon (MPD) job startup mechanism

MPI

(Message Passing Interface) A message passing standard. It defines a message passing API useful for parallel and distributed applications.

MPICH

A portable implementation of the MPI standard.

MPICH2

An MPI implementation for platforms such as clusters, SMPs, and massively parallel processors.

PAM

(Parallel Application Manager) The supervisor of any parallel job.

PJL

(Parallel Job Launcher) Any executable script or binary capable of starting parallel tasks on all hosts assigned for a parallel job.

RES

(Remote Execution Server) An LSF daemon residing on each host. It monitors and manages all LSF tasks on the host.

TS

(TaskStarter) An executable responsible for starting a task on the local host and reporting the process ID and host name to the PAM.

For more information

Files installed by lsfinstall

During installation, lsfinstall copies these files to the following directories

:
These files... Are installed to...
TaskStarter
LSF_BINDIR
pam
LSF_BINDIR
esub.intelmpi
LSF_SERVERDIR
intelmpi_wrapper
LSF_BINDIR
mpirun.lsf
LSF_BINDIR
pjllib.sh
LSF_BINDIR

Resources and parameters configured by lsfinstall

[ Top ]


Configuring LSF HPC to Work with Intel MPI

intelmpi_wrapper script

Modify the intelmpi_wrapper script in LSF_BINDIR to set MPI_TOPDIR The default value is:

MPI_TOPDIR="/opt/intel/mpi/2.0"

lsf.conf (optional)

To improve performance and scalability for large parallel jobs, tune the following parameters as described in Tuning PAM Scalability and Fault Tolerance:

The user's environment can override these.

[ Top ]


Working with the Multi-purpose Daemon (MPD)

The Intel® MPI Library ("Intel MPI") uses a Multi-Purpose Daemon (MPD) job startup mechanism. MPD daemons must be up and running on the hosts where an MPI job is supposed to start before mpiexec is started.

How Platform LSF HPC manages MPD rings

LSF HPC manages MPD rings for users automatically using mpdboot and mpdtrace commands.

Each MPI job running under LSF uses a uniquely labeled MPD ring. The ring is started by the intelmpi_wrapper during job launch and terminated by the intelmpi_wrapper after MPI application exits, either normally or abnormally. This allows coexistence of multiple MPI jobs belonging to different users as well as multiple jobs from the same user on the same set of hosts.

For more information

[ Top ]


Submitting Intel MPI Jobs

bsub command

Use bsub -a intelmpi to submit jobs.

If the starting command is mpd, you must submit your Intel MPI jobs as exclusive jobs (bsub -x).

bsub -a intelmpi -n number_cpus mpirun.lsf 
[-pam "pam_options"] [mpi_options] job [job_options]

For example:

% bsub -a intelmpi -n 3 mpirun.lsf /examples/cpi

A job named cpi will be dispatched and run on 3 CPUs in parallel.

Task geometry with Intel MPI jobs

Intel MPI supports the LSF HPC task geometry feature

Submitting a job with a job script

A wrapper script is often used to call Intel MPI. You can submit a job using a job script as an embedded script or directly as a job, for example:

% bsub -a intelmpi -n 4 < embedded_jobscript
% bsub -a intelmpi -n 4 jobscript

Your job script must use mpirun.lsf in place of the mpirun command.

Using Intel MPI configuration files (-configfile)

All mpiexec -configfile options are supported. -configfile should be the only option after the mpiexec command.

The placement options in the configuration file (-gn, -gnp, -n, -np, -host) must agree with the value of the LSB_MCPU_HOSTS and LSB_HOSTS environment variables.

mpiexec limitations

-file option is not supported

The -file option of mpiexec is not supported. You can use the -configfile option.

If you submit an Intel MPI job with -file, the intelmpi_wrapper will exit and fail the job. If you specify the log file for intelmpi_wrapper, an error message is appended to the log file:

Official host names

mpiexec requires host names as they are returned by the hostname command or the gethostname() system call. For example:

% hostname
hosta
% mpiexec -l -n 2 -host hosta.domain.com ./hmpi
mpdrun: unable to start all procs; may have invalid machine 
names
    remaining specified hosts:
        hosta.domain.com

% mpiexec -l -n 2 -host hosta ./hmpi
0: myrank 0, n_processes 2
1: myrank 1, n_processes 2
0: From process 1: Slave process 1!

-genvlist option

The -genvlist options does not work if the configuration file for -configfile has more than one entry.

For more information

[ Top ]


[ Platform Documentation ] [ Title ] [ Contents ] [ Previous ] [ Next ] [ Index ]


      Date Modified: August 20, 2009
Platform Computing: www.platform.com

Platform Support: support@platform.com
Platform Information Development: doc@platform.com

Copyright © 1994-2009 Platform Computing Corporation. All rights reserved.