Learn more about Platform products at http://www.platform.com

[ Platform Documentation ] [ Title ] [ Contents ] [ Previous ] [ Next ] [ Index ]



Using Platform LSF HPC with MPICH2


Contents

[ Top ]


About Platform LSF HPC and MPICH2

MPICH is a freely available, portable implementation of the MPI Standard for message- passing libraries, developed jointly with Mississippi State University. MPICH is designed to provide a high performance, portable, and convenient programming environment. MPICH2 implements both MPI-1 and MPI-2.

The mpiexec command of MPICH2 spawns all tasks, while LSF HPC retains full control over the tasks spawned. Specifically, LSF HPC collects rusage information, performs job control (signal), and cleans up after the job is finished. Jobs run within LSF allocation, controlled by LSF HPC.

Requirements

Assumptions and limitations

Glossary

MPI

(Message Passing Interface) A message passing standard. It defines a message passing API useful for parallel and distributed applications.

MPICH

A portable implementation of the MPI standard.

MPICH2

An MPI implementation that implements both MPI-1 and MPI-2.

PAM

(Parallel Application Manager) The supervisor of any parallel job.

PJL

(Parallel Job Launcher) Any executable script or binary capable of starting parallel tasks on all hosts assigned for a parallel job.

RES

(Remote Execution Server) An LSF daemon residing on each host. It monitors and manages all LSF tasks on the host.

TS

(TaskStarter) An executable responsible for starting a task on the local host and reporting the process ID and host name to the PAM.

For more information

See the Mathematics and Computer Science Division (MCS) of Argonne National Laboratory (ANL) MPICH Web page at www-unix.mcs.anl.gov/mpi/mpich/ for more information about MPICH and MPICH2.

Files installed by lsfinstall

During installation, lsfinstall copies these files to the following directories:

These files... Are installed to...
TaskStarter
LSF_BINDIR
pam
LSF_BINDIR
esub.mpich2
LSF_SERVERDIR
mpich2_wrapper
LSF_BINDIR
mpirun.lsf
LSF_BINDIR
pjllib.sh
LSF_BINDIR

Resources and parameters configured by lsfinstall

[ Top ]


Configuring LSF HPC to Work with MPICH2

  1. Make sure MPICH2 commands are in the PATH environment variable. MPICH2 commands include mpiexec, mpd, mpdboot, mpdtrace, and mpdexit.

    For example:

[174]- which mpiexec /pcc/app/mpich2/kernel2.4-glibc2.3-x86/bin/mpiexec

  1. Add an mpich2 boolean resource to the $LSF_ENVDIR/lsf.shared file.

    For example:

hmmer        Boolean     ()     ()     (hmmer availability) 
lammpi       Boolean     ()     ()     (lam-mpi available host) 
mpich2       Boolean     ()     ()     (mpich2 available host) <==== 
End Resource 
  1. Add mpich2 to each host that an mpich2 parallel job may run on.

    For example:

Begin  Host 
HOSTNAME  model  type  server  r1m  mem  swp  RESOURCES   #Keywords 
qat20      !      !        1   3.5  ()   ()   (mpich2) 
qat21      !      !        1   3.5  ()   ()   (mpich2) 
qat22      !      !        1   3.5  ()   ()   (mpich2) 
End  Host 
  1. Run lsadmin reconfig and badmin mbdrestart as root.
  2. Run lshosts to confirm that an mpich2 resource is configured on all hosts on which you would like to run mpich2 parallel jobs.

    For example:

[173]- lshosts 
HOST_NAME      type    model  cpuf  ncpus  maxmem  maxswp  server  RESOURCES 
qat20       LINUX86   PC1133  23.1      1    310M       -     Yes  (mpich2) 
qat21.lsf.p LINUX86   PC1133  23.1      1    311M    635M     Yes  (mpich2) 
qat22.lsf.p UNKNOWN UNKNOWN_   1.0      -       -       -     Yes  (mpich2)

  1. Configure and start an MPD ring.
    1. If you want to start an MPD ring per job, this is the default and recommended mechanism, and you do not need to do any extra configuration.
    2. If you want to start an MPD ring for all users, use the mpdboot command as root on all machines.

      To check if mpdboot ran successfully, use the mpdtrace command

[root@qat20 test]# mpdtrace -l 

qat20_37272 
qat21_52535 

      1. For MPICH2 1.0.3 only, add the following lines to $HOME/.mpd.conf for all users.
[61]- cat .mpd.conf 
MPD_USE_ROOT_MPD=Y <========== 
secretword=123579a 
      1. Make sure $HOME/.mpd.conf has a permission mode of 600 after you finish the modification.
      2. Set LSF_START_MPD_RING=N in your job script or in the environment for all users.
    1. If you want to start an MPD ring on all hosts, follow the steps described in the MPICH2 documentation to start an MPD ring across all LSF hosts for each user. The user MPD ring must be running all the time, and you must set LSF_START_MPD_RING=N in your job script or in the environment for all users.


Do not run mpdallexit or mpdcleanup to terminate the MPD ring.

  1. Make sure LSF uses system host official names (/etc/hosts): this will prevent problems when you run the application.
      1. Configure the $LSF_CONFDIRDIR/hosts file and the $LSF_ENVDIR/lsf.cluster.<clustername> file.

        For example:

172.25.238.91 scali scali.lsf.platform.com
172.25.238.96 scali1 scali1.lsf.plaform.com

      1. If the official host name returned to LSF is a short name, but LSF commands display host names that include domain names, you can use LSF_STRIP_DOMAIN in lsf.conf to display the short names.
  1. Change the $LSF_BINDIR/mpich2_wrapper script to make sure MPI_TOPDIR= points to the MPICH2 install directory.

[ Top ]


Building Parallel Jobs

  1. Use mpicc -o to compile your source code.

    For example:

    [178]- which mpicc /pcc/app/mpich2/kernel2.4-glibc2.3- x86/bin/mpicc

    5:19pm Mon, Sep-19-2005 qat21:~/milkyway/bugfix/test

    [179]- mpicc -o hw.mpich2 hw.c 3.2

  2. Make sure the compiled binary can run under the root MPD ring outside Platform LSF HPC.

    For example:

    [180]- mpiexec -np 2 hw.mpich2

Process 0 is printing on qat21 (pid =16160): 
Greetings from process 1 from qat20 pid 24787! 

[ Top ]


Submitting MPICH2 Jobs

bsub command

Use the bsub command to submit MPICH2 jobs.

  1. Submit a job from the console command line:
bsub <bsub_options> -n <###> -a mpich2 mpirun.lsf <mpiexec_options> job 
<job_options> 


Note that -np options of mpiexec will be ignored.

For example:

bsub -I -n 8 -R "span[ptile=4]" -a mpich2 -W 2 mpirun.lsf -np 3 ./hw.mpich2 
  1. Submit a job using a script:

    bsub < myjobscript.sh

    where myjobscript.sh looks like:

#!/bin/sh 
#BSUB -n 8 
#BSUB -a mpich2 
mpirun.lsf ./hw.mpich2 

The mpich2_wrapper script supports almost all original mpiexec options except those that will affect job scheduling decisions, for example, -np (-n).

-n syntax is supported. If you use the -n option, you must either request enough CPUs when the job is submitted, or set the environment variable LSB_PJL_TASK_GEOMETRY. See Running Jobs with Task Geometry for detailed usage of LSB_PJL_TASK_GEOMETRY.

Task geometry with MPICH2 jobs

MPICH2 mpirun requires the first task to run on the local node OR all tasks to run on a remote node (-nolocal). If the LSB_PJL_TASK_GEOMETRY environment variable is set, mpirun.lsf makes sure the task group that contains task 0 in LSB_PJL_TASK_GEOMETRY runs on the first node.

The environment variable LSB_PJL_TASK_GEOMETRY is checked for all parallel jobs. If LSB_PJL_TASK_GEOMETRY is set users submit a parallel job (a job that requests more than 1 slot), LSF attempts to shape LSB_MCPU_HOSTS accordingly.

[ Top ]


[ Platform Documentation ] [ Title ] [ Contents ] [ Previous ] [ Next ] [ Index ]


      Date Modified: August 20, 2009
Platform Computing: www.platform.com

Platform Support: support@platform.com
Platform Information Development: doc@platform.com

Copyright © 1994-2009 Platform Computing Corporation. All rights reserved.