On Quarry, how do I run multiple MPI jobs using a TORQUE script?

MPI libraries available through SoftEnv

The MPICH1 and OpenMPI libraries are installed on Quarry at Indiana University. To display the MPI libraries available on Quarry, use the softenv command and use grep to search for mpi , as follows:

[jdoe@Quarry] softenv | grep -i mpi . . . @lam A MPI implementation before OpenMPI, still in use @mpich1 Default MPICH1 (compiled with Intel compilers) @openmpi Default openmpi (compiled with Intel compilers) . . .

Submitting a parallel job on Quarry

To submit parallel MPICH1 jobs on Quarry:

  1. Add the MPICH1 key to SoftEnv.

    Add @mpich1 to your ~/.soft file. Your ~/.soft file should look similar to this:

    @mpich1 ## Some more keys you may have @quarry

    If you are not familiar with the SoftEnv system, see On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment? You may also enter man softenv-intro at the prompt on Big Red.

  2. Compile your MPI code: [jdoe@Quarry] mpicc -o myprog myprog.c
  3. Submit the parallel job to TORQUE (also called PBS).

    To submit a parallel job that runs your MPI program, first edit the sample script shown below, submit_parallel.sh (or create your own) with the correct number of nodes and tasks, the appropriate output/error files, and so forth:

    #PBS -l nodes=2:ppn=8,walltime=5:00 #PBS -m ae #PBS -N job_myprog mpirun -machine $PBS_NODEFILE -np 16 ${HOME}/simple_quarry_jobs/myprog

    Then use qsub to submit the program:

    qsub submit_parallel.sh

If you are not familiar with job submission on Quarry, refer to What is TORQUE/PBS?