# How to use the mpirun command on JUGENE?

## MPIRUN Options

Programs can be launched, controlled and monitored on JUGENE using the mpirun command.
The general syntax of this command is: mpirun [options]

mpirun offers the possibility to control the environment and the execution of an application using numerous parameters which can either be set by command-line options or by environment variables. The command-line options have higher priority, i.e. they override parameters specified by environment variables. In the following we describe some useful parameters for mpirun giving the command-line option / the corresponding environment variable.

-args "prg_args" / MPIRUN_ARGS="prg_args"

Passes "prg_args" to the launched application on the compute node.

-cwd path / MPIRUN_CWD=path or MPIRUN_WDIR=path

Specifies the full path to the current working directory on the compute nodes. The path must be specified as seen by the I/O and the compute nodes.

-env "ENVVAR=values" / MPIRUN_ENV="ENVVAR=value"

Sets the environment variable ENVVAR=value in the environment of the job on the compute nodes.

-env_all / MPIRUN_EXP_ENV_ALL

Exports all environment variables in the current environment of mpirun to the job on the compute nodes.

-exe executable / MPIRUN_EXE=executable

Specifies the full path to the executable to be launched on the compute nodes. The path must be specified as seen by the I/O and the compute nodes.

-exp_env ENVVAR / MPIRUN_EXP_ENV=ENVVAR

Exports the environment variable ENVVAR in the current environment of mpirun to the job on the compute nodes.

-mapfile mapfile / MPIRUN_MAPFILE=mapfile

The user can specify with this option the order in which the MPI tasks are distributed across the nodes and cores of the partition reserved. The standard mapping on JUGENE is to place the tasks in XYZT order, where X,Y, and Z are the torus coordinates of the nodes in the partition and T is the number of the cores within each node (T=0,1,2,3). When the tasks are distributed across the nodes the first dimension is increased first, i.e. for XYZT the first three tasks would be executed by the nodes with the torus coordinates <0,0,0,0>, <1,0,0,0> and <2,0,0,0>. 'mapfile' can either be a permutation of X,Y,Z and T or the name of a mapfile in which the distribution of the tasks is specified.

-mode SMP|DUAL|VN / MPIRUN_MODE=SMP|DUAL|VN

Specifies the mode in which the job will run. A node on JUGENE contains 4 cores which share 2 GB of physical memory. With this option, the user can decide how many tasks are executed by each node.

 SMP (Symmetrical Multiprocessing mode) Each node executes one task (2 GB of memory per task available) with up to 4 threads. This is the default mode on JUGENE. DUAL (Dual Node mode) Each node executes two tasks (2 GB of memory are shared between the tasks) with up to 2 threads per task. VN (Virtual Node mode) Each node executes 4 tasks (2 GB of memory are shared between the tasks).

-np n / MPIRUN_NP=n

Creates n MPI tasks. This parameter is only needed, if you want to use less nodes or cores than you have allocated. In general, mpirun creates automatically as many tasks as will fit on the partition booted. For example, if you reserve 1024 nodes and run in VN mode, mpirun will create 4096 MPI tasks (4 tasks per allocated node, see option -mode VN).

-verbose n / MPIRUN_VERBOSE=n

Sets the verbosity level of mpirun to n=1,2,3,4. All information generated by mpirun are passed to STDERR.

Hint: We recommend to use the -verbose 2 option even if your job finishes with exit code 0. Events might occur during the run of your application which are not critical (i.e. the application does not crash) but which can provide hints how to set certain parameters in order to obtain a better performance (see Ras Events for further information). Such events are reported only at verbosity level 2 or higher.

## Examples

Here we provide some examples using the command-line options of mpirun within a loadleveler script.

### Example 1

# @ job_name = example_1
# @ error = $(job_name).$(jobid).out
# @ output = $(job_name).$(jobid).out
# @ environment = COPY_ALL;# @ wall_clock_limit = 00:20:00
# @ job_type = bluegene# @ bg_size = 32
# @ queuempirun -exe myexe -mode VN -args "param.inp"

mpirun will launch the executable "myexe" in Virtual Node mode and passes "param.inp" as argument to the executable. The partition contains 32 nodes (bg_size = 32), 4 tasks should be executed by each node (VN mode), therefore 32x4=128 MPI tasks are created.

### Example 2

# @ job_name = example_1
# @ error = $(job_name).$(jobid).out
# @ output = $(job_name).$(jobid).out
# @ environment = COPY_ALL;
# @ wall_clock_limit = 12:00:00# @ notification = never
# @ job_type = bluegene# @ bg_connection = TORUS
# @ bg_size = 1024
# @ queuempirun -exe myexe -env OMP_NUM_THREADS=4

This is an example for running a hybrid application on JUGENE. mpirun will launch the executable "myexe" in SMP mode (default) and each task will start 4 threads. In total, 1024 MPI tasks are created.

### Example 3

# @ job_name = example_1
# @ error = $(job_name).$(jobid).out
# @ output = $(job_name).$(jobid).out
# @ environment = COPY_ALL;
# @ wall_clock_limit = 12:00:00
# @ job_type = bluegene
# @ bg_connection = TORUS
# @ bg_size = 1024
# @ queuempirun -exe myexe -env OMP_NUM_THREADS=4

mpirun will launch the executable "myexe" in Virtual Node mode in the directory /work/xyz001/run, the environment variable PRGOPT will be exported. The mapping order is set to TXYZ, i.e. the MPI tasks are distributed is such a way that each node is filled with 4 MPI tasks (T dimension first) before the next node is filled. 8192 MPI tasks are created in total.