link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service

Program execution on JUQUEEN with runjob

Command runjob

On a Blue Gene/Q system, the program to execute an application is always the command

runjob [options]

That means preparing a job for submission requires you to create a job command file that passes the appropriate arguments to the runjob command by adding the runjob call after the #@queue keyword in the job file.


--ranks-per-node <number>

Number of tasks per compute node

--args <prg_arg>

Passes "prg_arg" to the launched application on the compute node. For two or more arguments you need to separate the arguments with blanks, for arguments with options specify --args several times to assign the values correctly.

For example: (see also man runjob)

runjob --ranks-per-node 32 --args hello world --exe a.out

runjob --ranks-per-node 32 --args --one foo --args --two bar --exe a.out

--exe <executable>

Specifies the full path to the executable to be launched on the compute nodes. The path must be specified as seen by the I/O and the compute nodes.

--envs <ENV_Var1=Value1>[ <ENV_Var2=Value2> ...]

Sets the environment variable ENV_Var1=Value1in the environment of the job on the compute nodes. Multiple variables can be added as a space-separated list.

--exp-env <ENV_Var>

Sets the environment variable ENV_Var in the current environment of runjob to the job on the compute nodes.

--np <number>

Number of total MPI-tasks


You can also call runjob in the following format:

runjob [options] : exe [arg1 arg2 .. argn].

This way you can especially omit several calls of --args,e.g. the examples for --arg above can also be written as:

runjob --ranks-per-node 32 : a.out hello world

runjob --ranks-per-node 32 : a.out --one foo --two bar

For more details and further options please see runjob -h or man runjob.

A complete explanation of runjob can be found in the BG/Q system Administration manual.


Example for a hybrid job using 32 nodes with 1 task per node using 64 threads, for the threads the environment variable OMP_NUM_THREADS is used and exported to the job.

Note that the number of nodes are already specified via bg_size, it is not necessary to additionally specify --np if the full node number is used.

# @ job_name = Example
# @ comment = "Example Job"
# @ error = $(job_name).$(jobid).out
# @ output = $(job_name).$(jobid).out
# @ environment = COPY_ALL
# @ wall_clock_limit = 00:30:00
# @ notification = error
# @ notify_user =
# @ job_type = bluegene
# @ bg_size = 32
# @ queue

runjob --ranks-per-node 1 --exp-env OMP_NUM_THREADS --exe ./example_exe

Further examples can be found here.