Program execution on JUQUEEN with runjob
On a Blue Gene/Q system, the program to execute an application is always the command
That means preparing a job for submission requires you to create a job command file that passes the appropriate arguments to the runjob command by adding the runjob call after the #@queue keyword in the job file.
Number of tasks per compute node
Passes "prg_arg" to the launched application on the compute node. For two or more arguments you need to separate the arguments with blanks, for arguments with options specify --args several times to assign the values correctly.
For example: (see also man runjob)
runjob --ranks-per-node 32 --args hello world --exe a.out
runjob --ranks-per-node 32 --args --one foo --args --two bar --exe a.out
Specifies the full path to the executable to be launched on the compute nodes. The path must be specified as seen by the I/O and the compute nodes.
--envs <ENV_Var1=Value1>[ <ENV_Var2=Value2> ...]
Sets the environment variable ENV_Var1=Value1in the environment of the job on the compute nodes. Multiple variables can be added as a space-separated list.
Sets the environment variable ENV_Var in the current environment of runjob to the job on the compute nodes.
Number of total MPI-tasks
You can also call runjob in the following format:
runjob [options] : exe [arg1 arg2 .. argn].
This way you can especially omit several calls of --args,e.g. the examples for --arg above can also be written as:
runjob --ranks-per-node 32 : a.out hello world
runjob --ranks-per-node 32 : a.out --one foo --two bar
For more details and further options please see runjob -h or man runjob.
A complete explanation of runjob can be found in the BG/Q system Administration manual.
Example for a hybrid job using 32 nodes with 1 task per node using 64 threads, for the threads the environment variable OMP_NUM_THREADS is used and exported to the job.
Note that the number of nodes are already specified via bg_size, it is not necessary to additionally specify --np if the full node number is used.
# @ job_name = Example
# @ comment = "Example Job"
# @ error = $(job_name).$(jobid).out
# @ output = $(job_name).$(jobid).out
# @ environment = COPY_ALL
# @ wall_clock_limit = 00:30:00
# @ notification = error
# @ notify_user = firstname.lastname@example.org
# @ job_type = bluegene
# @ bg_size = 32
# @ queue
runjob --ranks-per-node 1 --exp-env OMP_NUM_THREADS --exe ./example_exe
Further examples can be found here.