link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service

Batch Job Processing on JUQUEEN

IBM Tivoli Workload Scheduler LoadLeveler is used as batch system on Blue Gene/Q.

The batch job priority depends on the job size: big jobs have a significant higher priority than smaller jobs.

Using LoadLeveler

Job submission to LoadLeveler is done using a job command file. The job command file is a shell script containing keywords embedded in comments beginning with #@ These keywords inform LoadLeveler of the resources required for the job to run, the program to execute, where to write output files and the job environment.

Jobs are submitted with

llsubmit <jobfile name>

Blue Gene specific keywords:

  • You have to indicate your job as an Blue Gene job with #@ job_type = bluegene. Otherwise the job is executed as a serial job on the login node without allocating a Blue Gene block.

  • The size of a job has to be specified by using #@ bg_size or #@ bg_shape.

    • The bg_size keyword specifies the number of compute nodes the job should use. Blue Gene/Q only allows blocks including 32, 64, 128, 256 and multiples of 512 compute nodes. Thus e.g. bg_size of 1 specifies a block of size 32 and bg_size of 129 requests a partition of size 256.
    • The bg_shape keyword specifies the shape of the block at the midplane level, not at the compute node level. A bg_shape value 1x2x2x2 means 1 midplane in the A direction, 2 in the B direction and 2 in the C direction and 2 in D direction, which are 8 midplanes = 4096 compute nodes. bg_shape defines the logical dimensions of your block. For an efficient scheduling LoadLeveler may allocate physically a permutation and ensures the correct mapping of the MPI-tasks.

      If - and only if - you are using your own mapfile (--mapping option in the runjob command) or your application relies on a correct physical size of the block you have to use the #@ bg_rotate = FALSE keyword together with bg_shape. This indicates LoadLeveler that only the requested shape satisfies the job requirement.

  • The topology of the partition can be specified with the #@ bg_connectivity keyword, which can be one of the values: Torus | Mesh | Either | Xa Xb Xc Xd. The specification for Xy is equal to Torus or Mesh, specified for the y dimension. Default is Mesh.
    This choice can have a big influence on the performance of your application. In case of doubt always add #@ bg_connectivity = Torus to your job script.

For a detailed description of the Blue Gene specific keywords and a table of the core general keywords, see LoadLeveler Keywords.


On a Blue Gene/Q system, the program to execute an application is always the runjob command. In other words, preparing a job for submission requires you to create a job command file that passes the appropriate arguments to the runjob command by adding the runjob call after the #@ queue keyword in the job file.

Since LoadLeveler automatically selects the appropriate block to run the job on, the –-block option should not be specified in the runjob command.

For more details, see: runjob or man runjob or runjob -h .