link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service

Quick Introduction

JUQUEEN Usage Model

The IBM Blue Gene/Q uses so called front-end nodes or login nodes running Linux for interactive access and the submission of batch jobs. Parallel applications have to be cross-compiled on the front-end nodes and can only be executed on the partition residing on the BG/Q compute nodes. Access is automatically controlled by the LoadLeveler batch system, which chooses the appropriate partition depending on the requested resources. Serial jobs are executed on the front-end node.

Batch System on JUQUEEN

The batch system on JUQUEEN is LoadLeveler. The batch system is responsible for managing jobs on the machine, returning job output to the user and provides for job control on the user's or system administrator's request.

Job Limits and Conditions

  • Compute nodes are used exclusively by jobs of a single user; no node sharing between jobs is done.
  • The smallest allocation unit is 32 nodes (512 cores).
  • The number of nodes as well as the ranks per node need to be chosen as a power of 2 (1,2,4,8,16,32,...)
  • The maximum number of ranks per node is 64.
  • Users will be charged for the number of compute nodes multiplied with the wall clock time used.
  • Running 1 rank per node, there will be approximately 16 GB (minus 16 MB for the kernel) available to the application.
    Running 'n' ranks per node (n>1), there will be approximately 16GB/n available to the application.

Job SizeResourcesValue
Batch jobsmax. wall clock time 24 h
default wall clock time 6 h
min. number of nodes 257 (allocating 512)
max. number of nodes28672
Medium batch jobsmax. wall clock time12 h
default wall clock time 6 h
min. number of nodes65 (allocating 128)
max. number of nodes256
Small batch jobsmax. wall clock time30 min
default wall clock time30 min
min. number of nodes 32
max. number of nodes64

Access to JUQUEEN

See here for details on how to log on to the system.


See Compiling and Tuning Applications on how to compile Fortran, C or C++ programs.

Write a Batch Script

Write a job script including the runjob command for the just compiled executable.
The minimal template to be filled is:

# @ job_name = LoadL_Sample_1
# @ comment = "BG Job by Size"
# @ error = $(job_name).$(jobid).out
# @ output = $(job_name).$(jobid).out
# @ environment = COPY_ALL
# @ wall_clock_limit = 00:40:00
# @ notification = error
# @ notify_user =
# @ job_type = bluegene
# @ bg_size = 32
# @ queue
runjob --exe myprogp.elf --args "-t 1" --ranks-per-node 8


  • Files for #@ output and #@ error messages have to be defined, here the they are named after the job (script) name and the LoadLeveler job number and combined into one file (same name for both).
  • The keyword #@ environment = COPY_ALL ensures exporting of all these variables defined in the current job script to the processes spawned by the runjob command.
  • A valid email address has to be specified in # @ notify_user.
  • # @ bg_size specifies the number of requested nodes.
  • The user commands start after the #@ queue keyword.
  • runjob starts the parallel executable.

    For runjob options, see the BG/Q Administration Manual, chapter 6.2.1, page 75.

Jobscript Examples

The above and further job script examples (inkl. multi-step jobs for job chaines with dependencies) can be found here or in /bgsys/local/samples/LoadLeveler.

Submit the Job

Use llsubmit to submit the job:

llsubmit <jobscript>

On success llsubmit returns the job ID of the submitted job.

Start an Interactive Session

Not available on JUQUEEN.

Advanced Information

Useful LoadLeveler Commands

llsubmit <jobscript>submit job
llq show status of all jobs
llq -s <jobid>get detailed information about a job
llqx [-l]show status of all jobs incl. priority
llcancel <jobid>cancel a job
llclass[x]show information about job classes
llstatusshow general status of LoadLeveler
llbgstatxshow status of BlueGene midplanes

For further information please see also LoadLeveler Commands.