Search

link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service


JUQUEEN - Configuration

System Configuration

  • 28 racks (7 rows à 4 racks) - 28,672 nodes (458,752 cores)

    Rack: 2 midplanes à 16 nodeboards (16,384 cores)
    Nodeboard: 32 compute nodes
    Node: 16 cores

  • Main memory: 448 TB
  • Overall peak performance: 5.9 Petaflops
  • Linpack: 5.0 Petaflops
  • I/O Nodes: 248 (27x8 + 1x32) connected to 2 CISCO Switches

IBM BlueGene/Q Specifications

  • Compute Card/Processor

    IBM PowerPC® A2, 1.6 GHz, 16 cores per node

  • Memory

    16 GB SDRAM-DDR3 per node

  • Networks
    5D Torus — 40 GBps; 2.5 μsec latency (worst case)
    Collective network — part of the 5D Torus; collective logic operations supported
    Global Barrier/Interrupt — part of 5D Torus, PCIe x8 Gen2 based I/O
    1 GB Control Network — System Boot, Debug, Monitoring
  • I/O Nodes (10 GbE)

    16-way SMP processor; configurable in 8,16 or 32 I/O nodes per rack

  • Operating system

    Compute nodes: CNK, lightweight proprietary kernel

Miscellaneous

  • BG/Q Power
    Direct-current voltage (48V to 5V, 3.3V, 1.8V on node boards)
    10 - 80 kW per rack (estimated); maximum 100 kW per rack
  • JSC power cables and usage
    app 4380 m power cables (63A/400V), app. 7900 kg
    1.9 MW in 2012 (compares to 5000 households)
  • Cooling
    90% water cooling (18-25°C,demineralized,closed circle); 10% air cooling
    Temperature: in: 18°C, out: 27°C
  • JSC, Cable Trays
    48 m, 1900 kg
  • Square Footage
    83 m2 (+ 43 m2 JUST Fileserver)

IBM BlueGene/Q Characteristics

  • Integrated 5D torus provides tremendous bisection bandwidth.
  • Quad floating point unit (FPU) for 4-wide double precision FPU SIMD and 2-wide complex SIMD.
  • “Perfect” prefetching for repeated memory reference patterns in arbitrarily long code segments.
  • Multiversioning cache with transactional memory eliminates the need for locks.
  • Speculative execution allows OpenMPthreading with data dependencies.
  • Atomic operations, pipelined at L2 with low latency even under high contention, provide a faster handoff for OpenMP work.
  • A wake-up unit allows SMT threads to sleep while waiting for an event and avoids register-saving overhead.
  • A 17th core manages OS related tasks thus reducing OS related noise.

Network Switches

  • 2 CISCO Nexus 7018 Switches
  • Total ports: 480

Infrastructure Nodes

  • 2 service nodes and 2 login nodes (IBM Power 740):

    • Total number of processors: 16
    • Processor type: Power7, 8C, 3.55 GHz
    • Total amount of memory: 128 GB
    • Operating system: RedHat Linux V6.2
    • Internet address of login node: juqueen.fz-juelich.de

Installation Phase:

  • May 2013 - Jan 2013

Position in Top 500

  • June 2012: 8 (8 racks)
  • Nov. 2012: 5 (24 racks) (Europe: 1)
  • June 2013: 7 (28 racks) (Europe: 1)
  • Nov. 2013: 8 (28 racks) (Europe: 2)

Servicemeu

Homepage