JUQUEEN - Configuration
28 racks (7 rows à 4 racks) - 28,672 nodes (458,752 cores)
Rack: 2 midplanes à 16 nodeboards (16,384 cores)
Nodeboard: 32 compute nodes
Node: 16 cores
- Main memory: 448 TB
- Overall peak performance: 5.9 Petaflops
- Linpack: 5.0 Petaflops
- I/O Nodes: 248 (27x8 + 1x32) connected to 2 CISCO Switches
IBM BlueGene/Q Specifications
IBM PowerPC® A2, 1.6 GHz, 16 cores per node
16 GB SDRAM-DDR3 per node
5D Torus — 40 GBps; 2.5 μsec latency (worst case)
Collective network — part of the 5D Torus; collective logic operations supported
Global Barrier/Interrupt — part of 5D Torus, PCIe x8 Gen2 based I/O
1 GB Control Network — System Boot, Debug, Monitoring
I/O Nodes (10 GbE)
16-way SMP processor; configurable in 8,16 or 32 I/O nodes per rack
Compute nodes: CNK, lightweight proprietary kernel
- BG/Q Power
Direct-current voltage (48V to 5V, 3.3V, 1.8V on node boards)
60 - 70 kW per rack (average)
- JSC power cables and usage
app 4380 m power cables (63A/400V), app. 7900 kg
app. 1.7 MW in 2012 (compares to app. 4000 households)
90% water cooling (18-25°C,demineralized,closed circle); 10% air cooling
Temperature: in: 18°C, out: 27°C
- JSC, Cable Trays
48 m, 1900 kg
- Square Footage
83 m2 (+ 43 m2 JUST Fileserver)
IBM BlueGene/Q Characteristics
- Integrated 5D torus provides tremendous bisection bandwidth.
- Quad floating point unit (FPU) for 4-wide double precision FPU SIMD and 2-wide complex SIMD.
- “Perfect” prefetching for repeated memory reference patterns in arbitrarily long code segments.
- Multiversioning cache with transactional memory eliminates the need for locks.
- Speculative execution allows OpenMPthreading with data dependencies.
- Atomic operations, pipelined at L2 with low latency even under high contention, provide a faster handoff for OpenMP work.
- A wake-up unit allows SMT threads to sleep while waiting for an event and avoids register-saving overhead.
- A 17th core manages OS related tasks thus reducing OS related noise.
- 2 CISCO Nexus 7018 Switches
- Total ports: 480
2 service nodes and 2 login nodes (IBM Power 740):
- Total number of processors: 16
- Processor type: Power7, 8C, 3.55 GHz
- Total amount of memory: 128 GB
- Operating system: RedHat Linux V6.2
- Internet address of login node: juqueen.fz-juelich.de
- May 2012 - Jan 2013
Position in Top 500
- June 2012: 8 (8 racks)
- Nov. 2012: 5 (24 racks)
- June 2013: 7 (Europe: 1)
- Nov. 2013: 8 (Europe: 2)
- June 2014: 8 (Europe: 2)
- Nov. 2014: 8 (Europe: 2)