High throughput simulations of two-phase flows on Blue Gene/Q
Panagiotis Hadjidoukas, ETH Zürich
CUBISM-MPCF is a high throughput software for two-phase flow simulations that has demonstrated unprecedented performance in terms of floating point operations, memory traffic and storage. The software has been optimized to take advantage of the features of the IBM Blue Gene/Q (BGQ) platform to simulate cavitation collapse dynamics using up to 13 Trillion computational elements. The performance of the software has been shown to reach an unprecedented 14.4 PFLOP/s on 1.6 million cores corresponding to 72% of the peak on the 20 PFLOP/s Sequoia supercomputer. It is important to note that, to the best of our knowledge, no flow simulations have ever been reported exceeding 1 Trillion elements or reaching more than 1 PFLOP/s or more than 15% of peak. The code is written in C++ and integrates accurate numerical methods with extensive data reordering and use of vector intrinsics under a hybrid MPI+OpenMP parallelization scheme. In this work, we first summarize the software optimization techniques that allowed us to reach 72% of the theoretical peak performance on BGQ systems. We evaluate the performance of all the three layers (core, node and cluster) of the software by running experiments on single core, single node and multi-rack BGQ configurations. Finally, we show how this performance has been achieved, both in terms of FLOPS and execution time of the simulation, and discuss the relation between these two metrics and possible venues for further improvements.