Direct Numerical Simulation of Fluid Turbulence at Extreme Scale with psOpen
Jens Henrik Göbbert, Jülich Aachen Research Alliance
The hybrid OpenMP/MPI code psOpen has been developed at the Institute for Combustion Technology, RWTH Aachen University, to study incompressible fluid turbulence by means of direct numerical simulations. For efficiency and accuracy psOpen employs a pseudo-spectral method, where a Fourier transform of the governing equations is solved in spectral space. A pseudo-spectral method requires frequently transformations between real and spectral space which is particularly challenging for massively-parallel setups. The frequently invoked Fourier transformation is a non-local operation and requires access to all data along a global grid line for each computational process. psOpen was improved by a new inhouse-developed 3D-FFT library optimised to the special needs of pseudo-spectral DNS. The scaling performance of psOpen with three different grid sizes, namely 40963, 61443 and 81923, has been studied on JUQUEEN up to the full machine. For an optimal use of the hardware resources two techniques have been combined to reduce the communication time significantly. Firstly, the number of operations and the size of data to be transposed while computing a 3D-FFT has been reduced by integrating the dealiasing cut-off filter into the 3D-FFT. Secondly, the new 3D-FFT library allows to overlap communication and computation of multiple FFTs at the same time.