JuZEA-1

Modern numerical simulations in the development of scientific instruments and processes

In the development of new scientific instruments and processes, the limits of what is feasible are often reached. This requires a careful design and optimisation of components that are exposed to extreme loads during subsequent operation and processes that are decisive for the success of the subsequent scientific experiment. Modern simulation methods can be used to evaluate concepts in advance, determine important influencing factors and detect development errors or weak points at an early stage. The more detailed the simulation models are, the more accurate the results that the experts at ZEA-1 receive from the calculations. However, suitable computer resources are required to calculate large simulation models. In this context, it is important to have sufficient main memory to be able to start the simulations in the first place and to prevent time-consuming swapping of data to hard disks, but a large number of quickly networked processors is also crucial for the level of detail of the simulation models that can be calculated within an acceptable period of time.

JuZEA-1 - Neues HPC-Cluster für aufwändige numerische Simulationen

Continuing the successful cooperation with JSC

Until 2013, calculations at ZEA-1 were carried out exclusively on the Simulation and Computation team's own cluster. At that time, supercomputers at the Jülich Supercomputing Centre (JSC), such as JuQUEEN or JuROPA2, still had two decisive disadvantages for the experts at ZEA-1: firstly, they did not offer the required main memory per computing node for complex simulations and, secondly, the available runtimes for individual calculations were too limited, so that some calculations were cancelled without results after reaching the maximum runtime. The supercomputers were therefore not suitable for the scenarios of the ZEA-1 with the existing simulation programmes.

As the expansion of the in-house cluster was not an option for ZEA-1 due to the space and cooling requirements and the comparatively high support costs, the joint acquisition and operation of a customised cluster for ZEA-1 was planned in 2013 in cooperation with the JSC. At this time, the JSC configured a flexible test cluster called JuROPA3, in which the ZEA-1 could participate with its special hardware requirements. The co-operation with the JSC was very successful for both sides and the needs and requirements of the ZEA-1 were met to the fullest satisfaction. The JSC was also able to open up new applications through the co-operation. Therefore, before the JuROPA3 expired at the end of 2018, it was decided to continue the collaboration and the acquisition of a new cluster for the ZEA-1 was initiated. After a short test phase, the new JuZEA-1 system was put into operation by the JSC in December 2018 and the JuROPA3 was shut down. The JuZEA-1 now provides the ZEA-1 computational engineers with a system that has around twice as many computing cores and twice as much main memory as its predecessor, significantly increasing the team's productivity.

Calculation model for the GLORIA instrument - illustrates deformations due to an excited oscillation

JuZEA-1 - Neues HPC-Cluster für aufwändige numerische Simulationen
Simulation model for the GLORIA instrument - deformations due to an excited vibration are visualised.
FZJ

The illustration shows a simulation of the GLORIA instrument for atmospheric research, which is used on the HALO research aircraft, among others, and is exposed to considerable vibrations during flight. In order to prove sufficient strength, in particular with regard to the formation of cracks due to the vibrations of the instrument, it is necessary to have precise knowledge of the natural frequencies, the vibration amplitudes caused by the excitations during the flight and the resulting stress fluctuations in the component. The calculations for the complex overall model were only possible to a limited extent on the old cluster due to the insufficient main memory and took several days due to the constant swapping of data to the hard drives. On the JuZEA-1, on the other hand, the vibration calculations could be carried out in just a few hours.

Technical data of the JuZEA-1

10 nodes with 48 cores each
100 Gbit/s InfiniBand network
4608 GByte main memory
43.2 TByte internal discs for scratch data
Operating system: CentOS Linux 7.4
Cluster control: Parastation middleware with batch system SLURM
Installed simulation software: ANSYS, CFX, Fluent, LS-DYNA
Separate front-end and service nodes

Last Modified: 09.12.2024