Navigation and service

JUWELS - Configuration

Hardware Configuration of the JUWELS Cluster Module

  • 2271 standard compute nodes

    • 2× Intel Xeon Platinum 8168 CPU, 2× 24 cores, 2.7 GHz
    • 96 (12× 8) GB DDR4, 2666 MHz
    • InfiniBand EDR (Connect-X4)
    • Intel Hyperthreading Technology (Simultaneous Multithreading)
    • diskless
  • 240 large memory compute nodes

    • Intel Xeon Platinum 8168 CPU, 2× 24 cores, 2.7 GHz
    • 192 (12× 16) GB DDR4, 2666 MHz
    • InfiniBand EDR (Connect-X4)
    • Intel Hyperthreading Technology (Simultaneous Multithreading)
    • diskless
  • 56 accelerated compute nodes

    • 2× Intel Xeon Gold 6148 CPU, 2× 20 cores, 2.4 GHz
    • 192 (12× 16) GB DDR4, 2666MHz
    • 2× InfiniBand EDR (Connect-X4)
    • Intel Hyperthreading Technology (Simultaneous Multithreading)
    • 4× NVIDIA V100 GPU, 16 GB HBM
    • diskless
  • 12 login nodes

    • 2× Intel Xeon Gold 6148 CPU, 2× 20 cores, 2.4 GHz
    • 12× 64 GB DDR4, 2666MHz
    • InfiniBand EDR (Connect-X5)
    • Intel Hyperthreading Technology (Simultaneous Multithreading)
    • 100 GigE
    • 2× 1TB HDD (RAID 1)
  • 4 visualisation nodes

    • 2× Intel Xeon Gold 6148 CPU, 2× 20 cores, 2,4 GHz
    • 768 (12× 64) GB DDR4, 2666 MHz
    • InfiniBand EDR (Connect-X5)
    • Intel Hyperthreading Technology (Simultaneous Multithreading)
    • 100 GigE
    • 2x 1TB HDD (RAID 1)
    • 1× NVIDIA Pascal P100
  • 122,768 CPU cores
  • 10.6 (CPU) + 1.7 (GPU) Petaflop per second peak performance
  • Mellanox InfiniBand EDR fat-tree network with 2:1 pruning at leaf level and top-level HDR switches

    • 40 Tb/s connection to Booster
  • 250 GB/s network connection to JUST for storage access

Hardware Configuration of the JUWELS Booster Module

  • 936 compute nodes

    • 2× AMD EPYC Rome 7402 CPU, 2× 24 cores, 2.7 GHz
    • 512 GB DDR4, 3200 MHz
    • 4× NVIDIA A100 GPU, 4× 40 GB HBM2e
    • 4× InfiniBand HDR (Connect-X6)
    • diskless
  • 4 login nodes

    • 2x 24 cores, 2.7 GHz
    • 12x 16 GB, 2666 MHz
    • EDR-Infiniband (Connect-X4)
    • Intel Hyperthreading Technology (Simultaneous Multithreading)
    • diskless
  • 3,744 GPUs
  • 73 Petaflop per second peak performance
  • Mellanox InfiniBand HDR DragonFly+ topology with 20 cells

    • 40 Tb/s connection to Cluster
  • 350 GB/s network connection to JUST for storage access (work in progress as of November 2020)

Software Overview

  • CentOS 8 Linux distribution
  • ParaStation Modulo
  • Slurm batch system with ParaStation resource management
  • Cluster software stack

    • Intel Professional Fortran, C/C++ Compiler

      • Support for OpenMP programming model for intra-node parallelization
    • Intel Math Kernel Library
    • ParTec ParaStation MPI (Message Passing Interface) implementation
    • Intel MPI
    • Open MPI
  • Booster software stack

    • CUDA-aware ParTec ParaStation MPI
    • Open MPI
    • NVIDIA HPC SDK
  • IBM Spectrum Scale (GPFS) parallel file system