Search

link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service


IBM Cluster for GPFS Fileserver

JUST Cluster (Phase3)Copyright: FZ Jülich

JUST3 DS5300 cabling at rearCopyright: FZ Jülich

JUST4 DS3512 with SAS-DisksCopyright: FZ Jülich

JUST4 DCS3700 with NL-SAS DisksCopyright: FZ Jülich

JUST Cluster (Phase1)Copyright: FZ Jülich

The configuration of the Juelich Storage Cluster (JUST) is more or less continuously under movement to new available storage technology respectively in expansion to fullfill the demands of the supercomputer applications related to capacity and I/O bandwidth. At the time the 5th generation of JUST consists of 22 DSS systems (Lenovo Distributed Storage Solution) and 3 older GSS systems (GPFS Storage Server). All these systems are based on the Spectrum Scale (GPFS) from IBM and provide in total a gross capacity more then 80 PB.

For details see

JUST Numbers

JUST-DSSJUST-TSMJUSTCOMJUST Total
Capacity

75 PB gross,

ca. 50 PB net

4.2 PB gross,
2.9 PB net
2.1 PB gross,
1.5 PB net
81.3 PB gross,
54.4 PB net
Racks162119
Server44 + 5 Mngt + 2 CES4 + 1 Mngt2 + 1 Mngt59
Disk Enclosures90126108
Disks (*)7516 + 44 SSD696 + 4 SSD348 + 2 SSD8610

(*) The failure rate for the disks of the complete system is 2-3 disks per week.

JUST Architecture

JUST architecture - physical view

JUST architecture - logical view


JUST Hardware Characteristics

 

JUST: Distributed Storage Solution
  • 1 x DSS-G 26 (10 TB)

    • each 2 x Lenovo x3650 M5 Systems (x-Series)

      • each 2 x Intel Xeon Processors E5-2690, 14 cores, 2.66 GHz, 384 GB Memory
      • each 3 x Quad-Port SAS 12Gb HBA
      • each 2 x Mellanox Dual-Port SFP + 100 Gigabit Ethernet Adapter
      • Software: RedHat 7.4, DSSG-2.0 (GPFS 5.0.X)
    • each 6 x DSS-Storage (JBODs)

      • each 2 Drawers with 42 slots
      • each 84 x 10 TB NL-SAS Disks (GPFS Native RAID)
      • 1 DSS-Storage with 2 x 400 GB SSD (GPFS-GNR Configuration and Logging)
    • each 502 NL-SAS Disks and 2 SSDs
    • each 5 PB gross, 3.6 PB net (8+3P)
    • JUST User Data and Metadata ($ARCH)
  • 18 x DSS-G 24 (10 TB)

    • each 2 x Lenovo x3650 M5 Systems (x-Series)

      • each 2 x Intel Xeon Processors E5-2690, 14 cores, 2.66 GHz, 384 GB Memory
      • each 3 x Quad-Port SAS 12Gb HBA
      • each 2 x Mellanox Dual-Port SFP + 100 Gigabit Ethernet Adapter
      • Software: RedHat 7.4, DSSG-2.0 (GPFS 5.0.X)
    • each 4 x DSS-Storage (JBODs)

      • each 2 Drawers with 42 slots
      • each 84 x 10 TB NL-SAS Disks (GPFS Native RAID)
      • 1 DSS-Storage with 2 x 400 GB SSD (GPFS-GNR Configuration and Logging)
    • each 334 NL-SAS Disks and 2 SSDs
    • each 3.3 PB gross, 2.4 PB net (8+3P)
    • JUST User Data and Metadata ($WORK and $DATA)
  • 3 x DSS-G 24 (10TB)

    • each 2 x Lenovo x3650 M5 Systems (x-Series)

      • each 2 x Intel Xeon Processors E5-2690, 14 cores, 2.66 GHz, 384 GB Memory
      • each 3 x Quad-Port SAS 12Gb HBA
      • each 2 x Mellanox Dual-Port SFP + 100 Gigabit Ethernet Adapter
      • Software: RedHat 7.4, DSSG-2.0 (GPFS 5.0.X)
    • each 4 x GSS-Storage (JBODs)

      • each 2 Drawers with 42 slots
      • each 84 x 10 TB NL-SAS Disks (GPFS Native RAID)
      • 1 GSS-Storage with 2 x 200 GB SSD (GPFS-GNR Configuration and Logging)
    • each 334 NL-SAS Disks and 2 SSDs
    • each 3.3 PB gross, 2.4 PB net (8+3P)
    • JUST User Data and Metadata ($HOME)
  • 2 x Lenovo GSS-26 (6 TB)

    • each 2 x IBM x3650 M4 HD Systems (x-Series)

      • each 2 x Intel Xeon Processors E5-2670, 10 cores, 2.5 GHz, 256 GB Memory
      • each 3 x LSI 9201-16e Quad-Port SAS 6Gb HBA (12x)
      • each 3 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter (6x)
      • Software: RedHat 6.5, GSS 2.0 (GPFS 4.1.x)
    • each 6 x GSS-Storage (JBODs)

      • each 5 Drawers with 12 slots
      • each 58 x 6 TB NL-SAS Disks (RAID6 (8+2))
      • 1 DSS-Storage with 2 x 400 GB SSD
    • each 348 NL-SAS Disks and 2 SSDs (GSS-Configuration Backup)
    • each 2088 TB gross, 1668 TB net (RAID6)
    • TSM and User Communitay Data (Dep. FSD)
  • 1 x IBM ESS GL6 (6 TB)

    • each 2 x IBM Power Server (8247-22L)

      • each 2 x 10 core 3.42 GHz POWER8 Processor Card, 256GB Memory
      • each 3 x LSI 9201-16e Quad-Port SAS 6Gb HBA (12x)
      • each 3 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter (6x)
      • Software: RedHat 6.5, GSS 2.0 (GPFS 4.1.x)
    • each 348 NL-SAS Disks and 2 SSDs (GSS-Configuration Backup)
    • each 2088 TB gross, 1668 TB net (RAID6)
    • TSM
JUST: Server
  • 2 x Management Server (ThinkSystem SR650)

    • each 2 x Intel Skylake Processors Gold 6142, 16 cores, 2.6 GHz, 384 GB Memory
    • each 2 x Mellanox ConnectX-4 Dual-Port 100 Gigabit Ethernet Adapter
    • Software: RedHat 7.4, xCAT 2.13
  • 1 x Monitoring

    • each 2 x Intel Skylake Processors Gold 6142, 16 cores, 2.6 GHz, 384 GB Memory
    • each 2 x Mellanox ConnectX-4 Dual-Port 100 Gigabit Ethernet Adapter
    • Software: RedHat 7.4, Check_MK 1.4.0
  • 5 x GPFS Management Server (ThinkSystem SR650)

    • each 2 x Intel Skylake Processors Gold 6142, 16 cores, 2.6 GHz, 384 GB Memory
    • each 2 x Mellanox ConnectX-4 Dual-Port 100 Gigabit Ethernet Adapter
    • Software: RedHat 7.4, GPFS 5.0.1
  • 2 x GPFS-CES (Cluster Export Service) (IBM Power6 520 System)

    • each 2 x Intel Skylake Processors Gold 6142, 16 cores, 2.6 GHz, 384 GB Memory
    • each 2 x Mellanox ConnectX-4 Dual-Port 100 Gigabit Ethernet Adapter
    • Software: RedHat 7.4, GPFS 5.0.1, NFS (Ganesha)
JUST-TSM: Server and Storage
  • 8 x TSM-Server (IBM Power System S822)

    • each 2 x Power8 Processor, 10 cores, 3.42 GHz, 256 GB Memory
    • each 4 x 16Gbps Dual-Port FC Adapter
    • each 2 x Mellanox Dual-Port 100 Gigabit Ethernet Adapter
    • Software: AIX 7.2, Spectrum Protect (TSM) Server + Client, Spectrum Scale (GPFS)
  • 1 x NIM Server (IBM Power Szstem S821)

    • 1 x Power8 Processor, 4 core, 3.0 GHz, 32 GB Memory
    • 1 x Quad-Port 10/1 Gigabit Ethernet Adapter
    • Software: AIX 7.1, NIM, TSM Client
  • 1 x Hardware Management Console (IBM Power System 7)

    • Power Systems Management
    • Software: Linux, HMC 8.7.0

JUST History and Roadmap

JUST-Roadmap (Capacity & Bandwidth)

In 2007 JUST started with classical storage building blocks consisting of IBM Power5 servers running AIX and storage controllers with FC and SATA disks like IBM DS4800, DS4700, and DCS9550 and 1 PB capacity gross serving 6-7 GB/s bandwidth.

The next milestones were in 2009 starting in March with the replacement of the servers by Power6 systems and in December followed by migration to new generation of storage controllers and disks with IBM DS5300. The capacity grew to 5 PB gross and the bandwidth was about 33 GB/s.

In 2012 additional IBM x-Series servers running Linux and IBM DS3512 and DCS3700 storage controllers with SAS and NL-SAS disks were installed and all data beside the fast scratch file system were migrated to the new technology. The free Power6 servers and storage were added to the scratch file system bringing up the bandwith to 66 GB/s and increasing overall capacity to 10 PB.

In January 2013 the installation and test of about 9 PB gross GSS-24 systems running the limited available GSS 1.0 version started. Mid September 2013 a new generally available fast scratch file system ($WORK) was introduced. At the same time a new special file system ($DATA) dedicated to large projects with big data in collaboration with JSC was built, where disk space quota is available on application at JSC management only. The overall JUST storage capacity was 13 PB and a bandwidth of 160 GB/s could be achieved.

In June 2014 additional 2.8 PB (gross) GSS storage were installed and used for migration of the classical $HOME file systems into GNR based file systems. The JUST storage capacity grows to about 16 PB (gross).
In December 2014 it was decided to transfer the remaining classical storage components to GSS-24 systems by reusing the storage infrastucture combined with new x-Series servers. This was done step by step and finished in March 2015. At the end free storage was added to $WORK and $DATA which increases the bandwidth to about 200 GB/s. At that time JUST consists of 31 GPFS Storage Server systems (GSS) with a capacity of 16 PB gross.

In June 2015 a global I/O reconfiguration took place to support the new HPC-system JURECA. At all storage servers the 2 times 30 GB ethernet channels were spitted into 3 times 20 GB ethernet channels which were distributed over three I/O switches. This implies also recabling and strong checking of the layout for redundancy at all levels. Mid 2015 additional 4 PB (gross) were installed by two capacity bound GSS-26 storage servers. They were partially used for migration of the HPC archive file systems ($ARCH). The thereby freed storage was added to the fast scratch file system $WORK and the project dedicated file system $DATA which increased their capacity by 25% and the I/O bandwidth to 220 GB/s .The overall capacity is 20 PB gross.

In April 2018 the 5th generation of JUST started the production. The old GSS hardware was replaced by new Lenovo Distributed Storage Solution (DSS). The software setup is the same as in JUST4: The parallel file system is based on Spectrum Scale (GPFS) in combination with the GPFS Native RAID (GNR) technology from IBM. 75 PB gross capacity is provided by this new installation.


Servicemeu

Homepage