Search

link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service


IBM Cluster for GPFS Fileserver

JUST Cluster (Phase3)Copyright: FZ Jülich

JUST3 DS5300 cabling at rearCopyright: FZ Jülich

JUST4 DS3512 with SAS-DisksCopyright: FZ Jülich

JUST4 DCS3700 with NL-SAS DisksCopyright: FZ Jülich

JUST Cluster (Phase1)Copyright: FZ Jülich

The configuration of the Juelich Storage Cluster (JUST) is more or less continuously under movement to new available storage technology respectively in expansion to fullfill the demands of the supercomputer applications related to capacity and I/O bandwidth. For details see the JUST History and JUST News.

At the time the 4th generation of JUST consists of 31 GPFS Storage Server systems (GSS) with a capacity of 16 PB gross including the TSM backup environment.

JUST Numbers

JUST-TSMJUST-GSSJUST Total
Capacity

450 TB gross,

320 TB net

15.7 PB gross,
12.6 PB net
16.2 PB gross,
13 PB net
Racks31720
Server8 + 2 Mngt62 + 8 Mngt80
Storage Controller404
Disk Enclosures / Drawers28620648
Disks (*)3367192 + 142 SSD7670

(*) The failure rate for the disks of the complete system is 2-3 disks per week.

JUST Architecture

JUST architecture - physical view

JUST architecture - logical view


JUST Hardware Characteristics

 

JUST: GPFS Storage Sever
  • 20 x IBM GSS-24

    • each 2 x IBM x3650 M4 Systems (x-Series)

      • each 2 x Intel Xeon Processors E5-2620, 8 cores, 2.6 GHz, 64 GB Memory
      • each 3 x LSI 9201-16e Quad-Port SAS 6Gb HBA (12x)
      • each 3 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter (6x)
      • Software: RedHat 6.5, XCAT, GSS 2.0 (GPFS 4.1.0.7)
    • each 4 x GSS-Storage (JBODs)

      • each 5 Drawers with 12 slots
      • each 58 x 2 TB NL-SAS Disks (RAID6 (8+2))
      • 3 GSS-Storage with additional 2 x 200 GB SSD
    • each 232 NL-SAS Disks
    • each 6 SSDs (GSS-Configuration and Logging)
    • each 464 TB gross, 371 TB net (RAID6)
    • JUST User Data and Metadata ($WORK and $DATA)
  • 3 x IBM GSS-24

    • each 2 x IBM x3650 M4 Systems (x-Series)

      • each 2 x Intel Xeon Processors E5-2620, 8 cores, 2.6 GHz, 128 GB Memory
      • each 3 x LSI 9201-16e Quad-Port SAS 6Gb HBA (12x)
      • each 3 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter (6x)
      • Software: RedHat 6.4, XCAT, GSS 1.5 (GPFS 3.5.0.17)
    • each 4 x GSS-Storage (JBODs)

      • each 5 Drawers with 12 slots
      • each 58 x 4 TB NL-SAS Disks (RAID6 (8+2))
      • 1 GSS-Storage with additional 2 x 200 GB SSD
    • each 232 NL-SAS Disks
    • each 2 SSDs (GSS-Configuration Backup)
    • each 928 TB gross, 742 TB net (RAID6)
    • JUST User Data and Metadata ($HOME)
  • 8 x IBM GSS-24

    • each 2 x IBM x3650 M4 HD Systems (x-Series)

      • each 2 x Intel Xeon Processors E5-2670, 10 cores, 2.5 GHz, 128 GB Memory
      • each 3 x LSI 9206-16e Quad-Port SAS 6Gb HBA (12x)
      • each 3 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter (6x)
      • Software: RedHat 6.5, XCAT, GSS 2.0 (GPFS 4.1.0.7)
    • each 4 x GSS-Storage (JBODs,old 1818-80E/C)

      • each 5 Drawers with 12 slots
      • each 58 x 2 TB NL-SAS Disks (RAID6 (8+2))
      • 1 GSS-Storage with additional 2 x 200 GB SSD
    • each 232 NL-SAS Disks
    • each 2 SSDs (GSS-Configuration Backup)
    • each 464 TB gross, 371 TB net (RAID6)
    • JUST User Data and Metadata ($ARCH, $WORK,$DATA)
JUST: Server
  • 2 x Management Server (IBM x3650 M3 Systems)

    • each 2 x Intel Xeon Processors E5640, 4 cores, 2.66 GHz, 24 GB Memory
    • each 2 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter
    • Software: RedHat 6.6, XCAT, GPFS Client, TSM Client, LDAP Client
  • 2 x Monitoring and GSS Administration Server (IBM x3550 M4 Systems)

    • each 2 x Intel Xeon Prozessoren E5-2670, 8 cores, 2.6 GHz, 32 GB Memory
    • each 1 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter
    • Software: RedHat 6.5, XCAT, Icinga, TSM Client
  • 4 x GPFS Management Server (IBM x3650 M3 Systems)

    • each 2 x Intel Xeon Processors E5640, 4 cores, 2.66 GHz, 24 GB Memory
    • each 2 x 8Gbps Dual-Port FC Adapter
    • each 2 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter
    • Software: RedHat 6.1, XCAT, GPFS
  • 1 x GPFS-NFS Gateway (IBM Power6 520 System)

    • 2 x Power6 Processors, 4.2 GHz, 8 GB Memory
    • 1 x 10 Gigabit Ethernet-SR PCI-X 2.0 DDR Adapter (virtualized)
    • Software: AIX 6.1.7, GPFS, TSM Client
    • Brain Project, INM1
JUST-TSM: Server and Storage
  • 8 x TSM-Server (IBM Power 720 Systems)

    • each 1 x Power7 Processor, 8 cores, 3.0 GHz, 64 GB Memory
    • each 5 x 8Gbps Dual-Port FC Adapter
    • each 2 x Dual-Port 10 Gigabit Ethernet Adapte
    • Software: AIX 6.1.7, TSM Server+ Client, GPFS Client, LDAP Client
  • 1 x NIM Server (IBM Power 710 System)

    • 1 x Power7 Processor, 4 core, 3.0 GHz, 32 GB Memory
    • 2 x Dual-Port 1 Gigabit Ethernet Adapter
    • Software: AIX 7.1, NIM, TSM Client
  • 1 x Hardware Management Console (IBM System x3550 M3)

    • Power Systems Management
    • Software: Linux, HMC 7.4.0
  • 4 x IBM DS3512 Storage Controller

    • each 7 Enclosures (a 12 Disks, Mix of SAS-Disks, 600 GB, 15000 RPM and NL-SAS, 2TB, 7200 RPM)
    • each 37 SAS-Disks (RAID5 (4+1))
    • each 47 NL-SAS-Disks (RAID5 (4+1))
    • each 116.2 TB gross, 79.4 TB net (RAID5, Hotspares)
    • TSM Data (DB, Log, Staging)

JUST History

In 2007 JUST started with classical storage building blocks consisting of IBM Power5 servers running AIX and storage controllers with FC and SATA disks like IBM DS4800, DS4700, and DCS9550 and 1 PB capacity gross serving 6-7 GB/s bandwidth.

The next milestones were in 2009 starting in March with the replacement of the servers by Power6 systems and in December followed by migration to new generation of storage controllers and disks with IBM DS5300. The capacity grew to 5 PB gross and the bandwidth was about 33 GB/s.

In 2012 additional IBM x-Series servers running Linux and IBM DS3512 and DCS3700 storage controllers with SAS and NL-SAS disks were installed and all data beside the fast scratch file system were migrated to the new technology. The free Power6 servers and storage were added to the scratch file system bringing up the bandwith to 66 GB/s and increasing overall capacity to 10 PB.

In January 2013 the installation and test of about 9 PB gross GSS-24 systems running the limited available GSS 1.0 version started. Mid September 2013 a new generally available fast scratch file system ($WORK) was introduced. At the same time a new special file system ($DATA) dedicated to large projects with big data in collaboration with JSC was built, where disk space quota is available on application at JSC management only. The overall JUST storage capacity was 13 PB and a bandwidth of 160 GB/s could be achieved.

In June 2014 additional 2.8 PB (gross) GSS storage were installed and used for migration of the classical $HOME file systems into GNR based file systems. The JUST storage capacity grows to about 16 PB (gross).
In December 2014 it was decided to transfer the remaining classical storage components to GSS-24 systems by reusing the storage infrastucture combined with new x-Series servers. This was done step by step and finished in March 2015. At the end free storage was added to $WORK and $DATA which increases the bandwidth to about 200 GB/s. At that time JUST consists of 31 GPFS Storage Server systems (GSS) with a capacity of 16 PB gross.


Servicemeu

Homepage