Search

link to homepage

Institute for Advanced Simulation (IAS)

Navigation and service


IBM Cluster for GPFS Fileserver

JUST Cluster (Phase3)Copyright: FZ Jülich

JUST3 DS5300 cabling at rearCopyright: FZ Jülich

JUST4 DS3512 with SAS-DisksCopyright: FZ Jülich

JUST4 DCS3700 with NL-SAS DisksCopyright: FZ Jülich

JUST Cluster (Phase1)Copyright: FZ Jülich


The Juelich Storage Cluster (JUST) consists of two parts due to different stages of expansion.

  • One part called JUST4 are the storage systems installed in January 2012 with about 4.4 PB gross that are used to store user home data ($HOME) and user archive data ($ARCH).
  • The second part JUST4-GSS is an expansion of the JUST4 part by new GPFS Storage Server systems (GSS) with a capacity of 12 PB gross . The installation of the first 9.2 PB and test of the limited available version started in January 2013. Mid September 2013 a new generally available fast scratch file system ($WORK) was introduced. At the same time a new special file system ($DATA) dedicated to large projects with big data in collaboration with JSC was built, where disk space quota is available on application at JSC management only.
    In June 2014 additional 2.8 PB (gross) GSS storage were installed for future migration of the classical $HOME file systems into GNR based file systems.

JUST Numbers

JUST4 JUST4-GSSJUST Total
Capacity4.4 PB gross,
3.4 PB net
12 PB gross,
9.6 PB net
16.4 PB gross,
13 PB net
Racks81220
Server344680
Storage Controller22022
Disk Enclosures / Drawers204412616
Disks24485336 + 126 SSD7910

The failure rate for the disks of the complete system is 2-3 disks per week.

JUST Architecture

JUST architecture - physical view

JUST architecture - logical view


JUST Hardware Characteristics

JUST4: Server
  • 4 x GPFS NSD-Metadata Server (IBM x3650 M3 Systems)

    • each 2 x Intel Xeon Processors E5640, 4 cores, 2.66 GHz, 24 GB Memory
    • each 2 x 8Gbps Dual-Port FC Adapter
    • each 2 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter
    • Software: RedHat 6.1, XCAT, GPFS
  • 16 x GPFS NSD-Data Server (IBM x3650 M3 Systems)

    • each 2 x Intel Xeon Processors E5640, 4 cores, 2.66 GHz, 24 GB Memory
    • each 2 x 8Gbps Dual-Port FC Adapter
    • each 2 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter
    • Software: RedHat 6.1, XCAT, GPFS
  • 8 x TSM-Server (IBM Power 720 Systems)

    • each 1 x Power7 Processor, 8 cores, 3.0 GHz, 64 GB Memory
    • each 5 x 8Gbps Dual-Port FC Adapter
    • each 2 x Dual-Port 10 Gigabit Ethernet Adapter
    • Software: AIX 6.1.7, TSM Server+ Client, GPFS Client, LDAP Client
  • 2 x Management Server (IBM x3650 M3 Systems)

    • each 2 x Intel Xeon Processors E5640, 4 cores, 2.66 GHz, 24 GB Memory
    • each 2 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter
    • Software: RedHat 6.4, XCAT, GPFS Client, TSM Client, LDAP Client
  • 2 x Monitoring and GSS Administration Server (IBM x3550 M4 Systems)

    • each 2 x Intel Xeon Prozessoren E5-2670, 8 cores, 2.6 GHz, 32 GB Memory
    • each 1 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter
    • Software: RedHat 6.4, XCAT, Icinga, TSM Client
  • 1 x NIM Server (IBM Power 710 System)

    • 1 x Power7 Processor, 4 core, 3.0 GHz, 32 GB Memory
    • 2 x Dual-Port 1 Gigabit Ethernet Adapter
    • Software: AIX 6.1.7, NIM, TSM Client
  • 1 x GPFS-NFS Gateway (IBM Power6 520 System)

    • 2 x Power6 Processors, 4.2 GHz, 8 GB Memory
    • 1 x 10 Gigabit Ethernet-SR PCI-X 2.0 DDR Adapter (virtualized)
    • Software: AIX 6.1.7, GPFS, TSM Client
    • Brain Project, INM1
  • 1 x Hardware Management Console (IBM System x3550 M3)

    • Power Systems Management
    • Software: Linux, HMC 7.4.0
JUST4: Storage
  • 2 x IBM DS3512

    • each 8 Enclosures (a 12 SAS-Disks, 600 GB, 15000 RPM)
    • each 96 SAS-Disks (RAID1)
    • each 57.6 TB gross, 28 TB net (RAID1, no Hotspares)
    • JUST Metadata ($HOME and $ARCH)
  • 16 x IBM DCS3700

    • each 10 Drawers (a 12 NL-SAS-Disks, 2TB, 7200 RPM)
    • each 120 NL-SAS-Disks (RAID6 (8+2))
    • each 240 TB gross, 192 TB net (RAID6, no Hotspares)
    • JUST User Data ($HOME and $ARCH)
  • 4 x IBM DS3512

    • each 7 Enclosures (a 12 Disks, Mix of SAS-Disks, 600 GB, 15000 RPM and NL-SAS, 2TB, 7200 RPM)
    • each 37 SAS-Disks (RAID5 (4+1))
    • each 47 NL-SAS-Disks (RAID5 (4+1))
    • each 116.2 TB gross, 79.4 TB net (RAID5, Hotspares)
    • TSM Data (DB, Log, Staging)

 

JUST4-GSS: GPFS Storage Server
  • 20 x IBM GSS-24

    • each 2 x IBM x3650 M4 Systems (x-Series)

      • each 2 x Intel Xeon Processors E5-2620, 8 cores, 2.6 GHz, 64 GB Memory
      • each 3 x LSI 9201-16e Quad-Port SAS 6Gb HBA (12x)
      • each 3 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter (6x)
      • Software: RedHat 6.3, XCAT, GPFS-GNR 1.1
    • each 4 x GSS-Storage (JBODs)

      • each 5 Drawers with 12 slots
      • each 58 x 2 TB NL-SAS Disks (RAID6 (8+2))
      • 3 GSS-Storage with additional 2 x 200 GB SSD
    • each 232 NL-SAS Disks
    • each 6 SSDs (GSS-Configuration and Logging)
    • each 464 TB gross, 371 TB net (RAID6)
    • JUST User Data and Metadata ($WORK and $DATA)
  • 3 x IBM GSS-24

    • each 2 x IBM x3650 M4 Systems (x-Series)

      • each 2 x Intel Xeon Processors E5-2620, 8 cores, 2.6 GHz, 64 GB Memory
      • each 3 x LSI 9201-16e Quad-Port SAS 6Gb HBA (12x)
      • each 3 x Mellanox Dual-Port SFP + 10 Gigabit Ethernet Adapter (6x)
      • Software: RedHat 6.4, XCAT, GPFS-GNR 1.5
    • each 4 x GSS-Storage (JBODs)

      • each 5 Drawers with 12 slots
      • each 58 x 4 TB NL-SAS Disks (RAID6 (8+2))
      • 1 GSS-Storage with additional 2 x 200 GB SSD
    • each 232 NL-SAS Disks
    • each 2 SSDs (GSS-Configuration Backup)
    • each 928 TB gross, 742 TB net (RAID6)
    • JUST User Data and Metadata (future $HOME, Q3'2014)



Servicemeu

Homepage