# GPFS data questions

What file system to use for different data?

In multiple GPFS file systems for different types of user data. Each file system has its own data policies.

• $HOME Acts as repository for the user’s personal data like the SSH key. There is a separate HOME folder for each HPC system and a shared folder which pointed on all systems to the same directory. Data within$HOME are backed up by TSM, see also

• $SCRATCH Is bound to a compute project and acts as a temporary storage location with high I/O bandwidth (measured 150 GB/s from an JUWELS application). If the application is able to handle large files and I/O demands,$SCRATCH is the right file system to place them. Data within $SCRATCH is not backed up and daily cleanup is done. • Normal files older than 90 days will be purged automatically. In reality modification and access date will be taken into account, but for performance reasons access date is not set automatically by the system but can be set by the user explicitly with touch -a <filename>. Time stamps that are recorded with files can be easily listed by stat <filename>. • Empty directories, as they will arise amongst others due to deletion of old files, will be deleted after 3 days. This applies also to trees of empty directories which will be deleted recursively from bottom to top in one step. •$PROJECT
Data repository for a compute project. It's lifetime is bound to the project lifetime. Data are backed up by TSM.

• $FASTDATA Belongs to a data project. This file system is bandwidth optimized (similar to$SCRATCH), but data are persistent backed up by TSM.

• $DATA Belongs to a data project. This file system is designed to store a huge amount of data on disk based storage. The bandwidth is moderate. Backup is realized with the GPFS snapshot feature. For more information, look at •$ARCHIVE
Is bound to a data project and acts as storage for all files not in use for a longer time. Data are migrated to tape storage by TSM-HSM. It is recommended to use tar-files with a maximum size of 1 TB. This is caused by the speed for reading/writing data from/to tape. All data in $ARCHIVE first has to be backed up which will take 10h for 1TB. Next the data will be migrated to tape which will take 3h per 1TB. Please keep in mind that a recall of the data will need approximately the same time. See also All GPFS file systems are managed by quotas for disk space and/or number of files. See also What data quotas do exist and how to list usage? For all data repositories the disk quota managing is enabled. The values are set to default values (defined by JSC) or depend on special requirements by the projects. Default data quota per user/project within GPFS file systems File System Disk Space Number of Files Soft LimitHard Limit Soft LimitHard Limit$HOME10 GB11 GB10.00011.000
$SCRATCH90 TB95 TB4 Mio4.4 Mio$PROJECT16 TB17TB3 Mio3.1 Mio
$FASTDATAdependdependdependdepend$DATAdependdependdependdepend
$ARCHIVE- (see note)500.000550.000 Note: No hard disk space limit for$ARCHIVE exists. Furthermore for some projects there may exist special guidelines.

File size limit

Although the file size limit on operation system level e.g. at JUWELS or JURECA is set to unlimited (ulimit -f) the maximum file size can only be the GPFS group quota limit for the corresponding file system. The actual limits can be listed by jutil.

List data quota and usage by project or user

Members of a group/project can display the hard limits, quotas (soft limit) and usage by each user of the project using the jutil command.

jutil project dataquota -p <project name>

The quota information for the users are updated every 8 hours.

Recommendation for users with a lot of small files

Users with applications that create a lot of relatively small files should reorganize the data by collecting these files within tar-archives using the

tar -cvf archive-filename ...

command. The problem is really the number of files (inodes) that have to be managed by the underlying operating system and not the space they occupy in total. On the other hand please keep in mind the recommendations under File size limit.

How to modify the users's environment.

When users login on an frontend node using ssh a shell will be started and a couple of environment variables will be set. These are defined in system profiles. Each user can add/modify his environment by using his own profiles in his HOME directory.

In the Jülich setup there will be a separate HOME directory for each HPC system. Which means that the environment differs between JUWELS, JURECA; JUDAC; ... and also the user can modify his own profiles for each system separately. Therefore a skeleton .bash_profile and .bashrc are placed in each $HOME directory when a user is joined to any HPC system. .bash_profile: # ************************************************** # bash environment file in$HOME
# http://www.fz-juelich.de/ias/jsc/EN/Expertise/D...
# **************************************************
# Get the aliases and functions: Copied from Cent...
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
export PS1="[\u@\h \W]\$" .bashrc: # ************************************************** # bash environment file in$HOME
# http://www.fz-juelich.de/ias/jsc/EN/Expertise/D...
# **************************************************
# Source global definitions: Copied from CentOS 7...
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi

Separate HOME directory for each HPC system

E.g. on JUDAC user graf1 will see $HOME="/p/home/jusers/graf1/JUDAC". The profiles located here were used for login. Only the shared folder (link) points always to the same directory /p/home/jusers/graf1/shared. Most side dependend variables are set automatically by the jutil env init command (system profile). The user can set the right core variables ($PROJECT, $ARCHIVE, ...) by using jutil env activate -p <project> For more information look at the jutil command usage. How to see the currently set budget: If a user has to change his budget account during a login session it might be helpful to see the currently set budget account in his prompt to be sure to work on the correct budget.Therefore one should replace the current "export PS!=..." line in .bash_profile by: prompt() { PS1="[${BUDGET_ACCOUNTS:-\u}@\h \W]\$" } PROMPT_COMMAND=prompt This results in the following behaviour: [user1@juwels07 ~]$ jutil env activate -p chpsadm
[hpsadm@juwels07 ~]$jutil env activate -p cslfse [slfse@juwels07 ~]$

How can I recall migrated data?

Normally migrated files are automatically recalled from TSM-HSM tape storage when the file is accessed on the login nodes of the HPC systems (e.g.. JUWELS, JURECA, ...) or the Data Access System (JUDAC).

For an explicit recall the native TSM-HSM command dsmrecall is not available. Please use

tail <filename>
or:

to start the recall process. These commands will not change any file attribute and the migrated version of the file as well as the backup version stay valid.

It is strongly recommended NOT to use

touch <filename>

because this changes the timestamp of the file, so a new backup copy must be created and the file has to be migrated again. These are two additional processes that waste compute ressources, if the file is used read only by further processing.

How can I see which data is migrated?

There are two file systems which hold migrated data: /arch and /arch2

• These are so called archive file systems.
• In principle all data in the file systems will be migrated to TSM-HSM tape storage in tape libraries.
• Data is copied to TSM backup storage prior to migration.

All files within the data project directories ($FASTDATA) are automatically backed up by TSM (Tivoli Storage Manager) function. To restore a file, use adsmback -type=fastdata & on JUDAC. This command grants access to the correct backup data of the project repository. Follow the GUI by selecting: Restore -> View -> Display active/inactive files File level -> p> fastdata -> group -> ... Select files or directories to restore Press [Restore] button If the data should be restored to original location then choose within the Restore Destination window • Original location Otherwise select: • Following location + <path> + Restore complete path ##$DATA - Data project repository (large capacity)

The files within the data project directories ($DATA) are not backed up by TSM (Tivoli Storage Manager) function. Here we use the snapshot feature from the file system (GPFS). The difference between the TSM backup and the snapshot based backup is, that TSM act on file changes while snapshots save the state at a certain point in time. Right now we have configured:  daily backup last three retention today, just after midnight weekly backup last three retention every Sunday, just after midnight monthly backup last three retention every 1st day of month, just after midnight The snapshots can be found in a special subdirectory of the project repository. Go to cd$DATA/.snapshots

and list contents

/p/largedata/jsc> ls
daily-20181129
daily-20181130
daily-20181203
weekly-20181118
weekly-20181125
weekly-20181202
monthly-20181001
monthly-20181101
monthly-20181201

In the subdirectory <type>-<YYYYMMDD> the file version which was valid at date DD.MM.YYYY can be retrieved using the same path as the actual file is placed in the $LARGEDATA repository. Due to the fact that the snapshot is part of the file system, the data restore can be performed on any system where it is mounted. ##$ARCHIVE - The Archive data repository

All files within the user's archive directory (\$ARCHIVE) for long term storage are automatically backed up by TSM (Tivoli Storage Manager) function. To restore a file, use

on JUDAC.

This command grants access to the correct backup data of the project's assigned archive directory.

Restore -> View -> Display active/inactive files
File level -> archX -> group -> ...
Select files or directories to restore
Press [Restore] button

If the data should be restored to original location then choose within the Restore Destination window:

• Original location

Otherwise select

• Following location + <path> + Restore complete path
How to share files by using ACLs?

Linux file permission define the access rights to read, write or execute (rwx) files and directory but is limited to one user, one group and all others. ACLs (Access Control Lists) allows a more fine-grained assignment of access rights. The owner of a file/directory can define specific rights for other users and groups.

## Linux commands to manage ACLs

- command to list ACLs of a file/directory:

getfacl <file/directory>

- Give user john1 read and write controle to file example.txt. Also give user lucy1 the right to read this file:

setfacl -m -u:john1:rw example.txt
setfacl -m -u:jim1:r example.txt

# file: example.txt
# owner: smith1
# group: cjsc
user::rw-
user:john1:rw-
user:lucy1:r--
group::---
other::---

- remove user john1 ACLs on example.txt:

setfacl -x u:john1 example.txt

# file: example.txt
# owner: smith1
# group: cjsc
user::rw-
user:lucy1:r--
group::---
other::---

- Allow users from group zam change to directory share:

setfacl -m g:zam:x share/

# file: share
# owner: smith1
# group: cjsc
user::rwx
group::---
group:zam:--x
other::---

- remove all ACLs from directory share::

setfacl -b share

# file: share
# owner: smith1
# group: cjsc
user::rwx
group::---
other::---

Further information (e.g. set ACLs recursivly, setting default ACLs, inherit ACLs, ...) can be found in the manual pages.

## Which files have an access control list?

The command

ls -l

will show a "+" for every file that has ACL set, eg.

drwx------+ 2 john1 cjsc 32768 Feb 21 09:25 share

How to avoid multiple SSH connections on data transfer?

When transferring multiple files, it can be problematic to use a separate SSH connection for each transfer operation. The network firewall can block a large amount of independent simultaneous SSH connections. There are different options to avoid multiple SSH connections:

Use rsync or use scp with multiple files:

rsync only copies new or changed files, this will reserve transfer bandwith.

will copy local_folder recursively

Use tar-container to transfer less files

Creating a tar file and transfer it can be much faster compared to transferring all files separately:

tar -cf tar.file local_folder

The tar file creation, transmission and extraction procress can also be done on the fly:

tar -c local_folder/ | ssh username@host \
'cd remote_folder; tar -x'

Use shared SSH connection

Shared SSH connection allow usage of the same connection multiple times:

Open master connection:

Reuse connection:

A shared connection can also be used when using scp:

scp -o 'ControlPath /tmp/ssh_mux_%h_%p_%r' \