Presentation is loading. Please wait.

Presentation is loading. Please wait.

ECMWF 1 COM INTRO 2004: Introduction to file systems Introduction to File Systems Computer User Training Course 2004 Carsten Maaß User Support.

Similar presentations


Presentation on theme: "ECMWF 1 COM INTRO 2004: Introduction to file systems Introduction to File Systems Computer User Training Course 2004 Carsten Maaß User Support."— Presentation transcript:

1 ECMWF 1 COM INTRO 2004: Introduction to file systems Introduction to File Systems Computer User Training Course 2004 Carsten Maaß User Support

2 ECMWF 2 COM INTRO 2004: Introduction to file systems Overview Workstation server ecgate – HOME – SCRATCH – SCRATCHDIR – (TMPDIR) Supercomputer: hpca – HOME – TEMP – TMPDIR File systems cross-mounted Important remarks - Summary

3 ECMWF 3 COM INTRO 2004: Introduction to file systems File system types Permanent – $HOME is the only permanent file system – backed up daily Temporary – files are kept as long as possible – no backups Automatically deleted – files are deleted at the end of your job or interactive session – no backups ECFS (archive) – long term archive – for excess data/files

4 ECMWF 4 COM INTRO 2004: Introduction to file systems Structure of ECMWF file systems

5 ECMWF 5 COM INTRO 2004: Introduction to file systems ecgate: $HOME NFS mounted from HP High Availability servers, 14GB backups permanent files: profile files + e.g. utilities, source, jobs quota of 40 MB ecquota –v will show disk usage $HOME examples /home/$GROUP/$USER /home/ms/$GROUP/$USER /home/ectrain/tra /home/ms/it/cnv /home/ms/spfrpech/aum

6 ECMWF 6 COM INTRO 2004: Introduction to file systems ecgate: $SCRATCH (locally) mounted, currently 1 file system, 530 GB to be used for temporary data quota: 2 GB (soft limit 1.5 GB) select/delete is running: – when 80% full, file system cleans up to 50% – files newer than 3 days not removed Please remove all files no longer needed! $SCRATCH examples /scratch/$GROUP/$USER /scratch/ms/$GROUP/$USER /scratch/ectrain/tra /scratch/ms/it/cn0 /scratch/ms/spfrpech/aum

7 ECMWF 7 COM INTRO 2004: Introduction to file systems ecgate: $SCRATCHDIR Part of $SCRATCH, 530 GB automatically deleted at the end of job suitable for temporary data $SCRATCHDIR examples: interactive: /scratch/$GROUP/$USER/scratchdir/$HOST.PID /scratch/us/usc/scratchdir/ecgate.144600 batch: /scratch/$GROUP/$USER/scratchdir/$HOST.JobID.StepID /scratch/us/usc/scratchdir/ecgate.31487.0

8 ECMWF 8 COM INTRO 2004: Introduction to file systems ecgate: $TMPDIR Local file system, 2.0 GB Dont use it! used by the system for temporary file storage automatically deleted at the end of job/session $TMPDIR examples: interactive: /var/tmp/tmpdir/$USER/jtmp.PID /var/tmp/tmpdir/usc/jtmp.14460 batch: /var/tmp/nqstmp/$USER/host.JobID.StepID /var/tmp/nqstmp/usc/ecgate.31489.0

9 ECMWF 9 COM INTRO 2004: Introduction to file systems hpca: $HOME General Parallel File System (GPFS), 8.7 GB permanent files: profile files + e.g. utilities, source, jobs block size 64 KB quota of 80 MB quota command will show disk usage $HOME examples /home/$GROUP/$USER /home/ms/$GROUP/$USER /home/ectrain/tra /home/ms/it/cn0 /home/ms/spfrpech/aum more permanent disk space for common library or data installation on request

10 ECMWF 10 COM INTRO 2004: Introduction to file systems hpca: $TEMP GPFS, 1.1 TB (shared between all MS users) to be used for temporary data block size 256 KB select/delete is running Please remove all files no longer needed! $TEMP examples /hpca/tmp/$GROUP/$USER /hpca/tmp/ms/$GROUP/$USER /hpca/tmp/ectrain/tra /hpca/tmp/ms/it/cn0 /hpca/tmp/ms/spfrpech/aum

11 ECMWF 11 COM INTRO 2004: Introduction to file systems hpca: $TMPDIR Part of $TEMP, 1.1 TB (shared between all MS users) automatically deleted at the end of job or interactive login can be used for data $TMPDIR examples: interactive: /hpca/tmp/$GROUP/TMP/JTMP/#/$HOST_$USER_PID /hpca/tmp/us/TMP/JTMP/7/hpca2501_usc_p77840 batch: /hpca/tmp/$GROUP/TMP/BATCH/#/$HOST.JobID.StepID /hpca/tmp/us/TMP/BATCH/2/hpca2401.36350.0

12 ECMWF 12 COM INTRO 2004: Introduction to file systems Cross mounted file systems Purpose: facilitate commands like ls etc. on remote machines, not suitable for large data transfers, preferably not to be used in batch ecgate /ws$HOME /ws/scratch/ms/$GROUP/$USER hpca/hpcb /hpca$HOME /hpcb$HOME (to be made available) hpca/hpcb /hpca$HOME /hpcb$HOME (to be made available) on hpca on ecgate

13 ECMWF 13 COM INTRO 2004: Introduction to file systems Important remarks – Summary Use only the following file systems hpcaTypeSuitable for $HOMEGPFSpermanent files: sources,.profile, utilities, libs … $TEMPGPFStemporary files $TMPDIRGPFSdata to be automatically at the end of a job ecgateTypeSuitable for $HOMEUFSpermanent files: sources,.profile, utilities, libs $SCRATCHUFStemporary files $SCRATCHDIRUFSdata to be automatically deleted at the end of a job

14 ECMWF 14 COM INTRO 2004: Introduction to file systems Important remarks – Summary Only $HOME is backed up All $HOME file systems are different A select/delete policy is applied on all temporary file systems Make use of the environment variables $HOME etc. Check your quota with – ecquota –v # on ecgate – quota # on hpca Clear your space as soon as possible Do not rely on select/delete ECFS is accessible from both hpca and ecgate

15 ECMWF 15 COM INTRO 2004: Introduction to file systems Exercise On Linux desktop or ecgate: 1.List for your training uid the paths for the following file systems: $ HOME, $TMPDIR, $SCRATCH, $SCRATCHDIR. 2.Check your quotas. 3.Which is your largest directory/file (in kb)? 4.Find any file in your directories larger than 100 kb! 5.How much disk space (in kb) is available in the /home/ectrain file system? 6.List your hpca HOME directory! 7.How many directories with the name Trash do you find under your HOME directory? 8.In your HOME directory create a directory called public with the following permissions: rwxrwxr-x.


Download ppt "ECMWF 1 COM INTRO 2004: Introduction to file systems Introduction to File Systems Computer User Training Course 2004 Carsten Maaß User Support."

Similar presentations


Ads by Google