1. HPSS @ CC-IN2P3 Pierre-Emmanuel Brinette Benoit Delaunay IN2P3-CC Storage Team 17 may 2011

2. HPSS@CCIN2P3Pierre-Emmanuel Brinette2 IN2P3, who are we ? IN2P3, is the National Institute for Nuclear and Particle Physics Research. Composed of 20 laboratories, 3 experimental sites and 1 computer center, all distributed across the French territory. Around 2500 collaborators, including 700 researchers. Involved into international experiments of High Energy Physics and Astrophysics. 2

3. HPSS@CCIN2P3Pierre-Emmanuel Brinette3 3 The Large Hadron Collider - LHC The LHC accelerator can be qualified with many superlatives : Largest accelerator in the world (27 km) Largest accelerator in the world (27 km) Largest instrument for fundamental science Largest instrument for fundamental science Largest cryogenics installation Largest cryogenics installation Colder than the outer space Colder than the outer space But collisions are 100 000 times hotter than the heart of the sun But collisions are 100 000 times hotter than the heart of the sun … and so on … 4 experiments which are worldwide collaborations ALICE – ATLAS – CMS - LHCb

4. HPSS@CCIN2P3Pierre-Emmanuel Brinette4 CCIN2P3, who are we ? 4 CCIN2P3 is the dedicated Computer Center of the Institute. Is a specialized laboratory offering computing and mass storage resources to researchers. At the service of all (international) experiments in which the Institute aims to collaborate. Employing 90 engineers. Using two computer rooms of 800m 2 each.

5. HPSS@CCIN2P3Pierre-Emmanuel Brinette5 Storage services overview Tape backend 11 PB dCacheXrootd iRods SRB 6 PB1 PB 250 TB 300 TB Mass Storage Systems Semi permanent Storage GPFS 900TB $HOME AFS BACKUP TSM 90 Groups (VO) (HEP,astro,bio,H&SS) 2000+Users

6. HPSS@CCIN2P3Pierre-Emmanuel Brinette6 HPSS @ IN2P3 Numbers –12 PB (10^15 B) –30 M Files (15 M < 64 MB) –16.000 tape –2000 accounts (most inactive) Daily access (Avg) –70.000 Transfers –1000 Tape mounts –25 TB Write –16 TB Read –Peak = 2*Avg Pierre-Emmanuel Brinette HPSS 7.3.2

7. HPSS@CCIN2P3Pierre-Emmanuel Brinette7 HPSS Infrastructure

8. HPSS@CCIN2P3Pierre-Emmanuel Brinette8 Storage policy Historical –7.3 PB –27 M files Subsys 1Sub 2Sub 3Sub 4Sub 5 Newly created –3.7 PB –3 M files –Should increase COS selection by size SMALL  T10K-A Sport | Medium  T10K-A | LARGE  T10K-B Specific COS Dedicated subsystem –Allow to dedicate DISK resources for specific set of users when using automatic COS selection –Specific database for a set users  faster query

9. HPSS@CCIN2P3Pierre-Emmanuel Brinette9 Storage Policy No more dedicated COS (since 2006) Use file size based COS –Small (<64 MB)  T10K-A Sport (120GB)2000 T –Medium (64-512 MB)  T10K-A (500 GB)4000 T –Big (512 MB – 8 GB)  T10K-B (1GB)8600 T –Huge (> 8GB)  T10K-B (1GB)800 T Use of Subsystems for dedicating disk resources Use file family for dedicating tape to a set of users Small files : –Use large cache disk (50 TB) –Most files remains on cache Pierre-Emmanuel Brinette

10. HPSS@CCIN2P3Pierre-Emmanuel Brinette10 HPSS World HPSS data access RFIO –Provide Unix like interface to end users : rfcp, rfdir, rfrm, … –rfcp client use HPSS API and mover protocol hpss_readlist hpss_writelist Benefits –Good performances, direct transfers from movers –Easy to use lightweight client –Access trough a control server Limit simultaneous Cx Logging RFIOd HPSS mover client $ rfcp rfiosrv:/hpss/in2p3.fr/test/10GB.dat /scratch 10485760000 bytes in 19 seconds through eth2 (in) and local (out) (538947 KB/sec) SetLoginCred() hpss_open() hpss_readlist() rfio_open() rfio_read() MOVER PROTOCOL IOD IOR log Pierre-Emmanuel Brinette 520 MB/s

11. HPSS@CCIN2P3Pierre-Emmanuel Brinette11 TReqS Read operations in HPSS are chaotic –Reprocessing activities implies read of all raw data, stored many years ago –Sometimes 1 read = 1 tape mount. Increase cache size is OK for small files but not applicable for big files. Solution : –Optimizing read operation by sorting file request by tape position. –First adapt a tape scheduler used at BNL –Design and code our own solution (TReqS) –Used in production for dCache et Xrootd HPSS dCacheXrootd RFIO TReqS GET files Status File TAPE ? Position ? STAGE ! rfcp

12. HPSS@CCIN2P3Pierre-Emmanuel Brinette12 HPSS Migration HPSS 7.3.2 –Migration from 6.2.3 on February 2011 –3 days shutdown Complete installation on a new Core server –Metadata transferred and converted on the new server –Allow fall back in case of failure of the migration procedure –Need too purge all Level 0 disk cache (~ 1 week) Benefits: –Stability improved ( in comparison of v6.2.3 ) Issues : –Before migration : Solaris HPSS client (API) doesn’t compile –Repack behaviors have changed Doesn’t works in ‘exclusive mode’ anymore Repack almost failed, still few segments remaining on tape Rerun repack works.

13. HPSS@CCIN2P3Pierre-Emmanuel Brinette13 Infrastructure evolution No relevant changes since last year Evolution –Replace EOL AIX tape mover by Linux (Dell) –Continue to use AIX core server as long as possible –Introduce T10K-C (5TB) Significant budget reduction for the next years –DDN disk array “given” to GPFS service –Currently testing “low cost” disk server in HPSS DELL PowerEdge R510 12 * 2TB (3”5) 7.2k SAS + DELL PowerVault MD1200 12 * 2TB (3”5) 7.2k SAS Or DELL PowerVault MD1220 24 * 600 GB (2”5) 10k SAS 0.27 € / GiB 54 TiB usable (2 MD1200) 4*1 Gbits 5 year support

14. HPSS@CCIN2P3Pierre-Emmanuel Brinette14 HPSS Operations Disaster recovery –Almost completely defined –Spare p550 server w/DB2 & HPSS –recovery script Copy last backup from TSM RESTORE databases ROLLFORWARD Monitoring –StorSentry for tape library –HPSS messages sent to central logging systems (whpss) –Tape activity RRD plot (pvljob) Nagios Pierre-Emmanuel Brinette