1. Nov 18, 20091/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Grid Storage on FermiGrid and GPCF Activity Discussion Nov 18, 2009 Gabriele Garzoglio Computing Division, Fermilab Overview Motivation for this activity The investigation on Storage on FG Proposed work on Storage and the FG virtual test bed A user-centered storage evaluation methodology Available experience for Lustre and Hadoop

2. Nov 18, 20092/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Motivations Investigation of data access patterns and troubles with storage (docdb 3279) –High-rate I/O accesses from FermiGrid (e.g. some user analysis) strain the mounted storage system i.e. Blue Arc (BA) –Compartmentalization and optimization possible, but non-trivial for lack of monitoring and control (see later) –Analyzed access pattern for DZero, CDF, CMS, Minos & IF for production processing, access to applications, and analysis. Synergy with other Storage activities –GPCF –DMS Storage investigation –Storage on OSG –Storage for LQCG and CC clusters Are there new technologies that meet the needs of our data access use cases? Is BA adequate with additional application-level constraints?

3. Nov 18, 20093/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Results of the Investigation I BA is a valuable solution for many use cases, but has limitations: –lack of monitoring and controls –lack of data traffic controls for the BlueArc –disk cost vs. performance –data migration policies do not work for all communities –unexpected changes in user access patterns Results of early measurements on BA* (* Alex Kulyavtsev, Ron Rechenmacher, Dec 2008) –Rates saturate at about ~20 clients –Distribution of bandwidth is “fare” –Rates have been tuned up since then

4. Nov 18, 20094/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Results of the Investigation II Some user behaviors and needs exacerbate the problem –uncoordinated concurrent access to storage –use of the file system as a file catalogue –users not adhering to recommended best practices (reading data directly from BA, ignoring recommended fair-share quotas on local disk; inappropriate use of home areas, …) –users transferring one file per security session (SRM / dCache), instead of multiple files –running “short” jobs Some classical solutions are presented –adoption of a peer-to-peer storage solution (e.g. SAM for DZero, Hadoop HDFS, DiskFarm) –data access queuing (e.g. SRM, FCP, cpN) –job throttling (e.g. via Condor)

5. Nov 18, 20095/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Possible further Investigations Investigating how emerging storage solutions, such as Hadoop or Lustre, and the synergy with the BA storage Investigating how peer-to-peer storage solutions adopted in the past, such as DiskFarm Investigating how SAM can interface BA to coordinate access to files for IF Investigating standard or commercial data queuing services, equiv. to fcp Investigating how to shape traffic to the BA by intercepting I/O directed to it, similarly to Parrot Implementing ephemeral home areas at the FermiGrid worker nodes Reorganizing the BA volumes to optimize rate of access and to compartmentalization Educating users in low-overhead usage of X509 file transfers Understanding if idle CPUs at CDF grants changing how SAM interfaces to dCache, from using the dcap protocol to using SRM Producing a catalogue of successful use cases and their storage solution

6. Nov 18, 20096/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Proposed Work on Storage We want to investigate the adequacy of Hadoop (HDFS) and Lustre for the use cases of the FG stakeholders. The solution should… –work efficiently and reliably for data and CPU intensive jobs (focus on user needs) –be “cheap” in terms of hardware + operational costs. We will use FermiCloud as a virtual test bed –32 VM on 4 Dual Quads node (details TBD) –16 GB RAM –~ 2TB disk / node –1GB eth The framework is a collaboration among OSG group, FG, FEF, DMS, et al.

7. Nov 18, 20097/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Proposed Lustre Test Bed VM ~2 TB VM ~2 TB VM ~2 TB VM ~2 TB eth Lustre FG Batch Sys. BA mount

8. Nov 18, 20098/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Proposed HDFSTest Bed VM ~2 TB VM ~2 TB VM ~2 TB VM ~2 TB eth HDFS FG Batch Sys. BA mount

9. Nov 18, 20099/15 Grid Storage on FermiGrid and GPCF – Activity Discussion System Measurements Two main methods: work done by real jobs vs. running benchmarks As for benchmarks… Use benchmark to ascertain “optimal” configuration, comparing new installation with previous evaluations Do not redo system evaluations As for real jobs… Load the FS with real user data Measure IO of real user jobs, with least “invasive” techniques (iostat, iotop, etc.) Are users interested in participating?

10. Nov 18, 200910/15 Grid Storage on FermiGrid and GPCF – Activity Discussion System Evaluation DMS storage requirement document (docdb 2576): evaluation of 2 workflows –A large number of processes with low average I/O rate, processing a few independent input files to produce a few output files. (HEP) –Correlated processing of input and output files in a parallel IO system. Many processes writing to one file or each to an individual (local) file to be merged. (LQCD) Categories of Requirements: –Capacity –Data storage functionality, scalability and IO Performance –Data integrity –Tape integration –Usability: Maintenance, Troubleshooting and problem isolation –Data accessibility –Namespace and Namespace performance –Platform and System –Bottom line –Security –Other (ACLs, quotas, allocation, auditing) Do we have a requirement document for RunII I/O, Minos I/O, CMS I/O, …?

11. Nov 18, 200911/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Lustre – Highlights* (* from Nirmal Seenu presentation on Lustre) Shared POSIX file system Key Characteristics –Aggregates many servers into one file system –Scales I/O throughput and capacity –Handles 10,000’s of clients Built-in storage networking, including routing –The client connection guarantees data integrity –No support for data replication (yet) No special hardware required –Any block device is supported as backend storage Open source

12. Nov 18, 200912/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Lustre @ FNAL* (* from Nirmal Seenu presentation on Lustre) LQCD –~2000 clients over IPoIB –Mounting ~2000 clients: a few secs; Un-mounting: a little longer –Performance limited by transfers rates of SATABeast –42 One-TB Disks 3 RAID6 volumes (12/14); Four 3TB LUNS on each RAID volume Computational Cosmology Cluster –Performance limited by the GigE network

13. Nov 18, 200913/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Lustre - DMS Evaluation Test Bed: –6 nodes attached to 6 1Gb ports of a Cisco switch –CPU: dual, quad core Xeon X5355 @ 2.66GHz with 4 MB cache –Memory: 16 GB –Disk: single 500 GB disk – hdparm -t = 77.9 MB/s Tests –Multiple Namespace and Data I/O tests –CC cluster live evaluation –GridFTP integration aggregate I/O rate in KB/sec vs. number of clients KB/sec # clients

14. Nov 18, 200914/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Hadoop – Highlights* (* from CMS paper to the IEEE Nuclear Science Symposium) Shared distributed file system POSIX through Fuse Key Characteristics –Aggregates cheap disks on the WN in a single (striped) FS –10 MB/s per application on loaded cluster; 400 MB/s aggregate rate on WAN with GridFTP –Handles 1,000’s of clients Built-in storage networking –Support for data replication No special hardware required Open source A low stress test of inter-site WAN transfer between SEs of Caltech and UCSD. The transfer rate was measured in a period of 30 days. The average transfer rate is 150 MB/s.

15. Nov 18, 200915/15 Grid Storage on FermiGrid and GPCF – Activity Discussion Conclusions We want to evaluate Lustre and HDFS file systems as an alternative to BA for batch jobs running on FermiGrid These evaluations should feed into the storage solutions of the GPCF We would like to do a user-oriented evaluation: –Will you run jobs on this test infrastructure? –Do we understand your requirements? We collaborate with other groups at FNAL and outside to avoid duplication of effort