1. mbutler@ncsa.illinois.edu BlueWaters Storage Solution Michelle Butler NCSA January 19, 2016

2. mbutler@ncsa.illinois.edu Overview Disk Storage Solution - Hardware and Software Archive Storage Solution – Hardware and Software Lustre – User needs to know more Our current work 2

3. mbutler@ncsa.illinois.edu Disk Storage Solution 3 Presentation Title /project 2.16 PB /home /scratch 21.6 PB QDR InfiniBand I/E DM esLoginesLogin esLoginesLogin HPSS DM UsersUsers UsersUsers

4. mbutler@ncsa.illinois.edu Disk Storage System – Lustre 2.1 OpenSFS SonExion Lustre Appliance from Cray /home and /project 2.16 PB useable 3 cabinets with 6 scalable storage units each. Each SSU has 2 sister pair failover OSS Each OSS has 8 OSTs of 10 drives each (80 drives) Total of 144 OSTs Software RAID in 8+2 config with 2 hot spares disks /scratch 21.60PB useable 30 SonExion cabinets 1440 OST’s Total aggregate throughput of > 1TB/s 4 Presentation Title

5. mbutler@ncsa.illinois.edu Archive Storage System RFP1107 from UofI - 100GB/s with 500PB media ~420 TS1140 drives with hot spares added 7 SpectraLogic tape libraries for 380PB slots ½ environment in 2012 ~ March 2012 ½ environment in early 2013 after production Buying 4TB tapes in chunks of 20PB $$ of media was the largest portion of cost 5 Presentation Title

6. mbutler@ncsa.illinois.edu Lustre – User Perspective Lustre does not stripe data on behalf of the user. Directories will be defaulted to a default stripe depth Users can change the stripe depth easily based on their data, but do need experience at these types of settings. Stripe depth is set by #of OST’s A single file can not stripe over 160 OST’s today. Stay tuned for best practices for the file systems including stripe depth and block sizes. 6 Presentation Title

7. mbutler@ncsa.illinois.edu Lustre – User perspective Current work Job Placement and OST’s to use for read and write Working with I/O libraries for Lustre file system configurations (Adios, NetCDF, HDF5) Data movement to and from archive Before job begins have all data in Lustre Store data that is wanted to be kept All within quotas Parallel user commands such as grep, cp, mv… 7 Presentation Title

8. mbutler@ncsa.illinois.edu Other Projects RAIT for HPSS Stability for these large file systems Back up of /home and /project file systems Data movement from outside projects and other large NSF centers and use of those resources OpenSFS directions for Lustre 8 Presentation Title