1. Active Storage and Its Applications Jarek Nieplocha, Juan Piernas-Canovas Pacific Northwest National Laboratory 2007 Scientific Data Management All Hands Meeting Snoqualmie, WA

2. 2 Outline Description of the Active Storage Concept New Implementation of Active Storage Programming Framework Examples and Applications

3. 3 Active Storage in Parallel Filesystems Active Storage exploits the old concept of moving computing to the data source to avoid data transfer penalties applications use compute resources on the storage nodes Storage nodes are full-fledged computers with lots of CPU power available, and standard OSes and Processors P P P P Network FS compute nodes I/O nodes Y=foo(X) x Y P P P P Network FS compute nodes I/O nodes Y=foo(X) Active Storage Traditional Approach

4. 4 Example BLAS DSCAL on disk Y = α. Y Experiment – Traditional: The input file is read from filesystem, and the output file is written to the same file system. The input file has 120,586,240 doubles. – Active Storage: Each server receives the factor, reads the array of doubles from its disk locally, and stores the resulting array on the same disk. Each server processes 120,586,240/N doubles, where N is the number of servers Speedup contributed to using multiple OSTs and avoiding data movement between client and servers (no network bottleneck)

5. 5 Related Work Active Disk/Storage concept was introduced a decade ago to use processing resources ‘near’ the disk On the Disk Controller. On Processors connected to disks. Reduce network bandwidth/latency limitations. References DiskOS Stream Based model (ASPLOS’98: Acharya, Uysal, Saltz) Active Storage For Large-Scale Data Mining and Multimedia (VLDB ’98: Riedel, Gibson, Faloutsos) Research proved Active Disk idea interesting, but Difficult to take advantage of in practice Processors in disk controllers not designed for the purpose Vendors have not been providing SDK Y=foo(X)

6. 6 Lustre Architecture Client OST MDS O(10) OST O(1000) O(10000) Network Directory Metadata & concurrency File IO & Locking Recovery, File Status, File Creation Client

7. 7 Active Storage in Kernel Space When the client writes to the file A: ASOBD makes a copy of data, and sends it to ASDEV The PC reads from and writes to the char device Original data in A, processed data in B Char device Disk NAL OBDfilter OST Ldiskfs Kernel space ASOBD ASDEV User space Processing component AB Active Storage Module

8. 8 Active Storage Application High Throughput Proteomics 9.4 Tesla High Throughput Mass Spectrometer 1 Experiment per hour 5000 spectra per experiment 4 MByte per spectrum Per instrument: 20 Gbytes per hour 480 Gbytes per day Application Problem Given 2 float input number for target mass and tolerance, find all the possible protein sequences that would fit into specified range Active Storage Solution Each OST receives its part of the float pair sent by the client stores the resulting processing output in its Lustre OBD (object-based disk) Next generation technology will increase data rates x200

9. 9 SC’2004 StorCloud Most Innovative Use Award Sustained 4GB/s Active Storage write processing Proteomics Application 320 TB Lustre 984 400GB disks 40 Lustre OSS's running Active Storage 4 Logical Disks (160 OST’s) 2 Xeon Processors 1 MDS 1 Client creating files Lustre OST Client System Gigabit Network Lustre OSS 39 Lustre OST Lustre OST Lustre MDS Lustre OSS 0 Lustre OSS 38

10. 10 Active Storage in User Space Problems with the Kernel Space implementation Portability, maintenance, extra memory copies We developed a User Space implementation Most file system allows the storage nodes to be clients Most file system allows to create files with a given layout Our framework launches Processing Components on the storage nodes which have the files to be processed Processing Components read from and write to local files Highly Portable Implementation Used with Lustre 1.6, PVFS2 2.7 Bug in Lustre 1.4 (and SFS): frequent kernel crashes when mounting the file system on the storage nodes Held initial discussions with IBM on GPFS port

11. 11 Active Storage in User Space Network Interconnect Compute Node Compute Node Compute Node Storage Node 0 Storage Node N-1 Metadata Server Parallel Filesystem's Components (also clients of the filesystem)..... Parallel Filesystem's Clients..... Data I/O Traffic asmaster ASRF Processing Component ASRF Processing Component Compute Node Active Storage Runtime Framework

12. 12 Performance Evaluation AMINOGEN Bioinformatics Application Input file: ASCII file, mass and tolerance pairs, one per line. Total size = 44 bytes Output file: binary file which contains amino acid sequences. Total size = 14.2 GB Overall execution time

13. 13 Enhanced Implementation of Active Storage for Striped Files Striped Files broadly used for performance not supported by earlier AS work Enhanced Implementation Use striping data from filesystem New component: AS Mapper Locality awareness in Processing Component: compute on local chunks Climate application with netCDF Computes statistics of key variables from Global Cloud Resolving simulation (U. Colorado) Eliminated >95% network traffic LIBAS read Local chunks 2 Contiguous file Write Local chunks 610 1418... Local chunks 234 Active Storage Runtime Framework... Processing component read callwrite call GLIBC readwrite 01 Processing Component

14. 14 Examples and Applications Juan Piernas-Canovas

15. 15 Active Storage in DSCAL Example OST31 /lustre OST43 /lustre Comp. Node /lustre Comp. Node /lustre Comp. Node /lustre Comp. Node /lustre Data I/O Traffic Parallel Filesystem's Clients Parallel Filesystem's Components dscal Network Interconnect MDS & MGS asmaster Doubles.20 Doubles.15.outDoubles.20.out Doubles.15

16. 16 Non-Striped Files /lustre/doubles.* /lustre/dscal 12345.67890 @ @.out /lustre/doubles.15 in OST43 /lustre/doubles.15.out in OST43 (new file)

17. 17 Climate Application Collaboration with SciDAC GCRM SAP (Karen) Problem: Compute averages for variables generated from scientific simulation stored in striped output files geodesic grid netCDF data format Objective: Optimize performance by exploiting data locality in AS Processing Components to minimize network traffic

18. 18 Non-Striped Files /lustre/doubles.* /lustre/dscal 12345.67890 @ @.out /lustre/doubles.20 in OST31 /lustre/doubles.20.out in OST31 (new file) Execution: /lustre/asd/asmaster /lustre/dscal.xml

19. 19 Processing Patterns In user space, it is easy to support different processing patterns: Active Storage PC Client data stream Active Storage Client data stream PC 1W  01W  #W

20. 20 No Output File (Pattern 1W  0) /lustre/doubles.* /lustre/dscal1 12345.67890 @ /lustre/doubles.15 in OST43

21. 21 Several Output Files (Pattern 1W  #W) /lustre/doubles.* /lustre/dscal3 12345.67890 @ @.out @.err /lustre/doubles.15 in OST43 /lustre/doubles.15.out in OST43 (new file) /lustre/doubles.15.err in OST43 (new file)

22. 22 Transparent Access to Striped Files /lustre/doubles.* /lustre/dscal 12345.67890 @{hidechunks} @{copystriping,hidechunks}.out Transparent access to the chunks of the input file Transparent access to the chunks of the output file New output file with the same striping of the input file

23. 23 Mapper and Striped netCDF Files Network Interconnect Storage Node 0 Metadata Server Data I/O Traffic asmaster ASRF PC Storage Node 1 ASRF Storage Node 2 ASRF PC Storage Node N-1 ASRF...... HeaderVar. data Mapper  (0, 2) Striped netCDF file......

24. 24 Processing of netCDF files @.out-${NODENAME} /lustre/data.* /lustre/processnetcdf.py @ ta /lustre/netcdfmapper.py @ ta ${CHUNKNUM} ${CHUNKSIZE} Striping information of /lustre/data.37 Variable name in the netCDF file /lustre/data.37 Non-striped output file /lustre/data.37.out-ost43

25. 25 PVFS2 support /lustre/doubles.* /lustre/dscal 12345.67890 @{hidechunks} @{copystriping,hidechunks}.out pvfs /pvfs2 PVFS2

26. 26 Local File System with Virtual Striping /lustre/doubles.* /lustre/dscal 12345.67890 @{hidechunks} @{copystriping,hidechunks}.out localfs 8:1048576 Virtual striping: - stripe size: 1MB - stripe count: 8 Local file system

27. 27 Further Information Technical paper J. Piernas, J. Nieplocha, E. Felix, “Evaluation of Active Storage Strategies for the Lustre Parallel Filesystem”, Proc. SC’07 Website: http://hpc.pnl.gov/projects/active-storage Upcoming release in December 2007 Support for Lustre 1.6, PVFS2, and Linux local file systems Source code available now under request. Just send us an e-mail! Jarek Nieplocha Juan Piernas-Canovas

28. Active Storage and Its Applications Jarek Nieplocha, Juan Piernas-Canovas Pacific Northwest National Laboratory Questions?