1. Enabling Cross-Layer Optimizations in Storage Systems with Custom Metadata Elizeu Santos-Neto Samer Al-Kiswany Nazareno Andrade Sathish Gopalakrishnan Matei Ripeanu University of British Columbia Universidade Federal de Campina Grande

2. HPDC'08http://netsyslab.ece.ubc.ca2 Motivation ~ 160 Exabytes of digital data created in 2006 [Reinsel07]  Harder to organize, search or navigate. Solution : Custom metadata  User-generated metadata assigned to files  Used to improve navigation and search  Communication between applications Idea: what about exploiting custom metadata for cross-layer optimizations too?

3. HPDC'08http://netsyslab.ece.ubc.ca3 Layered Architectures TCP/IP File System Good, but…  …it limits information flow across layers.

4. HPDC'08http://netsyslab.ece.ubc.ca4 Cross-Layer Optimizations Examples  HTTP (Cache directives)  IP (Optional fields) Applications  Storage System  Performance  QoS requirements  Consistency requirements Applications  Storage System  Provide storage-level information to applications Data Intensive Schedulers: Notification about data movements Data Intensive Applications: Co-usage of files

5. HPDC'08http://netsyslab.ece.ubc.ca5 Traditional Use of Custom Metadata Application Layer File System Layer Storage System Layer Metadata Manager File Organization Module Basic File System Author=Smith input.dat File Browser

6. HPDC'08http://netsyslab.ece.ubc.ca6 Cross-Layer Communication Application Layer File System Layer Storage System Layer Metadata Manager File Organization Module Basic File System Replicate input.dat 3x input.dat moved from node1 to node3 OK. Schedule Task on node3

7. HPDC'08http://netsyslab.ece.ubc.ca7 What are the anticipated gains? Application-agnostic cross-layer communication Opportunistic use of custom metadata New opportunities for usage-based optimization

8. HPDC'08http://netsyslab.ece.ubc.ca8 Challenges Standardization is important Should keep the advantages of a layered design Conflicting metadata Implementation complexity and scalability Policy versus Mechanism Incentive-compatible adoption  The lesson from IP optional fields

9. HPDC'08http://netsyslab.ece.ubc.ca9 Summary and Future Work Cross-layer optimizations in storage systems with custom metadata Build the cross-layer communication mechanism  Target cluster-based file systems  e.g. FreeLoader, Lustre File System Perform an experimental evaluation

10. Questions http://netsyslab.ece.ubc.ca

11. HPDC'08http://netsyslab.ece.ubc.ca11 Use cases Application  Storage System  Custom metadata describing co-usage of files  Caching mechanism can consider file bundling Storage System  Application  Custom metadata describing file location characteristics  Application scheduler can exploit this information

12. HPDC'08http://netsyslab.ece.ubc.ca12 Design Considerations Modular and Extendable: new mechanisms and attributes. Interfaces should be stable Components (layers) should be confined