1. Alok Choudhary Dept. of Electrical & Computer Engineering And Kellogg School of Management Northwestern University I/O and Storage: Challenges Moving Forward

2. 3/15/2004@ANC2 I/O for a Petaflop system!  Want 1 Terabyte/sec? Please buy 20,000 disks, each with 50MB/sec. Come up with $50m just for I/O  Want to spin all of them fully? Buy a 20 MW power plant  Want to stop computing for a second for I/O? Lose 1,000,000,000,000,000 operations  Pray that disks don’t fail while doing something – or double the disks  Want to Lock and Synchronize so you can write small fragment Are you joking?

3. 3/15/2004@ANC3 Emerging Storage Hierarchy Exabytes Terabytes Petabytes ≈ 1 µs ≈ seconds ≈ 2–10 µs ≈ 1–30 ms ≈ 250 ms Main Memory MEMs Low Power DRAM High Performance Disk Holographic Memory Massive Arrays of Idle Disks Tape Silos Want to manage I/O – good luck!

4. 3/15/2004@ANC4

5. 3/15/2004@ANC5 Some Complexity Dimensions

6. 3/15/2004@ANC6 A Typical System? Ack: Gary Grider (LANL)

7. 3/15/2004@ANC7  Proactive – not reactive  Self Learning  Self managing  Autonomic  Active  “What not how?” For Performance, power, load balancing, scalability  How do we get there? Next 5 years of R&D by our team I/O System HW/SW moving forward

8. 3/15/2004@ANC8 E.g., extend OSD concept for large-scale parallelism

9. 3/15/2004@ANC9 Usage Pattern  Immediate Use On-line analysis, Visualization, Data reduction Local parallel access model  Future Use Relationships, subsets, affinities Analysis types, use of results Is there a way to specify, storage, use this information for effective management and optimizations

10. 3/15/2004@ANC10 What we need to do moving forward  Immediate Use On-line analysis, Visualization, Data reduction Local parallel access model  Future Use Relationships, subsets, affinities Analysis types, use of results Is there a way to specify, storage, use this information for effective management and optimizations

11. 3/15/2004@ANC11 Metadata (who defines and manages?)  Application Level Date, run-time parameters, execution environment, comments, result summary, etc.  Program Level Data types, structures Association of multiple datasets and files File location, file structures (single/multiple datasets multiple/single file)  Performance Level I/O functions (eg. Collective/non-collective I/O parameters) Access hints, access pattern, storage pattern, dataset associations Striping, pooled striping, storage association Prefetching, staging, migration, caching hints Historical performance

12. 3/15/2004@ANC12 Passive/Active Storage  Passive Storage Traditional model of storage Store/retrieve files/data sets from storage  Active Storage Storage and software provides  Functions (user defined, e.g., analysis, compression)  Fail-in-place capability (active storage can reorganize, remap or replicate data upon failures)  Transparency of management a big plus for users Promising for large-scale usage Performance capability available in storage processors