1. © Copyright 2004 Instrumental, Inc I/O Types and Usage in DoD Henry Newman Instrumental, Inc/DOD HPCMP/DARPA HPCS May 24, 2004

2. © Copyright 2004 Instrumental, Inc Evaluation Issues What are the scaling problems?

3. © Copyright 2004 Instrumental, Inc Facts About Performance(1) System Feature19772004 System CPU PerformanceCDC 7600 25 MFLOPSEarth Simulator 40 TFLOPS Disk TechnologyCDC Cyber 819 3600 RPMs Seagate Cheetah 15K RPMs Disk Density80 MB146 GB Disk Transfer Rate3 MB/sec Half Duplex71.5 MB/sec Avg. per disk 200 MB/sec full duplex RAID Disk Seek+Latency24 ms6.0 ms write 5.6 ms read

4. © Copyright 2004 Instrumental, Inc Facts About Performance(2) ItemTimes Increase CPU1.6M RPMS4.1 Density1814 Transfer Rate disk23.8 Transfer Rate RAID133 Seek+Latency Read4.3

5. © Copyright 2004 Instrumental, Inc Device Utilization

6. © Copyright 2004 Instrumental, Inc Tape Facts Vendor DriveMediaYear Introduced Capacity MBPeak Transfer Rate MB/sec uncompressed Performance Increase IBM 3420Reel-to-Reel19741501.25 1.00 IBM 3480 19842003 2.40 IBM 3490348019892003 2.40 IBM 3490E348019914004.5 3.60 IBM 3490E 19928004.5 3.60 IBM 34903490E19958006 4.80 IBM 3590 1995100009 7.20 StorageTek SD-3 19955000011 8.80 StorageTek T9840A984019982000010 8.00 IBM 3590E 19992000014 11.20 IBM 3950E3590E20004000014 11.20 StorageTek T9940A994020006000014 11.20 LTO 200010000014 11.20 Sony GY-8240FC DTF-22000 200GB/60GB *** 24 19.20 StorageTek T9840B 9480 200140000020 16.00 StorageTek T9940B 9940 2002200000030 24.00 IBM 3590H3950200260000014 11.20 LTO LTO-IILTO2003200000035 28.00

7. © Copyright 2004 Instrumental, Inc File System Concerns Data fragmentation and allocation Metadata fragmentation and allocation Recovery from crash or metadata loss Performance that scales Support for >2TB LUNs Failover

8. © Copyright 2004 Instrumental, Inc Fragmentation Fragmentation is becoming a performance problem as file systems grow No major technology enhancements have been seen in decades 4 Object Storage Device (OSD new T10 spec) will change this Fragmentation of metadata can have dramatic impact on performance 4 Recently observed 600x slowdown in access at a site

9. © Copyright 2004 Instrumental, Inc USG Types Requirements What is DoD currently using?

10. © Copyright 2004 Instrumental, Inc Current Types of Requirements Database 4 Used by most sites, big and small, for data reference especially in the intelligence community 4 Not used by MSRCs much Real-time data capture 4 Requirement in intelligence community Application 4 Homogeneous shared file system access

11. © Copyright 2004 Instrumental, Inc Current Types of Requirements Archival 4 Used by MSRCs 4 Intelligence community Process Flow 4 Used after real-time capture 4 Could be used by MSRCs if shared file system between HPC and HSM systems were implemented

12. © Copyright 2004 Instrumental, Inc Database 4, 8 or 16 KB I/Os for indexes 4 Random 64 KB I/Os for log updates 4 Sequential Read and write Up to 256 KB Just about everyone uses a database somewhere in their HPC systems 4 Although some don’t have performance requirements

13. © Copyright 2004 Instrumental, Inc Real-Time Data Capture Large Block Requirement 4 4 MB-128 MB I/O requests Small Block Requirement 4 1 KB-8 KB files with millions of files per day Multiple Threads 4 4-8 threads to keep the devices busy with either type of I/O

14. © Copyright 2004 Instrumental, Inc Real-Time Data Capture Generally requires an HSM 4 Usually needs 100’s of MB/sec 7x24 to meet the requirements for capture Everything must run at rate 4 I/O Bus 4 RAID devices 4 Switches 4 Limitations of tape bandwidth are pushed 4 HBAs

15. © Copyright 2004 Instrumental, Inc Application Homogeneous Shared File System access 4 Must be able to get the data from the nodes to a single file over fibre channel High performance I/O from those nodes 4 Depends on the application but given that GPFS peek is about 400 MB/sec that seems to be the current requirement Support for a few 100,000 files 4 No where near the HSM requirements

16. © Copyright 2004 Instrumental, Inc Archival Large HSM Systems 4 MSRCs are a good example 4 High speed networks 4 TCP/IP (ftp) data movement Future movement to shared file systems which will make these look more like real-time capture requirements

17. © Copyright 2004 Instrumental, Inc Process Flow These are applications and processes that are done via an assembly line like concept 4 Each step uses a machine or machines, sometimes specialized, to move the task along 4 Data communication via a shared file system with multi- threaded large block I/O requests from each of the hosts to various data sets

18. © Copyright 2004 Instrumental, Inc Current MSRC Requirements Homogeneous shared file system for applications running on the HPC system HSM support and access via TCP/IP Process Flow should be supported for visualization Support for database but no performance requirement

19. © Copyright 2004 Instrumental, Inc Conclusion The future for HPCS machines and most application environments will be shared file systems Shared file systems were pioneered for real-time capture world Large file systems are seeing problems with fragmentation and scaling