1. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 1 The Rebirth of Database Machines Dina Bitton Jim Gray

2. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 2 Outline Active Disks are coming Disk Tutorial (not presented, but slides in deck) Disk Arms are important (optimize them) The Rebirth of Database Machines

3. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 3 Disks of 30 Years Ago 10 MB Failed every few weeks Cost more than 400$

4. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 4 Disk Arrays 24 cpus 384 disks More mips in the disks than in the cpus

5. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 5 Year 2003 Disks Big disk (10 $/GB) –3 –200 GB –150 kaps (k accesses per second) –30 MBps sequential Small disk (20 $/GB) –2 –40 GB –100 kaps –20 MBps sequential Both running DBMS, Mail, Web, and OS

6. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 6 From CMU Active Disk web site http://www.pdl.cs.cmu.edu/Active/

7. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 7 Research Problem: When every disk is a super-computer… And there are thousands of them... Who manages data placement? Query plans among 1,000 severs? How does –mirroring work? –backup work? Where does my program run?

8. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 8 Relevant University Research on Active Disks Kim Keeton & Dave Patterson @ UC Berkeley http://www.cs.berkeley.edu/~pattrsn/talks/sigmod98-keynote.ppt Erik Riedel & Garth Gibson @ CMU http://www.pdl.cs.cmu.edu/Active/ Mike Franklin @ U Maryland http://www.cs.umd.edu/projects/bdisk Anurag Acharya, Mustafa Uysal @ UC SB http://www.cs.ucsb.edu/TRs/TRCS98-06.html

9. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 9 Outline Active Disks are coming Disk Tutorial (not presented, but slides in deck) Disk Arms are important The Rebirth of Database Machines

10. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 10 Two Basic Shapes Circle (disk) –storage frequently returns to same spot –so less total surface area Line (tape) –Lots more area, –Longer time to get to the data. Key idea: multiplex expensive read/write head over large storage area: trade high GB/$ for lower access/second

11. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 11 Disk Terms Disks are called platters Data is recorded on tracks (circles) on the disk. Tracks are formatted into fixed-sized sectors. A pair of Read/Write heads for each platter Mounted on a disk arm Client addresses logical blocks (cylinder, head, sector) Disk arm Seeks to a cylinder, selects a head waits for data to rotate under head transfers data to or from Bad blocks are remapped to spare good blocks.

12. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 12 Disk Access Time Access time = SeekTime 6 ms + RotateTime 3 ms + ReadTime1 ms Rotate time: –5,000 to 10,000 rpm ~ 12 to 6 milliseconds per rotation ~ 6 to 3 ms rotational latency

13. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 13 Disk Access Time Improves Slowly Access time = SeekTime 6 ms 8%/y + RotateTime 3 ms 8%/y + ReadTime 1 ms 40%/y Other useful facts: –Power rises more than size 3 (small is indeed beautiful) –Small devices are more rugged –Small devices can use plastics (forces are much smaller) e.g. bugs fall without breaking anything

14. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 14 Disk Seek Time Seek time is ~ Sqrt(distance) (distance = 1/2 acceleration x time 2 ) Specs assume seek is 1/3 of disk Short seeks are common. (over 50% are zero length) Typical 1/3 seek time: 6 ms 4x improvement in 20 years. Full Accelerate Full Stop speed time

15. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 15 Read/Write Time: Density Time = Size / BytesPerSecond Bytes/Second = Speed * LinearDensity –5 to 15 MBps MAD (Magnetic Aerial Density) –Today 3 Gbits/inch 2 11 gbpsi in lab –Rising > 60%/year –ParaMagnetic Limit: ??? ~ 60 Gb/inch 2 –linear density is sqrt 10x per decade 1970 1980 1990 2000 10,000 1,000 100 10 1 Hoaglands Law MAD (Mbpsi)

16. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 16 Read/Write Time: Rotational Speed Bytes/Second = Speed * Density Speed greater at edge of circle Speed 3600 -> 10,000 rpm –5%/year improvement bit rate varies by ~1.5x today r 2 = 1 r 2 = 4 r = 1 r = 2

17. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 17 Read/Write Time: Zones Disks are sectored –typical: 512 bytes/sector –Sector is read/write unit –Failfast: can detect bad sectors. Disks are zoned –outer zones have more sectors –Bytes/second higher in outer zones. 14 sectors/track 8 sectors/track

18. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 18 Seek Rotate Transfer Seek Rotate Transfer Wait The Access Time Myth The Myth: seek time dominates The Reality: (1) Queuing dominates (2) Transfer dominates BLOB (3) Disk seeks often short Implication: many cheap servers better than one fast expensive server –shorter queues –parallel transfer –lower cost/access and cost/byte This is now obvious for disk arrays This will be obvious for tape arrays

19. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 19 Storage Ratios Changed 10x better access time 10x more bandwidth 4,000x lower media price DRAM/disk media price ratio changed –1970-1990100:1 –1990-1995 10:1 –1995-1997 50:1 –today ~.2$pMB disk 10$pMB dram

20. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 20 Disk Access Ratios Have Changed Key metrics: $/GB Kaps/GB (KB accesses per second per GB) SCAN: time to scan the disk Scan going from minutes to days Disk arms are precious resource (disk capacity is no longer the precious resource) Kaps/GB went from 500 to 7 and going to 1

21. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 21 Stripe For More Bandwidth N-stores have N-times the bandwidth Works great! Supported by most file systems

22. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 22 Mirrors: Replicate Stores for Availability Read one, write all If one fails, rebuild from survivor Run scrubber in background to fix faults N-replicas can give N-times the bandwidth UnAvailabity ~ A Million Years!!!

23. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 23 RAID5: Parity Saves Storage Space Mirrors: 50% storage overhead –read one, write both RAID5: 12% Storage overhead: –read one, write one plus parity PARITY

24. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 24 Interesting Fact: Mirrored Disks Optimize Disk Arms seek is min seekDoubles read bandwidth Sequential: Read stagger reads from each drive (stripe) Random: Read closest arm seek is min seek. Doubles write cost (write both) seek is max seek –Write time increases because seek is max seek.

25. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 25 If Mix Reads & Writes Mirror is Better Than Partition 2 servers are better than one Benefit is better than 2x write cost if reads writes

26. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 26 What if you have LOTS of Disks When you have BIG disks (200 GB), arms are precious, space is cheap. If you replicate 1000x –write seek time asymptotically approaches 1.7x –read seek time asymptotically approaches zero.

27. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 27 Outline Active Disks are coming Disk Tutorial (not presented, but slides in deck) Disk Arms are important The Rebirth of Database Machines

28. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 28 The Rebirth of Database Machines Dina BittonJim Gray IDSMicrosoft

29. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 29 Outline Performance hungry databases History: life and death of database machines What has changed that can make database machines work today Shared-Nothing Database Machine Where is the required bandwidth DMP : Shared-Nothing & Shared- Everything

30. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 30 Demand for Database Performance Larger Databases: –marketing data warehouses: TB of historical data –daily news broadcasts: 1 TB of searchable video/audio data Large Scans: Searches require access to large fraction of database Repeated Scans: DSS queries, Data mining algorithms

31. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 31 Life, Death & Reincarnation Database Machines are coming, Database Machines are coming... (Hsiao 1979) Then there was Britton-Lee, Direct, ICL … –Teradata builds highly-parallel shared-nothing SQL server –many university paper designs Database Machines, An Idea whose time has Passed? (Boral- DeWitt 1983) Then there was MMDBs, Grace, Gamma and more Teradata Then there was Software (Parallel Database Query) Next: PDQ + lots of disks with power controllers

32. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 32 And All Along Stonebrakers Opinion: The history of DBMS research is littered with innumerable proposals to construct hardware database machines to provide high performance operations. In general these have been proposed by hardware types with a clever solution in searchof a problem on which it might work. Readings in Database Systems, Morgan-Kaufmann

33. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 33 Why Not then, but Yes now Too early: small databases on 1 disk TB databases span thousands disks, need partitioning Disk filter designs: addressed only small part of DBMS requirements disk controllers are fast computers Exotic technologies (bubbles, CCD…) went away Special purpose hardware increased design time and cost Higher level of integration, VLSI design tools better Parallel query processing was not well-understood Large body of research, successful commercial implementations

34. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 34 Parallel Query Processing [DeWitt-Gray CACM91 ] Pipelining data streams flow from one operator to the next Partitioning tables are partitioned to allow concurrent processing on partitions

35. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 35 Data Pathway Contention [Patterson Sigmod 1998] Disk external I/O bus bottleneck to transfer rate, cost Network internal I/O bus interface is bottleneck to delivered bandwidth Memory-Processor processor-memory interface (cache+memory bus) is bottleneck to delivered bandwidth

36. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 36 Processor & Memory Processor & Memory Processor & Memory Processor & Memory... Scalable Interconnect A Shared-Nothing Database Machine No contention in memory access or parallel disk access => Embarrassingly Parallel Scan [Patterson] But: how fast need Interconnect be? Each processor has own OS, communication protocols,DB instance Exchange data streams for pipelining ops, for sort, merge Cant support M:N mapping between disks & threads

37. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 37 Share-Everything? Need more bandwidth for shipping data streams than network can provide Need M:N mapping from disks to processors for sort/merge Control & synchronization: Data-flow best to synchronize processors

38. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 38 Where to Get the Bandwidth?

39. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 39 Direct connection BAM RAM P 4 I/O interface adapter Bus adapter DMP BOARD To Host ComputerTo other DMP Boards via high-speed switch Massive Parallel Operation data-flow control M:N thread-to-disk RFM... P 1 NP 1NP 2NP 16 Direct processor to disk access Direct disk to memory connect 180... The Data Manipulation Platform

40. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 40 Scan tabX 1 Scan tabX 2 Scan tabX 32 Scan tabY 3 Scan tabY 1... Exchange 1 Exchange 2 HJoin Exchange 3 Exchange 4 Group 1Group 2 Sort 1 Sort 2 Exchange 5 1 23213 Select sum(tabX.amount*.08), tabY.region from tabX,tabY where tabX.key=tabY.region group by tabY.region, order by tabY.region; A DSS Query Execution Plan... Database Disks Temp Disks 1/3 selected 1/10 joined 1/10 grouped

41. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 41 Scan tabX 1 Scan tabX 2 Scan tabX 32 Scan tabY 3 Scan tabY 1... Exchange 1 Exchange 2 HJoin Exchange 3 Exchange 4 Group 1Group 2 Sort 1 Sort 2 Exchange 5 1 23213 Bandwidth Requirements... Database Disks Temp Disk Contentio n 32*20MB/s= 640 MB/s 2.1 MB/s 21 MB/s 210 MB/s

42. Bitton & Gray: The Rebirth of Database Machines, http://research.microsoft.com/~Gray/talks/DB_Machine_Rebirth.ppt 42 Conclusion DMP: shared-nothing and shared-everything IT ISNT THAT YOU CANT SHARE IT IS WHERE YOU SHARE ON A CHIP ON A BOARD ON A NETWORK