1 Store Everything Online In A Database Jim Gray Microsoft Research

2 Outline Store Everything Online (Disk not Tape) In a Database

3 How Much is Everything? Soon everything can be recorded and indexed Most bytes will never be seen by humans. Data summarization, trend detection anomaly detection are key technologies See Mike Lesk: How much information is there: See Lyman & Varian: How much information Yotta Zetta Exa Peta Tera Giga Mega Kilo A Book.Movi e All LoC books (words) All Books MultiMedia Everything ! Recorded A Photo 24 Yecto, 21 zepto, 18 atto, 15 femto, 12 pico, 9 nano, 6 micro, 3 milli

4 Storage capacity beating Moores law 3 k$/TB today (raw disk) 1k$/TB by end of 2002

5 Outline Store Everything Online (Disk not Tape) In a Database

6 Online Data Can build 1PB of NAS disk for 5M$ today Can SCAN ( read or write ) entire PB in 3 hours. Operate it as a data pump: continuous sequential scan Can deliver 1PB for 1M$ over Internet –Access charge is 300$/Mbps bulk rate Need to Geoplex data (store it in two places). Need to filter/process data near the source, –To minimize network costs.

7 The Absurd Disk 2.5 hr scan time (poor sequential access) 1 access per second / 5 GB (VERY cold data) Its a tape! 1 TB 100 MB/s 200 Kaps

8 Disk vs Tape Disk –80 GB –35 MBps – 5 ms seek time – 3 ms rotate latency – 3$/GB for drive 2$/GB for ctlrs/cabinet –15 TB/rack –1 hour scan Tape –40 GB –10 MBps –10 sec pick time – second seek time –2$/GB for media 8$/GB for drive+library –10 TB/rack –1 week scan The price advantage of disk is growing the performance advantage of disk is huge! At 10K$/TB, disk is competitive with nearline tape. Guestimates Cern: 200 TB 3480 tapes 2 col = 50GB Rack = 1 TB =12 drives

9 Building a Petabyte Disk Store Cadillac ~ 500k$/TB = 500M$/PB plus FC switches plus…800M$/PB TPC-C SANs (Brand PC 18GB/…) 60 M$/PB Brand PC local SCSI 20M$/PB Do it yourself ATA 5M$/PB

10 Cheap Storage and/or Balanced System Low cost storage (2 x 3k$ servers) 5K$ TB 2x ( 800 Mhz, 256Mb + 8x80GB disks + 100MbE) raid5 costs 6K$/TB Balanced server (5k$/.64 TB) –2x800Mhz (2k$) –512 MB –8 x 80 GB drives (2K$) –Gbps Ethernet + switch (300$/port) –9k$/TB 18K$/mirrored TB 2x800 Mhz 512 MB

11 Next step in the Evolution Disks become supercomputers –Controller will have 1bips, 1 GB ram, 1 GBps net –And a disk arm. Disks will run full-blown app/web/db/os stack Distributed computing Processors migrate to transducers.

12 Its Hard to Archive a Petabyte It takes a LONG time to restore it. At 1GBps it takes 12 days! Store it in two (or more) places online (on disk?). A geo-plex Scrub it continuously (look for errors) On failure, –use other copy until failure repaired, –refresh lost copy from safe copy. Can organize the two copies differently (e.g.: one by time, one by space)

13 Outline Store Everything Online (Disk not Tape) In a Database

14 Why Not file = object + GREP ? It works if you have thousands of objects (and you know them all) But hard to search millions/billions/trillions with GREP Hard to put all attributes in file name. –Minimal metadata Hard to do chunking right. Hard to pivot on space/time/version/attributes.

15 The Reality: its build vs buy If you use a file system you will eventually build a database system : –metadata, –Query, –parallel ops, – security,…. –reorganize, –recovery, –distributed, –replication,

16 OK: so Ill put lots of objects in a file Do It Yourself Database Good news: –Your implementation will be 10x faster than the general purpose one easier to understand and use than the general purpose on. Bad news: –It will cost 10x more to build and maintain –Someday you will get bored maintaining/evolving it –It will lack some killer features: Parallel search Self-describing via metadata SQL, XML, … Replication Online update – reorganization Chunking is problematic (what granularity, how to aggregate)

17 Top 10 reasons to put Everything in a DB 1.Someone else writes the million lines of code 2.Captures data and Metadata, 3.Standard interfaces give tools and quick learning 4.Allows Schema Evolution without breaking old apps 5.Index and Pivot on multiple attributes space-time-attribute-version…. 6.Parallel terabyte searches in seconds or minutes 7.Moves processing & search close to the disk arm (moves fewer bytes (qestons return datons). 8.Chunking is easier (can aggregate chunks at server). 9.Automatic geo-replication 10.Online update and reorganization. 11.Security 12.If you pick the right vendor, ten years from now, there will be software that can read the data.

18 DB Centric Examples TerraServer –All images and all data in the database (chunked as small tiles). – SkyServer & Virtual Sky –Both image and semantic data in a relational store. –Parallel search & NonProcedural access are important. – – – 45s&T=4&P=12&S=10&X=5096&Y=4121&W=4&Z=- 1&tile.2.1.x=55&tile.2.1.y=20http://virtualsky.org/servlet/Page?F=3&RA=16h+10m+1.0s&DE=%2B0d+42m+ 45s&T=4&P=12&S=10&X=5096&Y=4121&W=4&Z=- 1&tile.2.1.x=55&tile.2.1.y=20

19 Outline Store Everything Online (Disk not Tape) In a Database