1. The Data Avalanche Jim Gray Microsoft Research Gray@Microsoft.com http://research.microsoft.com/~Gray Talk at National Youth Leadership Forum on Technology, aka nerd camp July 2004

2. Numbers TeraBytes and Gigabytes are BIG! Mega – a house in san francisco Giga – a very rich person Tera – ~ The Bush national debt Peta – more than all the money in the world A Gigabyte: the Human Genome A Terabyte: 150 mile long shelf of books.

3. Outline Historical trends imply that in 20 years: 1.we can store everything in cyberspace. The personal petabyte. 2.computers will have natural interfaces speech recognition/synthesis vision, object recognition beyond OCR Implications 1.The information avalanche will only get worse. 2.The user interface will change: less typing, more writing, talking, gesturing, more seeing and hearing 3.Organizing, summarizing, prioritizing information is a key technology. We are here Yotta Zetta Exa Peta Tera Giga Mega Kilo

4. How much information is there? Soon everything can be recorded and indexed Most bytes will never be seen by humans. Data summarization, trend detection anomaly detection are key technologies See Mike Lesk: How much information is there: http://www.lesk.com/mlesk/ksg97/ksg.html http://www.lesk.com/mlesk/ksg97/ksg.html See Lyman & Varian: How much information http://www.sims.berkeley.edu/research/projects/how-much-info/ Yotta Zetta Exa Peta Tera Giga Mega Kilo A Book.Movi e All books (words) All Books MultiMedia Everything ! Recorded A Photo 24 Yecto, 21 zepto, 18 atto, 15 femto, 12 pico, 9 nano, 6 micro, 3 milli

5. Things Have Changed IBM 305 RAMAC 10 MB disk ~1M$ (y2004 $) 1956

6. The Next 50 years will see MORE CHANGE ops/s/$ Had Three Growth Curves 1890-1990 1890-1945 Mechanical Relay 7-year doubling 1945-1985 Tube, transistor,.. 2.3 year doubling 1985-2004 Microprocessor 1.0 year doubling Combination of Hans Moravac + Larry Roberts + Gordon Bell WordSize*ops/s/sysprice

7. Constant Cost or Constant Function? 100x improvement per decade Same function 100x cheaper 100x more function for same price Mainframe Mini Workstation PDA SMPConstellationCluster SMPConstellation Graphics/storage Camera/browser Constant Price Lower Price – New Category

8. Growth Comes From NEW Apps The 10M$ computer of 1980 costs 1k$ today If we were still doing the same things, IT would be a 0 B$/y industry NEW things absorb the new capacity

9. The Surprise-Free Future in 20 years. 10,000x more power for same price –Personal supercomputer –Personal petabyte stores Same function for 10,000x less cost. –Smart dust --the penny PC? –The 10 peta-op computer (for 1,000$).

10. 10,000x would change things Human computer interface –Decent computer vision –Decent computer speech recognition –Decent computer speech synthesis Vast information stores Ability to search and abstract the stores.

11. How Good is HCI Today? Surprisingly good. –Demo of making facesmaking faces http://research.microsoft.com/research/pubs/view.aspx?pubid=290 –Demo of speech synthesis Daisy, HalDaisyHal Synthetic voice –Speech recognition is improving fast, –Vision getting better –Pen computing finally a reality. –Displays improving fast (compared to last 30 years)

12. Outline Historical trends imply that in 20 years: 1.we can store everything in cyberspace. The personal petabyte. 2.computers will have natural interfaces speech recognition/synthesis vision, object recognition beyond OCR Implications 1.The information avalanche will only get worse. 2.The user interface will change: less typing, more writing, talking, gesturing, more seeing and hearing 3.Organizing, summarizing, prioritizing information is a key technology. We are here Yotta Zetta Exa Peta Tera Giga Mega Kilo

13. How much information is there? Almost everything is recorded digitally. Most bytes are never seen by humans. Data summarization, trend detection anomaly detection are key technologies See Mike Lesk: How much information is there: http://www.lesk.com/mlesk/ksg97/ksg.html http://www.lesk.com/mlesk/ksg97/ksg.html See Lyman & Varian: How much information http://www.sims.berkeley.edu/research/projects/how-much-info/ Yotta Zetta Exa Peta Tera Giga Mega Kilo A Book.Movi e All books (words) All Books MultiMedia Everything ! Recorded A Photo

14. And >90% in Cyberspace Because: Low rent min $/byte Shrinks time now or later Shrinks space here or there Automate processing knowbots Point-to-Point OR Broadcast Immediate OR Time Delayed Locate Process Analyze Summarize

15. MyLifeBits The guinea pig Gordon Bell is digitizing his life Has now scanned virtually all: –Books written (and read when possible) –Personal documents (correspondence, memos, email, bills, legal,0…) –Photos –Posters, paintings, photo of things (artifacts, …medals, plaques) –Home movies and videos –CD collection –And, of course, all PC files Recording: phone, radio, TV, web pages… conversations Paperless throughout 2002. 12 scanned, 12 discarded. Only 30GB Excluding videos Video is 2+ TB and growing fast

16. Capture and encoding

17. I mean everything

18. 25Kday life ~ Personal Petabyte 1PB Will anyone look at web pages in 2020? Probably new modalities & media will dominate then.

19. Challenges Capture: Get the bits in Organize: Index them Manage: No worries about loss or space Curate/ Annotate: atutomate where possible Privacy: Keep safe from theft. Summarize: Give thumbnail summaries Interface: how ask/anticipate questions Present: show it in understandable ways.

20. Memex As We May Think, Vannevar Bush, 1945 A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility yet if the user inserted 5000 pages of material a day it would take him hundreds of years to fill the repository, so that he can be profligate and enter material freely

21. Too much storage? Try to fill a terabyte in a year ItemItems/TBItems/day 300 KB JPEG3 M9,800 1 MB Doc1 M2,900 1 hour 256 kb/s MP3 audio 9 K26 1 hour 1.5 Mbp/s MPEG video 2900.8 Petabyte volume has to be some form of video.

22. How Will We Find Anything? Need Queries, Indexing, Pivoting, Scalability, Backup, Replication, Online update, Set-oriented access If you dont use a DBMS, you will implement one! Simple logical structure: –Blob and link is all that is inherent –Additional properties (facets == extra tables) and methods on those tables (encapsulation) More than a file system Unifies data and meta-data SQL ++ DBMS

23. Photos

24. Searching: the most useful app? Challenge: What questions for useful results? Many ways to present answers

26. Detail view

27. Resource explorer Ancestor (collections), annotations, descendant & preview panes turned on

28. Synchronized timelines with histogram guide

29. Value of media depends on annotations Its just bits until it is annotated

30. System annotations provide base level of value Date 7/7/2000

31. Tracking usage – even better Date 7/7/2000. Opened 30 times, emailed to 10 people (its valued by the user!)

32. Get the user to say a little something is a big jump Date 7/7/2000. Opened 30 times, emailed to 10 people. BARC dim sum intern farewell Lunch

33. Getting the user to tell a story is the ultimate in media value A story is a layout in time and space Most valuable content (by selection, and by being well annotated) Stories must include links to any media they use (for future navigation/search – transclusion). Cf: MovieMaker; Creative Memories PhotoAlbums Dapeng was an intern at BARC for the summer of 2000 We took him to lunch at our favorite Dim Sum place to say farewell At table L-R: Dapeng, Gordon, Tom, Jim, Don, Vicky, Patrick, Jim

34. Value of media depends on annotations Auto-annotate whenever possible e.g. GPS cameras Make manual annotation as easy as possible. XP photo capture, voice, photos with voice, etc Support gang annotation Make stories easy Its just bits until it is annotated Dapeng was an intern at BARC for the summer of 2000 We took him to lunch at our favorite Dim Sum place to say farewell At table L-R: Dapeng, Gordon, Tom, Jim, Don, Vicky, Patrick, Jim

35. 80% of data is personal / individual. But, what about the other 20%? Business –Wall Mart online: 1PB and growing…. –Paradox: most transaction systems < 1 PB. –Have to go to image/data monitoring for big data Government –Government is the biggest business. Science –LOTS of data.

36. CERN Tier 0 Instruments: CERN – LHC Peta Bytes per Year Looking for the Higgs Particle Sensors: 1000 GB/s (1TB/s ~ 30 EB/y) Events 75 GB/s Filtered 5 GB/s Reduced 0.1 GB/s ~ 2 PB/y Data pyramid: 100GB : 1TB : 100TB : 1PB : 10PB

37. Information Avalanche Both –better observational instruments and –Better simulations are producing a data avalanche Examples –Turbulence: 100 TB simulation then mine the Information –BaBar: Grows 1TB/day 2/3 simulation Information 1/3 observational Information –CERN: LHC will generate 1GB/s 10 PB/y –VLBA (NRAO) generates 1GB/s today –NCBI: only ½ TB but doubling each year, very rich dataset. –Pixar: 100 TB/Movie Image courtesy of C. Meneveau & A. Szalay @ JHU

38. Q: Where will the Data Come From? A: Sensor Applications Earth Observation –15 PB by 2007 Medical Images & Information + Health Monitoring –Potential 1 GB/patient/y 1 EB/y Video Monitoring –~1E8 video cameras @ 1E5 MBps 10TB/s 100 EB/y filtered??? Airplane Engines –1 GB sensor data/flight, –100,000 engine hours/day –30PB/y Smart Dust: ?? EB/y http://robotics.eecs.berkeley.edu/~pister/SmartDust/ http://www-bsac.eecs.berkeley.edu/~ shollar/macro_motes/macromotes.html

39. The Big Picture Experiments & Instruments Simulations facts answers questions Data ingest Managing a petabyte Common schema How to organize it? How to reorganize it How to coexist with others Query and Vis tools Support/training Performance –Execute queries in a minute –Batch query scheduling ? The Big Problems Literature Other Archives facts

40. FTP - GREP Download (FTP and GREP) are not adequate –You can GREP 1 MB in a second –You can GREP 1 GB in a minute –You can GREP 1 TB in 2 days –You can GREP 1 PB in 3 years. Oh!, and 1PB ~3,000 disks At some point we need indices to limit search parallel data search and analysis This is where databases can help Next generation technique: Data Exploration –Bring the analysis to the data!

41. The Speed Problem Many users want to search the whole DB ad hoc queries, often combinatorial Want ~ 1 minute response Brute force (parallel search): –1 disk = 50MBps => ~1M disks/PB ~ 300M$/PB Indices (limit search, do column store) –1,000x less equipment: 1M$/PB Pre-compute answer –No one knows how do it for all questions.

42. Next-Generation Data Analysis Looking for –Needles in haystacks – the Higgs particle –Haystacks: Dark matter, Dark energy Needles are easier than haystacks Global statistics have poor scaling –Correlation functions are N 2, likelihood techniques N 3 As data and computers grow at same rate, we can only keep up with N logN A way out? –Relax notion of optimal (data is fuzzy, answers are approximate) –Dont assume infinite computational resources or memory Combination of statistics & computer science

43. Analysis and Databases Much statistical analysis deals with –Creating uniform samples – –data filtering –Assembling relevant subsets –Estimating completeness –censoring bad data –Counting and building histograms –Generating Monte-Carlo subsets –Likelihood calculations –Hypothesis testing Traditionally these are performed on files Most of these tasks are much better done inside a database Move Mohamed to the mountain, not the mountain to Mohamed.

44. Outline Historical trends imply that in 20 years: 1.we can store everything in cyberspace. The personal petabyte. 2.computers will have natural interfaces speech recognition/synthesis vision, object recognition beyond OCR Implications 1.The information avalanche will only get worse. 2.The user interface will change: less typing, more writing, talking, gesturing, more seeing and hearing 3.Organizing, summarizing, prioritizing information is a key technology. We are here Yotta Zetta Exa Peta Tera Giga Mega Kilo

45. Information Avalanche In science, industry, government,…. –better observational instruments and –and, better simulations producing a data avalanche Examples –BaBar: Grows 1TB/day 2/3 simulation Information 1/3 observational Information –CERN: LHC will generate 1GB/s.~10 PB/y –VLBA (NRAO) generates 1GB/s today –Pixar: 100 TB/Movie New emphasis on informatics: –Capturing, Organizing, Summarizing, Analyzing, Visualizing Image courtesy C. Meneveau & A. Szalay @ JHU BaBar, Stanford Space Telescope P&E Gene Sequencer From http://www.genome.uci.edu/

46. The Evolution of Science Observational Science –Scientist gathers data by direct observation –Scientist analyzes data Analytical Science –Scientist builds analytical model –Makes predictions. Computational Science –Simulate analytical model –Validate model and makes predictions Data Exploration Science Data captured by instruments Or data generated by simulator –Processed by software –Placed in a database / files –Scientist analyzes database / files

47. e-Science Data captured by instruments Or data generated by simulatorData captured by instruments Or data generated by simulator Processed by softwareProcessed by software Placed in a files or databasePlaced in a files or database Scientist analyzes files / databaseScientist analyzes files / database Virtual laboratoriesVirtual laboratories –Networks connecting e-Scientists –Strong support from funding agencies Better use of resourcesBetter use of resources –Primitive today

48. The Big Picture Experiments & Instruments Simulations facts answers questions Data ingest Managing a petabyte Common schema How to organize it? How to reorganize it How to coexist with others Query and Vis tools Support/training Performance –Execute queries in a minute –Batch query scheduling ? The Big Problems Literature Other Archives facts

49. e-Science is Data Mining There are LOTS of data –people cannot examine most of it. –Need computers to do analysis. Manual or Automatic Exploration –Manual: person suggests hypothesis, computer checks hypothesis –Automatic: Computer suggests hypothesis person evaluates significance Given an arbitrary parameter space: –Data Clusters –Points between Data Clusters –Isolated Data Clusters –Isolated Data Groups –Holes in Data Clusters –Isolated Points Nichol et al. 2001 Slide courtesy of and adapted from Robert Brunner @ CalTech.

50. Data Analysis Looking for –Needles in haystacks – the Higgs particle –Haystacks: Dark matter, Dark energy Needles are easier than haystacks Global statistics have poor scaling –Correlation functions are N 2, likelihood techniques N 3 As data and computers grow at same rate, we can only keep up with N logN A way out? –Discard notion of optimal (data is fuzzy, answers are approximate) –Dont assume infinite computational resources or memory Requires combination of statistics & computer science

51. TerraServer/TerraService http://terraService.Net/ http://terraService.Net/ US Geological Survey Photo (DOQ) & Topo (DRG) images online. On Internet since June 1998 Operated by Microsoft Corporation Cross Indexed with –Home sales, –Demographics, –Encyclopedia A web service 20 TB data source 10 M web hits/day

52. USGS Image Data Digital OrthoQuads –18 TB, 260,000 files uncompressed –Digitized aerial imagery –88% coverage conterminous US –1 meter resolution –< 10 years old Digital Raster Graphics –1 TB compressed TIFF, 65,000 files –Scanned topographic maps –100% U.S. coverage –1:24,000, 1:100,000 and 1:250,000 scale maps –Maps vary in age

53. User Interface Concept Display Imagery: 316 m 200 x 200 pixel images 7 level image pyramid Resolution 1 meter/pixel to 64 meter/pixel Navigation Tools: 1.5 m place names Click-on Coverage map Longitude and Latitude search U.S. Address Search External Geo-Spatial Links to: USGS On-line Stream Gauges Home Advisor Demographics Home Advisor Real Estate Encarta Articles Steam flow gauges Concept: User navigates an almost seamless image of earth Buttons to pan NW, N, NE, W, E, SW, S, SE Click on image to zoom in Links to switch between Topo, Imagery, and Relief data Links to Print, Download and view meta-data information

54. Terra Service New Things A popular web service –Exactly the map you want. Dynamic Map Re-projection –UTM to Geographic projection –Dynamic texture mapping? New Data –1 foot resolution natural color imagery –Census Tiger data Lights Out Management –MOM –Auto-backup / restore on drive failure

55. New Urban Area Data Redundant Bunch 1 Microsoft Campus at 4 meter resolution Ball field at.25 meter resolution resolution

56. TerraServer Becomes a Web Service TerraServer.net -> TerraService.Net TerraService.Net Web server is for people. Web Service is for programs –The end of screen scraping –No faking a URL: pass real parameters. –No parsing the answer: data formatted into your address space. Hundreds of users but a specific example: –US Department of Agriculture

57. TerraServer Web Services Get image meta-data Query TS Gazetteer Retrieve TS ImageTiles Projection conversions Web Map Client –OpenGIS like –Landmarks layered on TerraServer imagery Geo-coded data of well- known objects (points), e.g. Schools, Golf Courses, Hospitals, etc. Polygons of well-known objects (shapes), e.g. Zip Codes, Cities, etc Fat Map Client –Visual Basic / C# Windows Form –Access Web Services for all data Terra-Tile-Service Landmark-Service http://terraservice.net Sample Apps

58. Web Services Web SERVER: –Given a url + parameters –Returns a web page (often dynamic) Web SERVICE: –Given a XML document (soap msg) –Returns an XML document –Tools make this look like an RPC. F(x,y,z) returns (u, v, w) –Distributed objects for the web. –+ naming, discovery, security,.. Internet-scale distributed computing Your program Data In your address space Web Service soap object in xml Your program Web Server http Web page

59. KVM / IP TerraServer Hardware Storage Bricks –White-box commodity servers –4tb raw / 2TB Raid1 SATA storage –Dual Hyper-threaded Xeon 2.4ghz, 4GB RAM Partitioned Databases (PACS – partitioned array) –3 Storage Bricks = 1 TerraServer data –Data partitioned across 20 databases –More data & partitions coming Low Cost Availability –4 copies of the data RAID1 SATA Mirroring 2 redundant Bunches –Spare brick to repair failed brick 2N+1 design –Web Application bunch aware Load balances between redundant databases Fails over to surviving database on failure ~100K$ capital expense.

60. Research Objectives Public: Access to remote sensing data with no GIS expertise required Ubiquitous: No special hw/sw required by client Delivery: All OnLine/Internet Based, no tape or CD distribution Simple: Designed to be used by a 6 th grade geography student Test/show scalability Test/show availability: Test/show lights out: –all operations & maintenance occurs remotely –Minimal ops and dev staff web service poster child User/App Goals Technology Goals

61. Virtual Observatory http://www.astro.caltech.edu/nvoconf/ http://www.voforum.org/ http://www.astro.caltech.edu/nvoconf/ http://www.voforum.org/ Premise: Most data is (or could be online) So, the Internet is the worlds best telescope: –It has data on every part of the sky –In every measured spectral band: optical, x-ray, radio.. –As deep as the best instruments (2 years ago). –It is up when you are up. The seeing is always great (no working at night, no clouds no moons no..). –Its a smart telescope: links objects and data to literature on them.

62. Why Astronomy Data? It has no commercial value –No privacy concerns –Can freely share results with others –Great for experimenting with algorithms It is real and well documented –High-dimensional data (with confidence intervals) –Spatial data –Temporal data Many different instruments from many different places and many different times Federation is a goal The questions are interesting –How did the universe form? There is a lot of it (petabytes) IRAS 100 ROSAT ~keV DSS Optical 2MASS 2 IRAS 25 NVSS 20cm WENSS 92cm GB 6cm

63. Time and Spectral Dimensions The Multiwavelength Crab Nebulae X-ray, optical, infrared, and radio views of the nearby Crab Nebula, which is now in a state of chaotic expansion after a supernova explosion first sighted in 1054 A.D. by Chinese Astronomers. Slide courtesy of Robert Brunner @ CalTech. Crab star 1053 AD

64. SkyServer.SDSS.org A modern archive –Raw Pixel data lives in file servers –Catalog data (derived objects) lives in Database –Online query to any and all Also used for education –150 hours of online Astronomy –Implicitly teaches data analysis Interesting things –Spatial data search –Client query interface via Java Applet –Query interface via Emacs –Popular -- 1% of Terraserver –Cloned by other surveys (a template design) –Web services are core of it.

65. Demo of SkyServer Shows standard web server Pixel/image data Point and click Explore one object Explore sets of objects (data mining)

66. Federation Data Federations of Web Services Massive datasets live near their owners: –Near the instruments software pipeline –Near the applications –Near data knowledge and curation –Super Computer centers become Super Data Centers Each Archive publishes a web service –Schema: documents the data –Methods on objects (queries) Scientists get personalized extracts Uniform access to multiple Archives –A common global schema

67. Federation: SkyQuery.NetSkyQuery.Net Combine 4 archives initially Just added 10 more Send query to portal, portal joins data from archives. Problem: want to do multi-step data analysis (not just single query). Solution: Allow personal databases on portal Problem: some queries are monsters Solution: batch schedule on portal server, Deposits answer in personal database.

68. 2MASS INT SDSS FIRST SkyQuery Portal Image Cutout SkyQuery Structure Each SkyNode publishes –Schema Web Service –Database Web Service Portal is –Plans Query (2 phase) –Integrates answers –Is itself a web service

69. SkyQuery: http://skyquery.net/ http://skyquery.net/ Distributed Query tool using a set of web services Four astronomy archives from Pasadena, Chicago, Baltimore, Cambridge (England). Feasibility study, built in 6 weeks –Tanu Malik (JHU CS grad student) –Tamas Budavari (JHU astro postdoc) –With help from Szalay, Thakar, Gray Implemented in C# and.NET Allows queries like: SELECT o.objId, o.r, o.type, t.objId FROM SDSS:PhotoPrimary o, TWOMASS:PhotoPrimary t WHERE XMATCH(o,t)<3.5 AND AREA(181.3,-0.76,6.5) AND o.type=3 and (o.I - t.m_j)>2

70. SkyNode Basic Web Services Metadata information about resources –Waveband –Sky coverage –Translation of names to universal dictionary (UCD) Simple search patterns on the resources –Cone Search –Image mosaic –Unit conversions Simple filtering, counting, histogramming On-the-fly recalibrations

71. Portals: Higher Level Services Built on Atomic Services Perform more complex tasks Examples –Automated resource discovery –Cross-identifications –Photometric redshifts –Outlier detections –Visualization facilities Goal: –Build custom portals in days from existing building blocks (like today in IRAF or IDL)

72. 2MASS INT SDSS FIRST SkyQuery Portal Image Cutout MyDB added to SkyQuery Let users add personal DB 1GB for now. Use it as a workbook. Online and batch queries. Moves analysis to the data Users can cooperate (share MyDB) Still exploring this MyDB

73. The Big Picture Experiments & Instruments Simulations facts answers questions Data ingest Managing a petabyte Common schema How to organize it? How to reorganize it How to coexist with others Query and Vis tools Support/training Performance –Execute queries in a minute –Batch query scheduling ? The Big Problems Literature Other Archives facts