1. 1 Managing Data for the World Wide Telescope aka: The Virtual Observatory Jim Gray Alex Szalay SLAC Data Management Workshop

2. 2 The Evolution of Science Observational Science –Scientist gathers data by direct observation –Scientist analyzes data Analytical Science –Scientist builds analytical model –Makes predictions. Computational Science –Simulate analytical model –Validate model and makes predictions Data Exploration Science Data captured by instruments Or data generated by simulator –Processed by software –Placed in a database / files –Scientist analyzes database / files

3. 3 Information Avalanche In science, industry, government,…. –better observational instruments and –and, better simulations producing a data avalanche Examples –BaBar: Grows 1TB/day 2/3 simulation Information 1/3 observational Information –CERN: LHC will generate 1GB/s.~10 PB/y –VLBA (NRAO) generates 1GB/s today –Pixar: 100 TB/Movie New emphasis on informatics: –Capturing, Organizing, Summarizing, Analyzing, Visualizing Image courtesy C. Meneveau & A. Szalay @ JHU BaBar, Stanford Space Telescope P&E Gene Sequencer From http://www.genome.uci.edu/

4. 4 The Big Picture Experiments & Instruments Simulations facts answers questions Data ingest Managing a petabyte Common schema How to organize it? How to reorganize it How to coexist with others Query and Vis tools Support/training Performance –Execute queries in a minute –Batch query scheduling ? The Big Problems Literature Other Archives facts

5. 5 FTP - GREP Download (FTP and GREP) are not adequate –You can GREP 1 MB in a second –You can GREP 1 GB in a minute –You can GREP 1 TB in 2 days –You can GREP 1 PB in 3 years. Oh!, and 1PB ~3,000 disks At some point we need indices to limit search parallel data search and analysis This is where databases can help Next generation technique: Data Exploration –Bring the analysis to the data!

6. 6 The Speed Problem Many users want to search the whole DB ad hoc queries, often combinatorial Want ~ 1 minute response Brute force (parallel search): –1 disk = 50MBps => ~1M disks/PB ~ 300M$/PB Indices (limit search, do column store) –1,000x less equipment: 1M$/PB Pre-compute answer –No one knows how do it for all questions.

7. 7 Next-Generation Data Analysis Looking for –Needles in haystacks – the Higgs particle –Haystacks: Dark matter, Dark energy Needles are easier than haystacks Global statistics have poor scaling –Correlation functions are N 2, likelihood techniques N 3 As data and computers grow at same rate, we can only keep up with N logN A way out? –Relax notion of optimal (data is fuzzy, answers are approximate) –Don’t assume infinite computational resources or memory Combination of statistics & computer science

8. 8 Analysis and Databases Much statistical analysis deals with –Creating uniform samples – –data filtering –Assembling relevant subsets –Estimating completeness –censoring bad data –Counting and building histograms –Generating Monte-Carlo subsets –Likelihood calculations –Hypothesis testing Traditionally these are performed on files Most of these tasks are much better done inside a database Move Mohamed to the mountain, not the mountain to Mohamed.

9. 9 Organization & Algorithms Use of clever data structures (trees, cubes): –Up-front creation cost, but only N logN access cost –Large speedup during the analysis –Tree-codes for correlations (A. Moore et al 2001) –Data Cubes for OLAP (all vendors) Fast, approximate heuristic algorithms –No need to be more accurate than cosmic variance –Fast CMB analysis by Szapudi et al (2001) N logN instead of N 3 => 1 day instead of 10 million years Take cost of computation into account –Controlled level of accuracy –Best result in a given time, given our computing resources

10. 10 World Wide Telescope Virtual Observatory http://www.ivoa.net/ http://www.ivoa.net/ Premise: Most data is (or could be online) The Internet is the world’s best telescope: –It has data on every part of the sky –In every measured spectral band: optical, x-ray, radio.. –As deep as the best instruments (2 years ago). –It is up when you are up. The “seeing” is always great (no working at night, no clouds no moons no..). –It’s a smart telescope: links objects and data to literature on them.

11. 11 Why Astronomy? Community has lots of data Data is real and well documented – High-dimensional (with confidence intervals) – Spatial, temporal Diverse and distributed – Many different instruments from many different places and many different times Community wants to share/cross compare –Can freely share data and algorithms. –“DataMining, Not Data MINE!!” Mark Ellisman, UCSD They are well organized Community is small and homogeneous No commercial or privacy concerns –All the problems are technical or social.

12. 12 The WWT Components Data Sources –Literature –Archives Unified Definitions –Units, –Semantics/Concepts/Metrics, Representations, –Provenance Object model Classes and methods Portals

13. 13 Data Sources Literature online and cross indexed –Simbad, ADS, NED, http://simbad.u-strasbg.fr/Simbad, http://adswww.harvard.edu/, http://nedwww.ipac.caltech.edu/ http://simbad.u-strasbg.fr/Simbadhttp://adswww.harvard.edu/http://nedwww.ipac.caltech.edu/ Many curated archives online –FIRST, DPOSS, 2MASS, USNO, IRAS, SDSS, VizeR,… –Typically files with English meta-data and some programs Groups, Researchers, Amateurs Publish –Datasets online in various formats –Data publications are ephemeral (may disappear) –Many have unknown provenance Documentation varies; some good and some none.

14. 14 Unified Definitions Universal Content Definitions http://vizier.u-strasbg.fr/doc/UCD.htx http://vizier.u-strasbg.fr/doc/UCD.htx –Collated all table heads from all the literature –100,000 terms reduced to ~1,500 –Rough consensus that this is the right thing. –Refinement in progress as people use UCDs Defines –Units: gram, radian, second, janski... –Semantic Concepts / Metrics Std error, Chi 2 fit, magnitude, flux @ passband, velocity,

15. 15 Provenance Most data will be derived. To do science, need to trace derived data back to source. So programs and inputs must be registered. Must be able to re-run them. Example: Space Telescope Calibrated Data –Run on demand –Can specify software version (to get old answers) Scientific Data Provenance and Curation are largely unsolved problems (some ideas but no science).

16. 16 Object Model General acceptance of XML Recent acceptance of XML Schema (XSD over DTD) Wait-and-See about SOAP/WSDL/… –“ Web Services are just Corba with angle brackets.” –FTP is good enough for me. Personal opinion: –Web Services are much more than “Corba + <>” –Huge focus on interop –Huge focus on integrated tools But the community says “Show me!” –Many technologists convinced, but not yet the astronomers Your program Data In your address space Web Service soap object in xml Your program Web Server http Web page

17. 17 Classes and Methods First Class: VO table http://www.us-vo.org/VOTable/ http://www.us-vo.org/VOTable/ –Represents an answer set in XML Defined by an XML Schema (XSD) Metadata (in terms of UCDs) Data representation (numbers and text) –First method Cone Search: Get objects in this cone http://voservices.org/cone/ http://voservices.org/cone/ Your program Data In your address space Web Service soap object in xml

18. 18 Other Classes Space-Time class –http://hea-www.harvard.edu/~arots/nvometa/STCdoc.pdfhttp://hea-www.harvard.edu/~arots/nvometa/STCdoc.pdf Image Class (returns pixels) –SdssCutout –Simple Image Access Protocol http://bill.cacr.caltech.edu/cfdocs/usvo-pubs/files/ACF8DE.pdf http://bill.cacr.caltech.edu/cfdocs/usvo-pubs/files/ACF8DE.pdf –HyperAtlas http://bill.cacr.caltech.edu/usvo-pubs/files/hyperatlas.pdf http://bill.cacr.caltech.edu/usvo-pubs/files/hyperatlas.pdf Spectral –Simple Spectral Access Protocol –500K spectra available at http://voservices.net/wavehttp://voservices.net/wave Query Services –ADQL and SkyNode http://skyservice.pha.jhu.edu/develop/vo/adql/ http://skyservice.pha.jhu.edu/develop/vo/adql/ –And http://SkyQuery.Nethttp://SkyQuery.Net Registry: –see below Your program Data In your address space Web Service soap object in xml

19. 19 The Registry UDDI seemed inappropriate –Complex –Irrelevant questions –Relevant questions missing Evolved Dublin Core –Represent Datasets, Services, Portals –Needs to be machine readable –Federation (DNS model) –Push & Pull: register then harvest http://www.ivoa.net/twiki/bin/view/IVOA/IvoaResReg

20. 20 Demo SkyServer: –navigator showing cutout web service –List: showing many calls and variant use. SkyQuery: –Show integration of various archives. –Explain spatial join xMatch operator.

21. 21 SkyServer.SDSS.org A modern Astronomy archive –Raw Pixel data lives in file servers –Catalog data (derived objects) lives in Database –Online query to any and all Also used for education –150 hours of online Astronomy –Implicitly teaches data analysis Interesting things –Spatial data search –Client query interface via Java Applet –Query interface via Emacs –Popular –Cloned by other surveys (a template design) –Web services are core of it.

22. 22 SkyQuery A Prototype WWT Started with SDSS data and schema Imported12 other datasets into that spine schema. (a day per dataset plus load time) Unified them with a portal Implicit spatial join among the datasets. All built on Web Services –Pure XML –Pure SOAP –Used.NET toolkit

23. 23 Federation: SkyQuery.NetSkyQuery.Net Combine 4 archives initially Added 9 more Send query to portal, portal joins data from archives. Problem: want to do multi-step data analysis (not just single query). Solution: Allow personal databases on portal Problem: some queries are monsters Solution: “batch schedule” on portal server, Deposits answer in personal database.

24. 24 2MASS INT SDSS FIRST SkyQuery Portal Image Cutout SkyQuery Structure Each SkyNode publishes –Schema Web Service –Database Web Service Portal is –Plans Query (2 phase) –Integrates answers –Is a web service

25. 25 MyDB http://skyservice.pha.jhu.edu/devel/casjobs/ http://skyservice.pha.jhu.edu/devel/casjobs/ Portal allows federation of data but… Intermediate results may be large. Intermediate results feed into next analysis step. Sending them back-and-forth to client is costly and sometimes infeasible. Solution: create a working DB for client at Portal: MyDB

26. 26 MyDB http://skyservice.pha.jhu.edu/devel/casjobs/ http://skyservice.pha.jhu.edu/devel/casjobs/ Anyone can create a personal DB at SkyServer portal. –It is about 100 MB –It is private Simple queries done immediately Complex queries done by batch scheduler All queries can create/read/write MyDB tables Very popular with “serious” users. MyDB will be sharable with by a group.

27. 27 Open SkyQuery SkyQuery being adopted by AstroGrid as reference implementation for OGSA-DAI (Open Grid Services Architecture, Data Access and Integration). SkyNode basic archive object http://www.ivoa.net/twiki/bin/view/IVOA/SkyNode http://www.ivoa.net/twiki/bin/view/IVOA/SkyNode SkyQuery Language (VoQL) is evolving. http://www.ivoa.net/twiki/bin/view/IVOA/IvoaVOQL http://www.ivoa.net/twiki/bin/view/IVOA/IvoaVOQL

28. 28 The WWT Components Outline Data Sources –Literature –Archives Unified Definitions –Units, –Semantics/Concepts/Metrics, Representations, –Provenance Object model Classes and methods Portals WWT is a poster child for the Data Grid. What we learned Astro is a community of 10,000 Homogenous & Cooperative If you can’t do it for Astro, do not bother with 3M bio-info. Agreement –Takes time –Takes endless meetings Big problems are non-technical –Legacy is a big problem. Plumbing and tools are there But… –What is the object model? –What do you want to save? –How document provenance?