1. VO Standards – Catalog Access Tamás Budavári Johns Hopkins University

2. 9/14/2004NVO Summer School, Aspen 20042 Background Virtual Observatory Query Language Level 3. Semantically rich XML language (TBD) Level 2. Portals combining astro archives e.g. SkyQuery and Open SkyQuery Level 1. Query a single database Astronomical Data Query Language VOTable Start with something simple Decided not to reinvent the wheel

3. 9/14/2004NVO Summer School, Aspen 20043 Outline SQL Primer How to query databases ADQL Extensions for astronomy Distributed resources SkyQuery SkyNode interface SkyQuery portal

4. 9/14/2004NVO Summer School, Aspen 20044 Structured Query Language Simple syntax SELECT FROM WHERE SELECT FROM WHERE For example: select ra, dec from galaxy where r<19 Joining tables More tables with common column (link, relation) Select from multiple tables with join criteria E.g. an object id: select g.ra, g.dec, s.z from galaxy g, spec s where g.objid=s.objid and g.r<19

5. 9/14/2004NVO Summer School, Aspen 20045 Astronomical Data Query Language SQL extended for astronomy Added survey name in from clause e.g. SDSS:Galaxy g, DLS:obj o Specify area on sky Added REGION keyword e.g. Region(' CIRCLE J2000 181.3 -0.76 6.5 ') Formulate federated query Added XMATCH keyword e.g. XMatch(g,o) using aliases

6. 9/14/2004NVO Summer School, Aspen 20046 ADQL Translator Representations String for humans /s XML for machines /x Helper service Converts both ways String to XML XML to String XML is pre-parsed So your service could accept both if needed

7. 9/14/2004NVO Summer School, Aspen 20047 ADQL representations ADQL/s ADQL/x < Ra Dec 180 0 6.5 < Ra Dec 180 0 6.5 select s.objd, t.id from TWOMASS:PhotoObj t, SDSS:Galaxy s where XMatch(s,t)<3.5 and Region('Circle J2000 180 0 6.5') XML

8. 9/14/2004NVO Summer School, Aspen 20048 Open SkyQuery SkyNode interface to archives Implements ADQL returns VOTable Basic node understands REGION Full node understands XMATCH SkyQuery portal Knows the SkyNodes from Registry Understands federated query

9. 9/14/2004NVO Summer School, Aspen 20049 Architecture Image cutout SkyNode SDSS SkyNode 2Mass SkyNode First SkyQuery Web App

10. 9/14/2004NVO Summer School, Aspen 200410 Step-by-step Analyze query Cost estimates Sort by size Execution plan Cross-match Recursively from small to large db Propagate necessary attributes only Return output in VO format SELECT o.objId, o.r, o.type, t.objId, t.j_m FROM SDSS:PhotoPrimary o, TWOMASS:PhotoObj t WHERE XMatch(o,t)<3.5 AND Region( CIRCLE J2000 … ) AND (o.i-t.j_m)>2 AND o.type=3 SELECT o.objId, o.r, o.type, t.objId, t.j_m FROM SDSS:PhotoPrimary o, TWOMASS:PhotoObj t WHERE XMatch(o,t)<3.5 AND Region( CIRCLE J2000 … ) AND (o.i-t.j_m)>2 AND o.type=3

11. 9/14/2004NVO Summer School, Aspen 200411 SkyNode Metadata functions Tables, Columns, Functions Uptime, Footprint, etc … Query functions PerformQuery (ADQL q) Simple query interface QueryCost (ADQL q) Cost estimate for query ExecutePlan (XPlan plan) Recursive probabilistic join

12. 9/14/2004NVO Summer School, Aspen 200412 Data flow SkyQuery query SkyNode 1 SkyNode 2 SkyNode 3 http://openskyquery.net Registry

13. 9/14/2004NVO Summer School, Aspen 200413 Cross-matching Probabilistic spatial join Likelihood that all measurements are actually observations of the same object Consider all objects from all surveys For surveys A, B and C, there are N A N B N C combinations, N X is the number of objects that meet criteria in survey X Massive optimization problem for true positions on the sky (x,y,z)

14. 9/14/2004NVO Summer School, Aspen 200414 XMatch variants XMatch (r, g, w) Mandatory matches in all surveys ! XMatch (r, g,!w) Look for dropouts in certain surveys Remove objects if found in w ? XMatch (r, g,?w) Return matches in w if found but don t discard if not Essentially an outer join

15. 9/14/2004NVO Summer School, Aspen 200415 SkyQuery portal Hides complexity Simple web services Methods Get to metadata of nodes Find location of nodes Send query to one node Send federated query Create execution plan Submit execution plan

16. 9/14/2004NVO Summer School, Aspen 200416

17. 9/14/2004NVO Summer School, Aspen 200417 And more… Cross-match your own catalog Upload file to temporary table, e.g. MyTable Use MyData:MyTable in federated query Browse metadata on SkyNodes Sample queries of all types

18. 9/14/2004NVO Summer School, Aspen 200418 Build your own SkyNode Download template SkyNode written in C# Uses SQL Server 2000 HTM for fast searches Where? http://openskyquery.net Ready to use web service http://skyserver.org Templates for database From scratch Write it in any language C ++, Java, C #, Perl, Python Use ADQL Translator? Open source.NET http://go-mono.org C # w/ DB access and WS Runs on your favorite OS

19. 9/14/2004NVO Summer School, Aspen 200419 Summary Astronomical Data Query Language Working VO standard for catalog access Constructs for area and cross-match Open SkyQuery Dynamically federates astronomical archives Uses VO Registry to lookup SkyNodes On-the-fly probabilistic cross-matching w/ dropout and outer join capabilities

20. 9/14/2004NVO Summer School, Aspen 200420 Future Works Things to add Implement missing outer join Full featured region specs Footprint services Test survey boundaries Automated intersection of footprints And more …

21. 9/14/2004NVO Summer School, Aspen 200421 Where to go Links in the talk http://openskyquery.net/ http://openskyquery.net/adqltranslator/ http://skyserver.org/ http://skyquery.net/ Other relevant links http://us-vo.org/ http://voservices.org/ http://www.ivoa.net/