VO Standards – Catalog Access Tamás Budavári Johns Hopkins University

9/14/2004NVO Summer School, Aspen Background Virtual Observatory Query Language Level 3. Semantically rich XML language (TBD) Level 2. Portals combining astro archives e.g. SkyQuery and Open SkyQuery Level 1. Query a single database Astronomical Data Query Language VOTable Start with something simple Decided not to reinvent the wheel

9/14/2004NVO Summer School, Aspen Outline SQL Primer How to query databases ADQL Extensions for astronomy Distributed resources SkyQuery SkyNode interface SkyQuery portal

9/14/2004NVO Summer School, Aspen Structured Query Language Simple syntax SELECT FROM WHERE SELECT FROM WHERE For example: select ra, dec from galaxy where r<19 Joining tables More tables with common column (link, relation) Select from multiple tables with join criteria E.g. an object id: select g.ra, g.dec, s.z from galaxy g, spec s where g.objid=s.objid and g.r<19

9/14/2004NVO Summer School, Aspen Astronomical Data Query Language SQL extended for astronomy Added survey name in from clause e.g. SDSS:Galaxy g, DLS:obj o Specify area on sky Added REGION keyword e.g. Region(' CIRCLE J ') Formulate federated query Added XMATCH keyword e.g. XMatch(g,o) using aliases

9/14/2004NVO Summer School, Aspen ADQL Translator Representations String for humans /s XML for machines /x Helper service Converts both ways String to XML XML to String XML is pre-parsed So your service could accept both if needed

9/14/2004NVO Summer School, Aspen ADQL representations ADQL/s ADQL/x < Ra Dec < Ra Dec select s.objd, t.id from TWOMASS:PhotoObj t, SDSS:Galaxy s where XMatch(s,t)<3.5 and Region('Circle J ') XML

9/14/2004NVO Summer School, Aspen Open SkyQuery SkyNode interface to archives Implements ADQL returns VOTable Basic node understands REGION Full node understands XMATCH SkyQuery portal Knows the SkyNodes from Registry Understands federated query

9/14/2004NVO Summer School, Aspen Architecture Image cutout SkyNode SDSS SkyNode 2Mass SkyNode First SkyQuery Web App

9/14/2004NVO Summer School, Aspen Step-by-step Analyze query Cost estimates Sort by size Execution plan Cross-match Recursively from small to large db Propagate necessary attributes only Return output in VO format SELECT o.objId, o.r, o.type, t.objId, t.j_m FROM SDSS:PhotoPrimary o, TWOMASS:PhotoObj t WHERE XMatch(o,t)<3.5 AND Region( CIRCLE J2000 … ) AND (o.i-t.j_m)>2 AND o.type=3 SELECT o.objId, o.r, o.type, t.objId, t.j_m FROM SDSS:PhotoPrimary o, TWOMASS:PhotoObj t WHERE XMatch(o,t)<3.5 AND Region( CIRCLE J2000 … ) AND (o.i-t.j_m)>2 AND o.type=3

9/14/2004NVO Summer School, Aspen SkyNode Metadata functions Tables, Columns, Functions Uptime, Footprint, etc … Query functions PerformQuery (ADQL q) Simple query interface QueryCost (ADQL q) Cost estimate for query ExecutePlan (XPlan plan) Recursive probabilistic join

9/14/2004NVO Summer School, Aspen Data flow SkyQuery query SkyNode 1 SkyNode 2 SkyNode 3 Registry

9/14/2004NVO Summer School, Aspen Cross-matching Probabilistic spatial join Likelihood that all measurements are actually observations of the same object Consider all objects from all surveys For surveys A, B and C, there are N A N B N C combinations, N X is the number of objects that meet criteria in survey X Massive optimization problem for true positions on the sky (x,y,z)

9/14/2004NVO Summer School, Aspen XMatch variants XMatch (r, g, w) Mandatory matches in all surveys ! XMatch (r, g,!w) Look for dropouts in certain surveys Remove objects if found in w ? XMatch (r, g,?w) Return matches in w if found but don t discard if not Essentially an outer join

9/14/2004NVO Summer School, Aspen SkyQuery portal Hides complexity Simple web services Methods Get to metadata of nodes Find location of nodes Send query to one node Send federated query Create execution plan Submit execution plan

9/14/2004NVO Summer School, Aspen

9/14/2004NVO Summer School, Aspen And more… Cross-match your own catalog Upload file to temporary table, e.g. MyTable Use MyData:MyTable in federated query Browse metadata on SkyNodes Sample queries of all types

9/14/2004NVO Summer School, Aspen Build your own SkyNode Download template SkyNode written in C# Uses SQL Server 2000 HTM for fast searches Where? Ready to use web service Templates for database From scratch Write it in any language C ++, Java, C #, Perl, Python Use ADQL Translator? Open source.NET C # w/ DB access and WS Runs on your favorite OS

9/14/2004NVO Summer School, Aspen Summary Astronomical Data Query Language Working VO standard for catalog access Constructs for area and cross-match Open SkyQuery Dynamically federates astronomical archives Uses VO Registry to lookup SkyNodes On-the-fly probabilistic cross-matching w/ dropout and outer join capabilities

9/14/2004NVO Summer School, Aspen Future Works Things to add Implement missing outer join Full featured region specs Footprint services Test survey boundaries Automated intersection of footprints And more …

9/14/2004NVO Summer School, Aspen Where to go Links in the talk Other relevant links