IVOA Interop, SL de El Escorial, Oct IVOA Data Access Layer Doug Tody (NRAO/NVO/IVOA) I NTERNATIONAL V IRTUAL O BSERVATORY A LLIANCE

IVOA Interop, SL de El Escorial, Oct IVOA DAL - Madrid DAL WG Status

IVOA Interop, SL de El Escorial, Oct IVOA Madrid DAL Session Data Access Layer Reminder –Is about access to astronomical data via the VO Catalogs, images, spectra/SEDs, time series, event and visibility data –Access is analysis-oriented Static (archival) vs virtual (subset/transformed) data –Both are important –Data management vs data access for analysis –Virtual data is essential for analysis (cutouts etc.) Data model-based data access –VO data models vs external-data –Object hierarchy Generic dataset (characterization, curation, etc.) Image, spectrum, etc. (catalog is different)

IVOA Interop, SL de El Escorial, Oct IVOA Madrid DAL Session Goals for this Interop –Identify issues remaining to be resolved to advance SSA to V1.0 Query interface, data model currently at V0.9x –Advance spectral line access protocol (SLAP) to V0.1 –Identify requirements for SIA V1.1 upgrade –Implementations Coordinate development of VO/DAL client implementations, verify/test frameworks –Update roadmap

IVOA Interop, SL de El Escorial, Oct IVOA Madrid DAL Session WG Agenda –WG Status –Simple Spectral Access Protocol (SSAP) Query interface (D. Tody) Characterization issues (F. Bonnarel) Data model (J. McDowell) –Simple Spectral Line Access Protocol (SLAP) Status, query interface (J. Salgado) BREAK 11:00 –Simple Image Access Protocol (SIAP) Some issues for SIA V1.1 (F. Bonnarel) –Implementations Demos (F. Pierfederici) –WCS-augmented graphics display –Image footprint description and plotting VO/DAL-Client, reference implementations, verification (D. Tody) –WG Summary

IVOA Interop, SL de El Escorial, Oct IVOA DAL - Madrid Simple Spectral Access Protocol (SSAP)

IVOA Interop, SL de El Escorial, Oct Simple Spectral Access (SSA) What is it? –Provides access to 1D spectra, time series, SEDs –Tabular spectrophotometric data (photometry points) –Represents second generation, data model-based DAL interfaces Status –Draft V0.9 query interface reviewed in Kyoto (May 05) –Revised based on Kyoto discussion (now at V0.91) –Current effort is to produce V1.0 WD for trials, PR –Some initial prototypes already exist (services, client apps)

IVOA Interop, SL de El Escorial, Oct Basic Usage SSA specification may be complex, but basic usage is simple Simple query –POS, SIZE, FORMAT - like cone search, SIA –Possibly refined by spectral or time bandpass, etc. –Most metadata in query response is optional Data retrieval –Simple retrieval is again URL-based –Get back a dataset "document" (VOTable, FITS, JPEG, etc.) –In simplest case could be wavelength, flux as text (for Spectrum) –Pass-through of external data is permitted Data Analysis –Standard data model isolates application from quirks of –external project data

IVOA Interop, SL de El Escorial, Oct Concepts - Dataset-oriented Data object type –Spectrum, TimeSeries, SED Dataset creation type –AtlasWhole datasets, uniform survey data –Pointed Whole datasets, variable instrumental data –Cutout Subset, data samples are not modified –Resampled Subset, data samples computed by service Dataset derivation –Observed An observation –Composite Combination of several observations –Dynamic Dynamically simulated obs. from real data –Synthetic Data from a theoretical model

IVOA Interop, SL de El Escorial, Oct Data Models Data models used in SSA –Spectral dataSpectrum, TimeSeries, SED –Dataset Generic dataset descriptor –Target Astronomical target observed –Curation Origin of data –CharacterizationPhysical characteristics of data –Provenance Instrument which generated the data User defined data models –Metadata extension mechanisms additional data model attributes (table fields) additional resources in VOTable, linked back to main table –Provide a mechanism to "subclass" dataset to tailor it for a given data collection

IVOA Interop, SL de El Escorial, Oct Spectral Data (SED) spectrum segment Photometry point

IVOA Interop, SL de El Escorial, Oct Query Interface Mandatory query parameters –POSRA, DEC (ICRS) –SIZEdiameter (decimal degrees) –TIME data1,date2 (epoch in decimal years UTC) –BANDwave1,wave2 (meters in vacuum; source or observer) –FORMATVOTable, fits, xml, text, graphics, html, external

IVOA Interop, SL de El Escorial, Oct Query Interface Recommended query parameters –APERTURE approx spatial resolution (decimal degrees) –SPECRES spectral resolution (meters) –TOP number of top-ranked records to return –OBJTYPEmandatory if service returns multiple object types –COLLECTIONdata collection identifier

IVOA Interop, SL de El Escorial, Oct Query Interface Optional parameters –CREATORID creator-assigned dataset identifier (at most 1) –PUBIDpublisher-assigned dataset identifier (at most N) –COMPRESSenable compression (for both data _and_ queries?) –SNR signal-to-noise ratio –REDSHIFT redshift range (dlambda/lambda) –TARGETCLASSstar, galaxy, pulsar, PN, QSO, AGN, etc.

IVOA Interop, SL de El Escorial, Oct Query Response Classes of query metadata –Query metadataDescribes the query itself –Dataset metadataDescribes data object; object-specific –Target metadata Astronomical target –Curation metadata External identification of dataset –Characterization Coverage, Accuracy, Frame, etc. –Instrument metadata Service-defined; hard to standardize –Access metadata Describes how to access the dataset

IVOA Interop, SL de El Escorial, Oct Query Response Query Metadata –Query.ScoreHow well object matches query –Query.LNameLogical name (identifier) –Query.LNameKeyLogical name key (id-ref) Example: LName="MyObj123" LNameKey="server,format"

IVOA Interop, SL de El Escorial, Oct Query Response Dataset Metadata –Dataset.Type Spectrum, TimeSeries, SED, etc. –Dataset.DataModel DM name, e.g., "SSA-V0.90" –Dataset.Title Brief descriptive title of dataset –Dataset.SSA.NSamples Total samples in dataset Dataset.SSA.Aperture Characteristic aperture diameter –Dataset.SSA.TimeAxis TimeCoord axis (external data) –.SSA.SpectralAxis SpectralCoord axis (external data) –Dataset.SSA.FluxAxis Flux axis (external data) –Dataset.CreationType atlas, pointed, cutout, resampled –Dataset.Derivation observed, composite, dynamic, synthetic

IVOA Interop, SL de El Escorial, Oct Query Response Target Metadata –Target.NameName of astronomical object –Target.Class Target class (star, galaxy, QSO, etc.) –Target.SpectralClassSpectral class (e.g., 'O', 'B', etc.) –Target.Redshift Nominal redshift for object –Derived.VarAmpl Variability amplitude (fraction 0-1) –Derived.SNR Observed signal to noise ratio

IVOA Interop, SL de El Escorial, Oct Query Response Curation Metadata –Curation.CollectionData collection name (identifier) –Curation.Creator Creator identify (identifier) –Curation.CreatorID Creator-assigned dataset identifier –Curation.PublisherID Publisher-assigned dataset identifier –Curation.Date Dataset creation date (ISO date string) –Curation.Version Dataset version (within same ID) –Curation.MTimeDataset modify/update time

IVOA Interop, SL de El Escorial, Oct Query Response Characterization1 - Coverage –.Location.SpatialPosition (e.g., RA, DEC) –.Location.Time Observation time characteristic value –.Location.Spectral Spectral bandpass characteristic value –.Location.Spectral.BandID Bandpass ID (band or filter name) –.Bounds.Spatial Aperture footprint (polygon on sky) –.Bounds.Time Low/High time values –.Bounds.Spectral Low/High spectral values –.Bounds.Flux Limiting flux, saturation limit (Jansky) –.Fill.Spatial Spatial sampling filling factor (0-1) –.Fill.Time Time sampling filling factor (0-1) –.Fill.Spectral Spectral sampling filling factor (0-1)

IVOA Interop, SL de El Escorial, Oct Query Response Characterization2 - Accuracy –Accuracy.*.Calibrateduncalibrated, relative, absolute –Accuracy.*.Resolution Resolution of measured signal –Accuracy.*.StatErr Statistical error (measured) –Accuracy.*.SysErr Systematic error (estimated) ('*' = Spatial, Time, Spectral, Flux)

IVOA Interop, SL de El Escorial, Oct Query Response Characterization3 - Reference Frames –Frame.Spatial.TypeCoordinate frame (default ICRS) –Frame.Spatial.Equinox Coordinate system equinox (J2000) –Frame.Time.System Timescale (TT) –Frame.Time.SIDim SI factor and dimension –Frame.Spectral.SIDim SI factor and dimension –Frame.Flux.SIDim SI factor and dimension –Frame.Flux.UCD UCD of flux value (flux type) (These apply only to the query response) (SIDim metadata still under construction)

IVOA Interop, SL de El Escorial, Oct Query Response Instrument Metadata –Instrument.NameInstrument name (identifier) –Instrument.Exposure Total exposure time (seconds) –Instrument. Service-defined Notes –Optional; provided for instrumental data collections –In general, Collection, Bounds.Time, etc. are preferred –In general Instrument metadata is service-defined –Use Observation model as a starting point

IVOA Interop, SL de El Escorial, Oct Query Response Access Metadata –Access.ReferenceData access URL –Access.Format MIME type of returned dataset –Access.Size Approximate dataset size (bytes) –Access.Server Server endpoint URL Staging support goes here in the future –e.g., will dataset access require asynchronous staging –estimated cost to construct dataset

IVOA Interop, SL de El Escorial, Oct Service Metadata Usage –Describe service type and capabilities –Characterize service (data resources served, coverage, etc.) –Describe interface (optional query parameters) Interface –Requires new service metadata query method –Returns resource metadata descriptor (XML) Format –Registry resource descriptor (XML)

IVOA Interop, SL de El Escorial, Oct Data Retrieval Based on GET as with SIA –Variety of formats available –Compression supported Data representation –Data model defines logical content of data –The same data object may be represented in various formats –Hence we need to specify both the data model, and the file format

IVOA Interop, SL de El Escorial, Oct Data Retrieval Data models –SSA data model for fully-compliant data –Provider-defined data model for external data Data formats –VOTable (a container), native XML (direct serialization) –FITS binary table (another container; uses FITS spectral WCS) –Text, e.g., CSV –Graphics (JPEG etc.) –text/html (rendered into browser page)

IVOA Interop, SL de El Escorial, Oct IVOA DAL - Madrid Simple Image Access (SIA) V1.1

IVOA Interop, SL de El Escorial, Oct Future Development SIA V1.1 –Based on work done on SSA –Expanded query interface no longer limited to positional queries –Much richer query response generic dataset identification, characterization, etc. metadata extension mechanism –Selected features VOTable 1.1 with UCD 1+, GROUP, UTYPE query response can be ordered by "score" logical groupings of related query records compression support –Versioning required to make protocol upgrades manageable

IVOA Interop, SL de El Escorial, Oct Future Development Service verification –for testing at development time –when registered; level of compliance metric Grid capabilities –Data staging asynchronous image generation (long running jobs) batch generation of images (multiple images) –Data management support for single sign-on authentication, authorization network data caching, third party delivery (VOStore etc.) –Web service interface resource metadata service availability (etc.)

IVOA Interop, SL de El Escorial, Oct

IVOA Interop, SL de El Escorial, Oct

IVOA Interop, SL de El Escorial, Oct IVOA DAL - Madrid VO/DAL Client

IVOA Interop, SL de El Escorial, Oct VO-Client Functionality Data discovery –Registry query –Virtual directory service (logical name space) –Support for footprint service DAL queries –Cone search, SkyNode query –SIA, SSA queries Data retrieval –SIA, SSA data retrieval includes support for async i/o, authenticated access

IVOA Interop, SL de El Escorial, Oct VO-Client Functionality (2) Data management –Network data cache (VOStore) file-oriented data storage, i/o to VO for HPC, file store is a cluster file system? probably requires integrated DBMS capability keep track of whether data is cached locally if data is retrieved, will be found in cache –HPC data cache (for cluster parallel i/o) this is an interface; data is common with VOStore DAO controls access to metadata compute node can have direct access to bulk data

IVOA Interop, SL de El Escorial, Oct VO-Client Functionality (3) Data Access Objects –OO multilanguage APIs for astronomical data catalog, image, spectrum, time series etc. also data containers (VOTable, FITS, etc.) –DAO mediates access to object metadata safeguards concurrent access, supports scalability direct HPC access to bulk data (for images) –Implement the VO/DAL data models catalog, image, spectrum, SED, time series, etc.

IVOA Interop, SL de El Escorial, Oct VO-Client Architecture VOClient daemon –Runs locally on client workstation, cluster, etc. –Probably will be implemented in Java –Uses VOClient Java library (dalclient follow-on) –Includes integral VOStore implementation APIs –Java API –C/C++ API –Python client –PHP client etc. Reference applications –Python/Java GUIs –Data browser (VODA)

IVOA Interop, SL de El Escorial, Oct