What does LOFAR have to do with the Virtual Observatory (VO)? LOFAR Science Day 16 December 2003 Melbourne David Barnes The University of Melbourne

VO is Standards (1) data description –metadata, content tags, context tags –IVOA: VOTable, UCD WGs data retrieval –http, ftp, web services, replica catalogs –IVOA: Data Access Layer, Grid & WS WGs data models –structure and relationships in data –IVOA: Data Modelling WG

VO is Standards (2) data and service compilation and indexing –registries, resource description –IVOA: Registry WG finding data and services –VO Query Language, SQL, brokers, agents –IVOA: VO Query Language WG workflow and interface –portals, portlets, authentication, authorisation –IVOA: no WG yet

VO is Grid Computing local, national, global compute grids –transparent, secure access to processors –job submission & monitoring, workflows local, national, global data grids –replica catalogues (caching) –global filesystems local, national, global networks –move code to data or data to code –realtime client-server processing, analysis and visualisation

VO Utopia (1) diverse data (eg. X-ray photon event lists and SPH cosmo simulation) –found in standard VO registries –described using standard vocabulary –tagged with pointers into a standard astronomy data model –accessed via standard mechanism eg. SOAP query to Web Service –provided in standard storage format, eg. VOTable, HDF, FITS, … data sets are interoperable

VO Utopia (2) diverse tools (eg. source extractor, volume visualiser, PCA algorithm) –found in standard VO registries –described using standard vocabulary –tagged with pointers into a standard astronomy data model –accessed via standard mechanism eg. SOAP query to Web Service –read and write standard storage format, eg. VOTable, HDF, FITS, … tools are interoperable tools AND data are interoperabletools AND data are interoperable

VO Utopia (3): LOFAR naif: registry search : LOFAR exists! : free data is available! traditionalist: query LOFAR archive via VO, download images and analyse at home aspirant: use on-line VO LOFAR pipeline to recalibrate intermediate data product/s and re-image with modified visibility weights braveheart: incorporate any or all LOFAR data products in VO workflows exploiting several other data sets at once

VO Utopia (4): LOFAR Grid: this data is popular in France : replicate data CDS : provide transparently to UK users Grid: this data is owned by David and in propietary period : you cant have it yet Grid: you have requested 4 TB of data : click here to upload and run your code at the data centre rather than fetching for days. Telescope: I am just a long-queue data archive : search for data returns null result : would you like to queue an observing request based on your search metadata?

LOFAR data products will –be large … accurate and standardised metadata will reduce unnecessary downloads and processing –be heavily qualified … pointers into a global data model will help software and astronomers make valid use of the data LOFAR should make a substantial contribution to data modeling in the VO –experience widely varying usage patterns … VO and Grid will provide intelligent caching and data distribution mechanisms

adopting VO buys: functionality: new and emerging VO tools and services, plus wrapped legacy software, immediately available interoperability: all VO tools and services work with and understand LOFAR data to the best of their ability archive & data centre: reference implementations of standards will provide data centre in a box software wider user community

Reality… VO standards are moving targets VOTable stores metadata and small catalogues; most real data still stored in legacy formats (FITS, HDF, …) Tools implementing or supporting VO standards are rare! Portals, workflows, embracing the Grid paradigm, …, are all in the future

end ( unless you care about LOFAR data rates … )

LOFAR data: baseband input & output total baseband data rate is ~20 Terabits/s –13000 antennae, dual pol –64 Msamples/s –12-bit sampling total station data rate is ~150 Gigabits/s –100 stations –16 beams –4 Msamples/s (2 MHz bandwidth) –6-bit sampling

LOFAR data: correlator input & output central correlator input rate is ~150 Gigabit/s output rate is ~10 GB per cycle –5000 baselines (100 stations) –2000 channels (2 MHz bandwidth, 1kHz resolution) –16 beams –4 polarisation products –32-bit IEEE floats ~36 TB/hr for 1s samples ~3.6 TB/hr for 10s samples