Metadata for the SKA - Niruj Mohan Ramanujam, NCRA

Overview Any auxillary data needed to fully interpret and process the science data is termed as metadata. This metadata is generated by multiple domains. Metadata is collected by the M&C system from various components and sub-systems of the SKA (engineering and array configuration data, alarms etc) & external Elements and equipment connected to the M&C (signal transport, weather monitoring equipment etc), based on multiple tables, throughout the course of an observation, is written to a repository along with the science data, with links provided to the rest of the relevant M&C Archive data, to be analysed and interpreted by science data analysis tools

Overview The issues related to metadata include Definition Diversity Generation Storage Drill-down or backtracking Formats

Metadata Definitions Metadata for radio telescopes have traditionally been a collection of keywords (and tables) in ascii in FITS files, and some text in log files.

Metadata Definitions LOFAR and ALMA have enlarged the concept of metadata and this is even more so for the SKA. We expect metadata to be generated through metadata definitions read from Static configuration files, which encode certain default metadata which remain constant through substantial parts of the observation, e.g. array configuration, frequency settings, Dynamic files, the contents and nature of which are changing, e.g. system alarms, RFI and weather monitoring, User-specified files, wherein the user can decide to include certain M&C-accessible data in their metadata SKA needs for metadata will be defined by Software and Computing.

Diversity of Metadata Metadata differ in their specification method, origin, modes of acquisition, timescale for change, storage method etc. Both metadata generation and storage need to be able to handle this diversity, which is dependent on an SKA Science Data Model (ALMA SDM may be generic enough to handle SKA needs). We attempt to provide an non-exhaustive list of the various types of metadata, to help show the complexity involved.

Diversity of Metadata Metadata typeOriginsExamples Array configurationInstallation teamTree from sub array down to receiver Observation configuration Observation Preparation Pointing direction, channel frequencies, receiver id,etc. Observing projectProject id, observer, links to other project data Log fileArchiveLog file Alarms and EventsAlarm Handler, Archive Antennae down, number of dipoles in a station less than optimal etc. Data processingScience Data Archive Flagging, initial calibration, visibility weights etc. System performance M&C Stability of power levels, temperature etc., available disk memory, real-time correction performance etc. RFITime-tagged, frequency tagged information on RFI EnvironmentTemperature, precipitation, wind speed, TEC estimates Applied correctionsPointing corrections, instrument offsets, weights etc. Instrument dataTotal power, frequency stability, clock errors etc. Performance gapsPointing errors, non working dipoles etc.

Metadata Generation These metadata are generated by the M&C system by abstracting from the information acquired by its sub-systems, collected by the M&C system by subscribing to custom applications and interfacing with other domains (e.g. engineering, science, external monitors, data analysis etc), generated and archived by other domains and applications M&C needs to ensure that all relevant M&C data will be stored in the Metadata archive.

Metadata Generation This leads to two models for the generation of Metadata... A central Metadata Engine sits inside the Central M&C and assembles the required metadata by subscribing to whichever service is required (e.g. LOFAR), and Every Regional M&C has its own Metadata engine which collects necessary information locally, and sends it upwards, along with a System M&C Metadata Engine. These have an impact on synchronicity, metadata aggregation, alarm generation and so on.

Storage Metadata will be stored in the Archive in suitable form(s), with drill-down/backtracking, and bundled together with the science data. The nature of storage of different types of Metadata will depend partly on the SDM. Some issues are.. Differing time granularities - separate for efficiency (ALMA) Differing latencies, especially those needed as feedback for other applications Ability to change metadata post-observations (AWE) Ability to recreate metadata from the Archive Ability to track back (drill-down) from the data (AWE) Different delivery times for different metadata

Drill-down Starting from the metadata, the user should be able to drill-down to the Component level M&C information through the Engineering Archive - called drill-down or trackability. Any changes to the data downstream of acquisition will also need to be registered in the metadata, which is therefore dynamic. AstroWise (AWE) has this capability, wherein the metadata are stored in a relational database. Metadata will also need to be linked to the Log Archive.

Metadata Formats The recommended practice for Metadata is to define them through an XML schema, using controlled vocabularies and link them to the binary files containing science data. The structure and format will also be influenced by the science data format. LOFAR uses HDF5 files ALMA uses CASA MS tables AWE uses relational databases The format of the metadata will depend on the SKA Science Data Model and may well change with time initially. This imposes requirements on the flexibility of analysis tools and SDM formats

Metadata Standards There are quite a few Metadata standards which could be useful... ARDA Metadata Grid Application (AMGA) for the LHC AstroGrid-D, RTML, RDF - Grid and VO protocols Open Archives Initiative Protocol for Metadata harvesting ISO IEC an international metadata registry standard

Summary Any auxillary data needed to fully interpret and process the science data is termed as metadata. This metadata is generated by multiple domains. M&C supports science operation by collecting Metadata from various components and sub-systems of the SKA (engineering and array configuration data, alarms etc) & from external Elements and equipment connected to the M&C (signal transport, weather monitoring equipment etc), based on multiple tables, throughout the course of an observation, is written to a repository along with the science data, with links provided to the rest of the relevant M&C Archive data, to be analysed and interpreted by science data analysis tools.

See section 5.5 of M&C Design Concept Descriptions, “04_WP TD-002v0.2_dcd”