CAA Database Overview Sinéad McCaffrey

Metadata ObservatoryExperiment Instrument Mission Dataset File

Adding data to database CEF FileCEH (header) File XML Metadata CEF2XML converter Add data to dB Administration Tool ‘Add Dataset’ Parameters Inventory File

Adding a dataset to the dB A file can be ingested once that dataset exists on the database. The instrument/experiment/observatory/mission must exist on the dB before the dataset can be added. Procedure ‘AddDataset’ takes CEF file from a new dataset delivery, extracts metadata and adds to the database. Additional parameters (such as location on disk etc included at addition time). It copies the metadata in XML format to XML dB area. If it is a graphical dataset, additional datasets details (plot size etc) are included at addition time. Non-metadata items (group, display title, unit, display order etc) added/modified by Administration GUI.

Ingesting a file Pre-ingestion script runs on delivery directory and validates files (runs our CEFpass validator) Valid files moved to ingestion area Ingestion runs on (specified) ingestion area and ingests file list If it is a non-CEF product, it opens CSV file in defined area and processes May rename non-CEF file (as specified in input file) All details stored on Ingestion table Copies file to disk (location in Destination table) Nightly job runs so successful files are moved to Catalogue table and are then available to users.

Ingesting a file At ingestion time, some simple validation is carried out:  Filename/Logical FileID is unique  Same/higher version for this dataset/interval does not exist If previous version exists on dB, upon successful ingestion, previous version marked as inactive (this is part of the batch job to move records to catalogue)

Inventory for files Daily (CP) job runs to check for recently added files. For each new file it reads the CEF file, identifies inventory information and outputs to CSV file. For non-CEF products only file delivery is indicated (for example, any gaps in file not identified) This inventory information is imported nightly into a table on the dB which is then used by inventory/web query. Undelivered data, gap in data and data found are stored

Ingestion System Process (2005)