Sharing Data With ODI D. Heynderickx, DH Consultancy, Belgium 29/05/2019 Sharing Data With ODI IRENE Workshop, Sykia, Greece D. Heynderickx, DH Consultancy, Belgium H.D.R. Evans, ESA/ESTEC, The Netherlands P. Wintoft, Solar Analytics, Sweden

Data sharing Two types of sharing 29/05/2019 Two types of sharing Bulk downloads for internal processing Scientific studies (e.g. radiation belt modelling) Application services (e.g. radiation effects analysis for historic SPE) Data services (this talk) User downloads of selected data slices Time series, time averages, coordinate binning Locally added data (e.g. geomagnetic coordinates) Move away from file based distribution (too) many file standards Remove the burden of file processing from the end user Supply data in easily interpreted format (JSON, CSV) Standards should shift from file formats to APIs, output streams and metadata RESTful interfaces (e.g. HAPI, recently endorsed by COSPAR resolution) SPASE metadata (recently endorsed by COSPAR resolution) PRBEM (needs updating) IRENE Workshop, Sykia, Greece

ODI overview 29/05/2019 Open Data Interface (ODI) is a database system for retrieving, processing and storing space environment (and other) data and metadata in a MySQL (MariaDB) database. Development started in 2008 (Swedish Institute of Space Physics, DH Consultancy, Solar Analytics) Compliant with CDF/ISTP data model (SPASE in progress) Server and client software for data downloading and pre/post-processing, and data retrieval. Currently support for ~250 datasets. Extensible with user added functionality. Available on the ESA European Space Software Repository (https://essr.esa.int/) IRENE Workshop, Sykia, Greece

ODI applications ESA projects EC FP7 projects 29/05/2019 ESA projects SEPEM, SEDAT, SPENVIS, SAAPS, JHelioViewer, VSWMC SRREM, RENELLA, VALIRENE, GALEM, ESPREM, ecIRENE SSA P2-SWE-II EC FP7 projects SEPServer SPACECAST, SPACESTORM EURISGIC IRENE Workshop, Sykia, Greece

ODI server software 29/05/2019 PHP engine for database communication and process flow Download and parser scripts NASA/GSFC CDFexport tool to ingest CDF data Python tool to ingest NetCDF data Ingestion of FITS and PDS datasets is under development Dedicated parsers for other file types Coordinates for Earth orbiting spacecraft Download of TLEs from https://space-track.org Generation of spacecraft coordinates (GEI) using NASA/JPL SPICE library (http://naif.jpl.nasa.gov/) Magnetic coordinates for spacecraft in Earth’s magnetosphere using UNILIB cron/quartz setup for automatic download and ingestion User may add functionality triggered by hooks on download and ingestion IRENE Workshop, Sykia, Greece

Datasets supported in ODI distribution (~250) 29/05/2019 ACE: archive and real time EPAM, SWEPAM, MAG, SIS DSCOVR: real time IMF and plasma data (JSON streams) GOES: archive and real time SEM/EPS data SREM: PROBA1, Integral, GioveB, Rosetta, Herschel, Planck Magnetic and solar indices (Kp, Dst, F10.7, ISN, OMNI, …) Interplanetary particle datasets: HELIOS, IMP8, Voyager, Pioneer, Wind Radiation belt missions: AZUR/EI-88, S3-3/PT, CRRES/MEA/HEEF/PROTEL, UARS/PEM, SAMPEX/PET, NOAA/POES/SEM2, XMM/ERMD, PROBA-V/EPT, RBSP/HOPE/MAGEIS/REPT/RPS, HIMAWARI/SEDA “Proprietary” datasets: MIR/REM, STRV1B/REM, AMPTE/UKS, EQUATOR-S, ISEE1/WIM/KED, Meteosat/SEM, Galileo/EMU IRENE Workshop, Sykia, Greece

ODI client software APIs for php, Java, IDL, Matlab, Jython, Python 29/05/2019 APIs for php, Java, IDL, Matlab, Jython, Python Standardised procedure syntax Outputs in language specific objects Remote connection over HTTP/REST (server/client), JSON/CSV output https://spitfire.estec.esa.int/OdiREST/odi/api/v0.1/list-datasets https://spitfire.estec.esa.int/OdiREST/odi/api/v0.1/read-data?dataset=proba1_srem_pacc_l2&columns=cdf_epoch,fpdo_1,fpdo_2&start=2018-01-01T00:00:00&end=2018-01-10T00:00:00 https://spitfire.estec.esa.int/OdiREST/odi/api/v0.1/read-data?dataset=proba1_srem_pacc_l2&columns=avg(fpdo_1)&start=2018-01-01T00:00:00&end=2019-01-01T00:00:00&where=fpdo_quality=0 HAPI (NASA Heliospheric API: https://github.com/hapi-server/data-specification) server/client Excel interface IRENE Workshop, Sykia, Greece

HAPI Heliophysics Application Programmer’s Interface (HAPI) 29/05/2019 Heliophysics Application Programmer’s Interface (HAPI) NASA developed RESTful API standard for time series data Four required endpoints (GET only) Describe server capabilities (output formats) https://spitfire.estec.esa.int/OdiHAPI/capabilities List the catalogue of datasets https://spitfire.estec.esa.int/OdiHAPI/catalog Show information about a dataset (metadata) https://spitfire.estec.esa.int/OdiHAPI/info?id=integral_irem_pacc_l2 Stream data content for a selected dataset https://spitfire.estec.esa.int/OdiHAPI/data?id=integral_irem_pacc_l2&time.min=2018-10-03T06:00:00&time.max=2018-10-04 IRENE Workshop, Sykia, Greece

29/05/2019 IRENE Workshop, Sykia, Greece http://spase-group.org/

Discussion points Updates of PRBEM standard 29/05/2019 Updates of PRBEM standard Introduction of ‘FESA’ in RBSP datasets, should be ‘FExS’ Strict definition of coordinate systems Introduction of SPASE metadata Extensions of SPASE metadata Update of the dictionary with radiation quantities (e.g. dose) Introduction of ‘dependent variable’ Specification of individual energy bins in energy spectra HAPI standard extensions Does not allow for quantified queries, e.g. using WHERE or AVG Or, PRBEM defined set of REST keywords? Release of post-processed data after peer review IRENE Workshop, Sykia, Greece