1. Sharing Data With ODI D. Heynderickx, DH Consultancy, Belgium
29/05/2019
Sharing Data With ODI
IRENE Workshop, Sykia, Greece
D. Heynderickx, DH Consultancy, Belgium
H.D.R. Evans, ESA/ESTEC, The Netherlands
P. Wintoft, Solar Analytics, Sweden

2. Data sharing Two types of sharing
29/05/2019
Two types of sharing
Bulk downloads for internal processing
Scientific studies (e.g. radiation belt modelling)
Application services (e.g. radiation effects analysis for historic SPE)
Data services (this talk)
User downloads of selected data slices
Time series, time averages, coordinate binning
Locally added data (e.g. geomagnetic coordinates)
Move away from file based distribution
(too) many file standards
Remove the burden of file processing from the end user
Supply data in easily interpreted format (JSON, CSV)
Standards should shift from file formats to APIs, output streams and metadata
RESTful interfaces (e.g. HAPI, recently endorsed by COSPAR resolution)
SPASE metadata (recently endorsed by COSPAR resolution)
PRBEM (needs updating)
IRENE Workshop, Sykia, Greece

3. ODI overview
29/05/2019
Open Data Interface (ODI) is a database system for retrieving, processing and storing space environment (and other) data and metadata in a MySQL (MariaDB) database.
Development started in 2008 (Swedish Institute of Space Physics, DH Consultancy, Solar Analytics)
Compliant with CDF/ISTP data model (SPASE in progress)
Server and client software for data downloading and pre/post-processing, and data retrieval.
Currently support for ~250 datasets.
Extensible with user added functionality.
Available on the ESA European Space Software Repository (
IRENE Workshop, Sykia, Greece

4. ODI applications ESA projects EC FP7 projects
29/05/2019
ESA projects
SEPEM, SEDAT, SPENVIS, SAAPS, JHelioViewer, VSWMC
SRREM, RENELLA, VALIRENE, GALEM, ESPREM, ecIRENE
SSA P2-SWE-II
EC FP7 projects
SEPServer
SPACECAST, SPACESTORM
EURISGIC
IRENE Workshop, Sykia, Greece

5. ODI server software
29/05/2019
PHP engine for database communication and process flow
Download and parser scripts
NASA/GSFC CDFexport tool to ingest CDF data
Python tool to ingest NetCDF data
Ingestion of FITS and PDS datasets is under development
Dedicated parsers for other file types
Coordinates for Earth orbiting spacecraft
Download of TLEs from
Generation of spacecraft coordinates (GEI) using NASA/JPL SPICE library (
Magnetic coordinates for spacecraft in Earth’s magnetosphere using UNILIB
cron/quartz setup for automatic download and ingestion
User may add functionality triggered by hooks on download and ingestion
IRENE Workshop, Sykia, Greece

6. Datasets supported in ODI distribution (~250)
29/05/2019
ACE: archive and real time EPAM, SWEPAM, MAG, SIS
DSCOVR: real time IMF and plasma data (JSON streams)
GOES: archive and real time SEM/EPS data
SREM: PROBA1, Integral, GioveB, Rosetta, Herschel, Planck
Magnetic and solar indices (Kp, Dst, F10.7, ISN, OMNI, …)
Interplanetary particle datasets: HELIOS, IMP8, Voyager, Pioneer, Wind
Radiation belt missions: AZUR/EI-88, S3-3/PT, CRRES/MEA/HEEF/PROTEL, UARS/PEM, SAMPEX/PET, NOAA/POES/SEM2, XMM/ERMD, PROBA-V/EPT, RBSP/HOPE/MAGEIS/REPT/RPS, HIMAWARI/SEDA
“Proprietary” datasets: MIR/REM, STRV1B/REM, AMPTE/UKS, EQUATOR-S, ISEE1/WIM/KED, Meteosat/SEM, Galileo/EMU
IRENE Workshop, Sykia, Greece

7. ODI client software APIs for php, Java, IDL, Matlab, Jython, Python
29/05/2019
APIs for php, Java, IDL, Matlab, Jython, Python
Standardised procedure syntax
Outputs in language specific objects
Remote connection over HTTP/REST (server/client), JSON/CSV output
HAPI (NASA Heliospheric API: server/client
Excel interface
IRENE Workshop, Sykia, Greece

8. HAPI Heliophysics Application Programmer’s Interface (HAPI)
29/05/2019
Heliophysics Application Programmer’s Interface (HAPI)
NASA developed RESTful API standard for time series data
Four required endpoints (GET only)
Describe server capabilities (output formats)
List the catalogue of datasets
Show information about a dataset (metadata)
Stream data content for a selected dataset
IRENE Workshop, Sykia, Greece

9. 29/05/2019
IRENE Workshop, Sykia, Greece

10. Discussion points Updates of PRBEM standard
29/05/2019
Updates of PRBEM standard
Introduction of ‘FESA’ in RBSP datasets, should be ‘FExS’
Strict definition of coordinate systems
Introduction of SPASE metadata
Extensions of SPASE metadata
Update of the dictionary with radiation quantities (e.g. dose)
Introduction of ‘dependent variable’
Specification of individual energy bins in energy spectra
HAPI standard extensions
Does not allow for quantified queries, e.g. using WHERE or AVG
Or, PRBEM defined set of REST keywords?
Release of post-processed data after peer review
IRENE Workshop, Sykia, Greece