1. A. Della Vecchia, D. Guerrucci, M. Albani (ESA)
Federated Earth Observation (FedEO)
CEOS WGISS Meeting #46
A. Della Vecchia, D. Guerrucci, M. Albani (ESA)
Yves Coene (Spacebel)
23/10/2018

2. Outline Introduction Activities & Evolution Metrics
Software refactoring
ESA Catalogue TTO within Common Service
IDN metadata population
WGISS Data Asset
Metrics

3. FedEO: Federated Earth Observation Gateway System
FedEO = Federated Earth Observation missions access
The FedEO system provides a unique entry point to a growing number of scientific catalogues and services.

4. WGISS Connected Data Assets

5. Outline Introduction Activities & Evolution Metrics
Software refactoring
ESA Catalogue TTO within Common Service
IDN metadata population
WGISS Data Asset
Metrics

6. Software Refactoring – Objectives
Optimization of the gateway and the catalog - quicker time response
Optimization of the dataset metadata ingestion job – faster ingestion time
Porting all FedEO components to Docker and Kubernetes – fast and easy deployment and horizontal scalability
Preserving all functional/interoperability requirements

7. Data Ingestion

8. Performance
Google Cloud Platform (2017) N=3 .. N=9 ESA Cloud Platform (2018) N=4

9. Time Response 9N Std-2 10M slower than 9N Std-2 1M (0.27sec) (0.8sec)
(12M real index files)
3N Std-4 faster than 3N Std-2, 1M entries
Interoute (4N std4 - 12M) Google (9N st2 - 10M)
Google Cloud Platform
ESA cloud

10. Software Refactoring – Results
New FedEO SW boost significantly metadata ingestion (+400% up to 20M entries) and time response (3x up to 10x faster wrt concurrent users).
New FedEO SW preserves all the current functional/interoperability requirements
New FedEO SW will be deployed at ESA beginning 2019

11. Outline Introduction Activities & Evolution Metrics
Software refactoring
ESA Catalogue TTO within Common Service
IDN metadata population
WGISS Data Asset
Metrics

12. ESA Collaborative Environment
To provide the ESA PDGS with a set of interoperable services permitting the users to:
Access to missions/platforms information supported by a common ontology
Discovery and, if applicable, direct download of EO data:
Copernicus Missions (e.g., Sentinels)
Third Party Missions - TPMs (e.g. SPOT, Landsat …)
Heritage Missions - HMs (e.g., ERS-1/2, ENVISAT instruments …)
Earth Explorer – EEs (e.g., SMOS, Cryosat, SWARM, …)
International repositories (e.g., NASA CMR, CEOS IDN)
Discovery and access to basic services (e.g., datacube):
Browse/visualization tools and time series extraction
EO data extraction, resampling and reprojection
Hosted Processing for authorised users/communities (e.g., CAL/VAL)

13. Core Services close to the data
M2M Interfaces
Online Data Storage
Distribution Facility
Catalogue Clients
Web Service Interface (e.g., OADS, ftp, http)
Hosted Processing
Data/Service Catalogue
Remote Desktop Access
- CAL-VAL Activities
- Access Point for Application Platforms
EO-SIP
VM – Browse Images Generation
VM – DataCube Engine/API
Specific Web Portal:
- SWARM
- SMOS
- Cryosat
- External
Thesauri Service
Query
Population
Multi Mission Portal
– ESA eoli
Data Access
Information Page:
- ESA EO Gateway
TPMs
HMs
EEs
VM – CAL/VAL Processors
International EO Gateway Information Page:
- CEOS IDN
Core Services close to the data
EO Data Visualization and pre-analysis clients
ESA PDGS Data Cube

14. ESA Catalogue TTO by Q1 2019 Core Services close to the data
M2M Interfaces
Online Data Storage
Distribution Facility
Catalogue Clients
Web Service Interface (e.g., OADS, ftp, http)
Hosted Processing
Data/Service Catalogue
Remote Desktop Access
- CAL-VAL Activities
- Access Point for Application Platforms
EO-SIP
VM – Browse Images Generation
VM – DataCube Engine/API
Specific Web Portal:
- SWARM
- SMOS
- Cryosat
- External
Thesauri Service
Query
Population
Multi Mission Portal
– ESA eoli
Data Access
Information Page:
- ESA EO Gateway
TPMs
HMs
EEs
VM – CAL/VAL Processors
International EO Gateway Information Page:
- CEOS IDN
Core Services close to the data
EO Data Visualization and pre-analysis clients
ESA PDGS Data Cube
ESA Catalogue
TTO by Q1 2019

15. Third Party & Earth Explorer & Heritage Missions
Visualisation Layer
Metadata Layer
Data Layer
Google/Qwant
Search
ESA Gateway &
Collection Catalogue
ERS SAR
ENVISAT ASAR
SciHub
Sentinel-1/2/3
Sentinel Data Repository
ESA
Third Party & Earth Explorer & Heritage Missions
SMOS
CRYOSAT-2
LANDSAT
SPOT
OCEANSAT
TROPFOREST
SEASAT
IKONOS
ESA Earth OnLine
ESA Map Viewer
Copernicus Dataset
EO Products Discovery
CCMs
Repositories
Catalogue Clients
Catalogue API publicly available:
OGC OpenSearch Specification
CEOS WGISS OpenSearch Best Practice
Distribution Facility
Online Data Storage

16. ESA Catalogue TTO – Results
ESA Catalogue shall:
be the centralised metadata repository of ESA Collaborative Environment
reuse same FedEO SW
manage collections Digital Object Identifiers (DOIs)
be part of CEOS WGISS Data Asset via FedEO

17. Outline Introduction Activities & Evolution Metrics
Software refactoring
ESA Catalogue TTO within Common Service
IDN metadata population
WGISS Data Asset
Metrics

18. Metadata Export into IDN – Today
20 ESA collections, providing two step search, today on IDN via FedEO DIF-10 generator

19. FedEO Metadata Mediator – Ongoing
Automatic Procedure – beginning ’19 on FedEO operational environment at ESA
Partner Metadata repository
Metadata Import
Harvester tool
FedEO Collection Catalogue
FedEO Gateway
ISO to DIF-10
Metadata Mediator
Metadata Preparation
IDN
Complementary
Information gcmd keyword
Metadata Preparation
IDN guideline for Information Content completeness and consistency
ESA Thesauri Service
DIF-10 Validator
Metadata Export
DIF-10 Encoding
IDN
repository
DIF-10 Validation

20. Metadata Export into IDN – Ongoing
A fully automatic metadata mediator is under development and testing. New collections almost ready on development platform at Spacebel In BLUE collections ready to be uploaded to IDN In RED partners where technical contacts for IDN population update through FedEO procedure need to be started
Repository
Collections
Verified
To be Verified
ESA
170 20
150
Copernicus Sentinel 5 -
DLR
184 66
118
EUMETSAT
734
CNES 8
ROSCOSMOS 28
VITO 31 11
JAXA 45
ESA CCI
125 48 77
CMEMS 2
168

21. Open Issues with DIF-10
Metadata Preparation (
Completeness
Missing values for instrument/platform.
Correct DIF-10 values for “project name”
Consistency
Multiple DIF-10 GCMD platforms/instruments keywords appear, required explicit keywords relation
Use of “GOME” (ERS-2) while actually GOME-2 is meant (METOP), METOP-AB instead of specifying METOP-A and/or METOP-B, etc…

22. Open Issues with DIF-10
NASA DIF-10 Validation (
ISO MIME type “application/xml” valid instead of “application/vnd.iso xml”, recommended by CEOS OpenSearch Best Practice 1.2 [CEOS-BP-012C] NASA extended MIME Type
Invalid Keyword Relation Issue with relationship between Platform and Instrument keywords. Error message even with DIF-10 files were OK in the past (e.g., 20 OADS ESA files). GCMD vocabularies do not provide skos attributes to link platform and instrument concepts. In some (rare) cases, the skos:definition contains some formatted text (see ERS-1 Example) which refers to an instrument or platform as text. Even in this case, the GCMD UUID (the only thing which is not ambiguous) is not mentioned. NASA confirmed GCMD vocabularies do not provide platform/instrument relationship. Keyword Management Server (KMS) to be updated to check and manage consistently CEOS platform/instrument relationship , by 2019.

23. Open Issues with DIF-10
Consistency between DIF-10 Writer Page wrt Validation Tools (DocBuilder / IDN DIF-10) and (QAViewer / CMR validation API). DocBuilder defines Required, Highly Recommended and Recommended fields, but there is some gray area on the subfields.
GCMD DocBuilder
GCMD DIF Guide
IDN DIF10 Guide
E.g.: According to IDN guide, Platform field is mandatory, nowhere is specified if “Short Name” and “Long Name” subfields are mandatory. According to DocBuilder, and NASA support, it is assumed that Platform is mandatory, Short name mandatory and Long Name optional (also due to missing values from GCMD vocabularies, e.g., Sentinel-1 example).
A clear table listing DIF-10 fields and subfields, pointing to authorized values (e.g., GCMD vocabularies), defining related cardinality (Optional/Mandatory/Sinlge/Multiple values), consistent with validation SW is required for unsupervised DIF-10 production (e.g., FedEO)

24. DIF-10 Next Steps
ESA reports to NASA about understanding of mandatory/optional DIF-10 fields, and identified inconsistencies between DIF-10 Writer Guide and DIF-10 validators
FedEO Ingestion tool shall be enhanced (Q1 2019) to generate a log file showing mapping information and let it generate a human readable “Ingestion report”. This will simplify the internal process of metadata owner to make the metadata IDN ready, passing through FedEO.
Proceed with systematic European Partner metadata ingestion into FedEO and automatic export into IDN

25. Outline Introduction Activities & Evolution Metrics
Software refactoring
ESA Catalogue TTO within Common Service
IDN metadata population
WGISS Data Asset
Metrics

26. CEOS Connected Data Asset
Scenario 1 – M2M Interface APIs allows two steps search to external clients aligned to CEOS OpenSearch BP 1.2 Scenario 2 – GUI Interface CEOS Connected Data Access allow the users to discover/access both collections and products metadata

27. Scenario 2 – GUI Interface
ESA Collection Landing Page
Second Step OSDD
FedEO Client
ISO Metadata

28. Outline Introduction Activities & Evolution Metrics
Software refactoring
ESA Catalogue TTO within Common Service
IDN metadata population
WGISS Data Asset
Metrics

29. Metrics See slide 20 * TotalResults is not returned by catalog.
** Step 2 under construction.
See slide 20