Presentation is loading. Please wait.

Presentation is loading. Please wait.

ESA UNCLASSIFIED – For Official Use ExoMars ESWT#7 10 Dec 2014 1 ESWT #7 Data Archive Overview of the Data Archive Plan by Luc Joudrier & Dave Heather.

Similar presentations


Presentation on theme: "ESA UNCLASSIFIED – For Official Use ExoMars ESWT#7 10 Dec 2014 1 ESWT #7 Data Archive Overview of the Data Archive Plan by Luc Joudrier & Dave Heather."— Presentation transcript:

1 ESA UNCLASSIFIED – For Official Use ExoMars ESWT#7 10 Dec ESWT #7 Data Archive Overview of the Data Archive Plan by Luc Joudrier & Dave Heather

2 ESA UNCLASSIFIED – For Official Use Data Archive Plan The Data Archive Plan describes the process of archiving the data for the 2018 ExoMars Rover Mission. It is derived from the 2016 ExoMars Data Archive Plan.  It governs the interface between ROCC and Planetary Science Archive at ESAC.  The creation of the data to be archived is handled by ROCC.  Mirroring of the data to the Russian PSA will be handled by ESAC-PSA. ExoMars Project Data Archive Plan ROCC – ESAC ICD ESAC PSA IKI Mirror Archive ROCC Science Teams ROCC – PPL ICDs

3 ESA UNCLASSIFIED – For Official Use Which PDS Format ?  New missions, starting from ExoMars 2016 mission will be compliant to PDS4, version 4 of the NASA’s Planetary Data System.  ExoMars 2016 SOC developed at ESAC is gaining experience with this new format by developing PDS4 raw data pipeline, reusing the concepts of GDP. This experience is being shared with ALTEC and ultimately fed into the ROCC to PPL ICDs discussions.  Whether data are produced at ROCC or not (advanced products may be produced with support of the PI’s home base), they must be delivered to ROCC that is the entity who will deliver them to ESAC-PSA. ExoMars Project 3

4 ESA UNCLASSIFIED – For Official Use Science Data Deliverables  Science data deliverables are divided into primary products and supplementary products  Primary products contain the scientific data resulting from observations, including housekeeping and calibration data if applicable  Supplementary products include all non-observational data required directly or indirectly in the processing and analysis of the observational data.  Science data are delivered within a PDS bundle (PDS4 naming convention, former PDS3 dataset) ExoMars Project From: ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 4

5 ESA UNCLASSIFIED – For Official Use Primary Products - Data Processing Levels  Telemetry data (ROCC)  Original binary stream from an instrument encoded in telemetry packets.  Raw data (produced at ROCC with PI’s pipelines)  Original data from an instrument. If compression, reformatting, packetization, or other translation has been applied to facilitate data transmission or storage, those processes will be reversed so that the archived data are in a PDS approved archive format.  Partially processed data (produced at ROCC with PI’s tools)  Data that have been processed beyond the raw stage but which have not yet reached calibrated status.  Calibrated data (produced at ROCC with PI’s pipelines)  Data converted to physical units where all of the calibration based on the instrument has been done.  Derived data (PI teams for scientific products, ROCC for Engineering products)  Results that have been generated from one or more calibrated products (for example: mosaics, maps, resampled data, fits to data models, temperature profiles etc…).  Calibration tables or tables of viewing geometry, used to interpret observational data should also be classified as ‘derived’ data if not easily matched to one of the other three categories. Engineering products (mobility, Drilling, SPDS) will be classified as Derived. From: ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 5 ExoMars Project

6 ESA UNCLASSIFIED – For Official Use Supplementary Products - Types  Calibration  Calibration products include all data used at any stage of the calibration process. They can include, for example, observations of an on-board hardware calibrator or an external target used for calibration, or instrument response curves and bad pixel maps.  Geometry  Geometry products include pointing, orientation and positioning information. These pieces of information are linked to the localisation (position and attitude) of the rover.  Browse  Browse products are browse representations (thumbnails) or quick-look plots/reports of the observational products.  Document  Document products include all documentation or supporting information to assist in understanding and using the Primary Products. The specific contribution of each instruments to the ROCC to ESAC ICD is a required document product – This contribution is initially formulated in the ROCC to PPL ICDs (equivalent to Experimenter to Archive Interface Control Document (EAICD) naming convention of the 2016 Data Archive Plan). ExoMars Project From: ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 6

7 ESA UNCLASSIFIED – For Official Use Supplementary Data Types  Miscellaneous  Miscellaneous products include any additional information and documentation products generated (e.g. tables to track planned and executed observations, spacecraft manoeuvres, pointing modes, ground-station communication passes, target identification, spacecraft and hardware anomalies), if not easily classified as one of the above-described Supplementary products. Anything that might affect, influence or offer an insight into the interpretation or understanding of the science data can be included in the archive.  Context  Context products include descriptions of the mission, spacecraft, instrumentation and targets in sufficient detail to give the user an insight into the nature of the data returned and the calibration processes used. From: ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 7 ExoMars Project

8 ESA UNCLASSIFIED – For Official Use Which process? 1/2  Short Term  Define which data are generated for science and support of operations (Draft ROCC to PPL ICDs)  Medium Term  Definition of data types, data structures and the data set organization for each instrument -> Refine ROCC to PPL ICDs so to prepare the ROCC to ESAC ICD  Agreement of the format of the generated products -> freezing of ROCC to ESAC ICD  Implementation of the data pipelines generating the agreed products (part of the archive to be delivered to PSA)  Readiness of ROCC: Verification of the generation of the data products.  Archive Working Group will be organized among ALTEC – PPL Teams – ESAC- PSA - ESA as mean to support and monitor the above process.  Schedule to be agreed. ExoMars Project 8

9 ESA UNCLASSIFIED – For Official Use Which process? 2/2  At the time of operations  Use of the data pipelines and generation of the products in the agreed format  Regular Peer Reviews of the data products generated over a period of 1 (TBC) month. The Peer Reviews are organized by the Project Scientist at ROCC.  Regular Release of the Peer- Reviewed data to ESAC PSA  Mirroring of data to IKI  Release to public of the data products after the 6 months of proprietary period. ExoMars Project 9

10 ESA UNCLASSIFIED – For Official Use ExoMars Project 10

11 ESA UNCLASSIFIED – For Official Use ExoMars Project 11

12 ESA UNCLASSIFIED – For Official Use COPY OF THE ESWT #6 PRESENTATION ABOUT 2016 ARCHIVE PLAN. FOR INFORMATION ONLY ExoMars Project 12

13 ESA UNCLASSIFIED – For Official Use 6 th Feb 2014 Nicolas Manaud, Dave Heather, Olivier Witasse ExoMars 2016 Data Handling and Archiving Working Group Meeting #1

14 ESA UNCLASSIFIED – For Official Use Agenda 1.Introduction (N. Manaud)  Working Group / points of contact  ExoMars 2016 Science Archive  Activities and Process  Quality Control Process  Data Rights and Release Policy 2.Introduction to the PSA/PDS4(D. Heather) 3.Data Handling and Archiving Concept(N. Manaud)  Science Data Deliverables  Data Flow Overview  Scoping Data Processing Effort 4.Instrument Required Support(PI teams) 5.Combined theme-based data products(O. Witasse) 6.Upcoming work schedule(N. Manaud) 7.Discussions/AOB(All) ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 14

15 ESA UNCLASSIFIED – For Official Use DHA Working Group  This working group is the forum for discussion and agreement on activities related to science data processing and archiving, including associated services.  Composition  Archive Scientists (Chairman)  Nicolas Manaud (ESAC)  Oleg Batanov (IKI)  PI representatives  PSA Coordinator  David Heather (ESAC)  Project Scientists  Olivier Witasse (ESTEC)  Daniel Rodionov (IKI)  ESAC SOC S/W Engineers  Antonio Vilacorta (ESAC)  Jon Brumfitt (ESAC) ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 15  Frequency of meetings combined with SWT/SOWG regular DHAWG telecons. ad-hoc meetings (as needed)

16 ESA UNCLASSIFIED – For Official Use Points of Contact ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 16 Archive ScientistNicolas Manaud (ESAC) Instrument Teams TGO ACS Alexey Shakun CaSSIS Nick Thomas FREND Alexey Malakhov NOMAD Eddy Neefs (IR) Jon Mason (UVIS) EDM AMELIA Francesca Ferri (UNIPD-CISAS, DECA Detlef Koschny (?) DREAMS Pietro Schipani (INAF-OAC,

17 ESA UNCLASSIFIED – For Official Use ExoMars 2016 Science Archive  ExoMars TGO and EDM data will be archived into the ESA’s Planetary Science Archive (PSA) following existing process and standards from previous ESA’s planetary missions.  ExoMars data will be compliant to PDS4, version 4 of the NASA’s Planetary Data System.  ESA/PSA is the archiving authority  Agreements at management level between IKI and ESA  Russian Deep Science Archive will contain a duplicate of the ESA’s ExoMars Science Archive ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 17

18 ESA UNCLASSIFIED – For Official Use DHA Activities and Process Overview 1.Pre-Launch Preparation  Planning: Science Data Generation, Validation and Archiving Plan (“Archive Plan”)  Plan for producing, validating, archiving and disseminating all EM16 science data  Design: Experiment to Archive Interface Control Document (EAICD)  Definition of data types, data structures and the data set organisation for each instrument  Development  Development and/or monitoring of data processing pipeline development 2.Routine Operations  Data Delivery and Validation  Quality Control  Data Access and Exploitation 3.Post-Operations and Legacy ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 18

19 ESA UNCLASSIFIED – For Official Use Quality Control Process Overview  Formal reviews of the science data ingested in the archive will be organised by the ESAC SOC, in coordination with the IKI SOC and the Project Scientist(s). The review is a three-step process consisting of: 1.Review of the Science Data Generation, Validation and Archiving Plan and the individual Experiment-to-Archive Interface Control Documents (EAICDs), including (if possible) sample data and processing pipelines documentation (Archiving Documentation Review). 2.Review of the first data delivery (Initial Peer Review). 3.Review of all data together after the final delivery (Final Peer Review)  After successful Initial Peer Review, successive data deliveries will be syntactically validated prior to ingestion into the PSA to guarantee PDS compliancy.  Scientific quality of observational data will be assessed by the Instrument Team and reported in data products meta-data ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 19

20 ESA UNCLASSIFIED – For Official Use Data Rights and Release Policy  Instrument teams are granted a proprietary period of 6 months for the generation, validation and analysis of their data. Once data proprietary period is over, science data products will be publicly released through the PSA, and IKI Deep Science Archive.  A data collection period of 1 month is defined to facilitate the communication with the Instrument Teams for notification/confirmation before science data products are made publicly available. The data collection period defines the content of each data delivery to the PSA. ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 20

21 ESA UNCLASSIFIED – For Official Use Science Data Deliverables  Science data deliverables are divided into primary products and supplementary products  Primary products contain the scientific data resulting from observations, including housekeeping and calibration data if applicable  Supplementary products include all non-observational data required directly or indirectly in the processing and analysis of the observational data.  Science data are delivered within a PDS bundle (PDS4 naming convention, former PDS3 dataset) ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 21

22 ESA UNCLASSIFIED – For Official Use Primary Products - Data Processing Levels  Telemetry data (SOC)  Original binary stream from an instrument encoded in telemetry packets.  Raw data (SOCs and/or PI teams)  Original data from an instrument. If compression, reformatting, packetization, or other translation has been applied to facilitate data transmission or storage, those processes will be reversed so that the archived data are in a PDS approved archive format.  Partially processed data (PI teams)  Data that have been processed beyond the raw stage but which have not yet reached calibrated status.  Calibrated data (PI teams)  Data converted to physical units where all of the calibration based on the instrument has been done.  Derived data (PI teams for scientific products, SOC for SPICE)  Results that have been generated from one or more calibrated products (for example: mosaics, maps, resampled data, fits to data models, temperature profiles, gravity or magnetic fields, or ring particle size distributions).  Calibration tables or tables of viewing geometry, used to interpret observational data should also be classified as ‘derived’ data if not easily matched to one of the other three categories. Navigation products (including SPICE kernels) will be classified as Derived. ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 22

23 ESA UNCLASSIFIED – For Official Use Supplementary Products - Types  Calibration  Calibration products include all data used at any stage of the calibration process. They can include, for example, observations of an on-board hardware calibrator or an external target used for calibration, or instrument response curves and bad pixel maps.  Geometry  Geometry products include pointing, orientation and positioning information. Geometry information can be provided in two formats (1) as calculated values (e.g. pixel-by-pixel geometry parameters computed for each image/product in the data set), and (2) as SPICE kernels, which can be used with the NAIF SPICE software toolkit to calculate the appropriate values.  Browse  Browse products are browse representations (thumbnails) or quick-look plots/reports of the observational products.  Document  Document products include all documentation or supporting information to assist in understanding and using the Primary Products. The Experimenter to Archive Interface Control Document (EAICD) is a required document product. ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 23

24 ESA UNCLASSIFIED – For Official Use Supplementary Data Types  Miscellaneous  Miscellaneous products include any additional information and documentation products generated (e.g. tables to track planned and executed observations, spacecraft manoeuvres, pointing modes, ground-station communication passes, target identification, spacecraft and hardware anomalies), if not easily classified as one of the above-described Supplementary products. Anything that might affect, influence or offer an insight into the interpretation or understanding of the science data can be included in the archive.  Context  Context products include descriptions of the mission, spacecraft, instrumentation and targets in sufficient detail to give the user an insight into the nature of the data returned and the calibration processes used. ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 24

25 ESA UNCLASSIFIED – For Official Use Data Flow Overview ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 25

26 ESA UNCLASSIFIED – For Official Use Scoping Data Processing Effort  Your Raw Data Generation Primary Pipelines options: 1.Instrument SOC 2.ESAC SOC  In this case we also offer to generate PDS bundles. 3.Russian SOC  Current understanding  ACS using Russian SOC Pipeline (TBC)  CaSSIS using ESAC SOC Pipeline (confirmed)  FREND using ESAC or Russian SOC Pipeline (TBD)  NOMAD using Instrument SOC (confirmed)  IR channels: existing TM processing pipeline (PDS4 adaptation needed)  Other channels will implement the same approach ExoMars 2016 Data Archiving | N. Manaud | DHAWG#1 | 6 Feb | ExoMars 2016 | Pag. 26


Download ppt "ESA UNCLASSIFIED – For Official Use ExoMars ESWT#7 10 Dec 2014 1 ESWT #7 Data Archive Overview of the Data Archive Plan by Luc Joudrier & Dave Heather."

Similar presentations


Ads by Google