1. ESA Scientific Archives and Virtual Observatory Systems Christophe.Arviset@esa.int Science Archives and VO Team Research and Scientific Support Department ESA – ESAC – Madrid, Spain

2. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 2 Research & Scientific Support Department European Space Astronomy Centre  ESAC default location for:  Science operations,  long history with astronomical missions,  now expanding with solar system missions.  Science archives,  Astronomy  Planetary  ESA VO activities,  ESAC to be the European VO node for space-based astronomy. http://www.esa.int/SPECIALS/ESAC/ Located near Madrid, Spain

3. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 3 Research & Scientific Support Department Archives at ESAC  ESAC is the Centre where most of ESA Scientific Directorate’s Scientific Archives are developed, maintained and operated.  ESAC Science Archives Team is giving support to various projects ISO Data Archive Since December 1998 XMM-Newton Science Archive Since April 2002 Integral SOC Science Data Archive Since July 2005 Planetary Science Archive Giotto, Mars Express Rosetta, Venus Express Smart-1, Huygens Since March 2004 Herschel, Planck, GAIA, … in the future

4. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 4 Research & Scientific Support Department Archives at ESAC (cont’d)  Scientific Archives geared to various users  Scientific Community (public access)  PI team and observers (controlled access)  Science Operations Team (privilege access)  Common Architecture and Look and Feel  Better corporate image for ESA  Faster, cheaper, better development  Re-use of expertise  User friendly access  Eg PSA Mars Map Browser

5. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 5 Research & Scientific Support Department Science Archives Team @ ESAC  A core Science Archives and VO Team (~15 people) in ESAC  Support many projects (horizontal support vs vertical organization) ISO, XMM-Newton, Integral, Planetary missions, Herschel, Virtual Observatory Soho, Planck, Gaia in the future…  Organized by functions: database, user interface, data distribution, inter- operability  Close collaboration with Archive Scientist (1 per project)  Expert of the science field, close to the community  Consolidate user requirements  Set development priorities  Final acceptance tests  Strong re-use of design/system/code between projects  Not responsible for writing data reduction software (pipeline) nor data production

6. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 6 Research & Scientific Support Department Code re-use across Archives  Manpower shared between projects  Knowledge transfer between projects  Small fraction of FTE for “old” projects  All Archives share a common part of the code  Same functionalities across projects  But still some very specific projects functionalities  Faster and reliable development of new projects  Easier and cheaper maintenance of existing systems  Same Look & Feel for ESA Science Archives User Interface  Powerful and easy to use  Better corporate image for ESA  Easy learning curve for users

7. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 7 Research & Scientific Support Department Data Repository  All Data from all archives are stored on magnetic disks  Standard formats (FITS for astronomy, PDS for planetary)  Save various levels of data  IDA (0.5TB), XSA (3TB), ISDA (6TB), PSA (4TB)  Small amount still  Unique disk repository for proprietary and public data  Access controlled by the application server  Access to data is immediate  Via the Archive User I/F or for the VO  Except for on-the-fly reprocessing requests  Clever FTP server using links instead of real files  Same disk space for FTP  Higher quotas for users

8. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 8 Research & Scientific Support Department DataBase  Strong effort of Data Modeling before starting any project  Unique database for all uplink and downlink metadata  All metadata is systematically extracted from uplink AND downlink data  Done when products are generated  Can be re-run with for new metadata is user reqs change  Sybase for some projects  ISO (10GB), XMM-Newton (6GB), Planetary (2GB)  Oracle for others  Integral (6GB)  PostGres for future projects (Herschel, Soho, …)

9. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 9 Research & Scientific Support Department Existing Access – GUI Project Business Logic Translation Layer Meta Data in Database Data Products on hard disks Product Request Products GUI Database/Product Request GUI Database Result / Product DB/Product Request DB Result / Product  User friendly web access : Java Applet  Powerful queries, organized by panel, hundreds of queriable and displayable parameters  Public data accessible to all, proprietary data accessible only to privilege users  Images preview (icons, full image)  Quick download (1 click) at various processing levels, shopping basket retrieval  Remote access to other archives (ADS, and tools (Aladin, VOSpec)  Project prior to the VO  Existing data model, data format, access mechanisms

10. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 10 Research & Scientific Support Department Existing Access : Scriptable I/F Project Business Logic Translation Layer Meta Data in Database Data Products on hard disks Product Request Products GUI Database/Product Request GUI Database Result / Product DB/Product Request DB Result / Product Scriptable Interface Project Product Request Project Product foreach obs in (observation_list) AIOgetfile (obs, file_required) analyse_process file_required save results delete file_required end DB/Product Request DB Result / Product  Access to metadata (via XML scripting)  Access to data through socket (direct data stream) or FTP (several files or individual file)  Own Interoperability standards and needs as required by the project  Not using VO protocols, but ready for it…

11. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 11 Research & Scientific Support Department VO compliant Architecture Project Business Logic Translation Layer Meta Data in Database Data Products on hard disks Product Request Products GUI Database/Product Request GUI Database Result / Product DB/Product Request DB Result / Product Project Product Request Project Product DB/Product Request DB Result / Product SIAP/SSA Request VOTable / Product Translation Layer Scriptable Interface Project Product Request Project Product (Scriptable Interface) foreach obs in (observation_list) AIOgetfile (obs, file_required) analyse_process file_required save results delete file_required end DB/Product Request DB Result / Product  Existing scriptable I/F already carried all the functionalities required by the new VO standards  Access to metadata (database)  Access to products (data repository)  By using “Translation Layer”, easy to convert the VO standards interfaces into our existing I/F  Little effort required  Translation Layer easy to code using XML  As VO standards evolve, our scriptable I/F system remains, but we just have to adapt our translation layers

12. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 12 Research & Scientific Support Department ESA-VO Main Goals  ESA part of the EURO-VO  ESA part of the IVOA (International VO Alliance)  RSSD astronomical archives being VO compatible  VO in mind when building archives  Archives and VO work done by the same team  Develop VO applications  eg VOSpec, VOQuest, …  Develop VO publishing services  Registry of VO resources,  DALToolkit, DMMapper, …  ESAC to disseminate VO technology  Link to existing and new ESA astronomy projects  VO outreach (workshop, conferences, …) http://esavo.esac.esa.int/

13. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 13 Research & Scientific Support Department VO in action : VOSpec application Select Source Find/Select in the VORegistry VO Spectral Servers IUE spectra ISO spectra VEGA Theoretical Model Display Theoretical Spectra and Observational Spectra IUE spectra ISO spectra VEGA Theoretical Model Normalize Theoretical Spectra IUE spectra ISO spectra VEGA Theoretical Model Theoretical Spectra Fits Observational Spectra

14. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 14 Research & Scientific Support Department Interoperability ESA PSA – NASA PDS  (Prototype in development) in the context of the IPDA (International Planetary Data Alliance)  From Mars Map Browser, Select region of interest  Contact PSA and PDS using the common data (image) access protocol  Display NASA PDS and ESA PSA images

15. Christophe ARVISET Science Archives in the 21st Century, 26/04/2007, page 15 Research & Scientific Support Department Conclusion  ESA Scientific Archives built and operated within a same team  ESA Archives flexible architecture allows easier and long-term maintenance  ESA strongly believes in the Virtual Observatory, both for astronomy and for planetary missions  Strong link between Archives development and Virtual Observatory work  ESAC is becoming the Data Centre for ESA space-based archives  ESAC is becoming the VO node for European space-based astronomy