1. Advanced Data Products at ESO P.Rosati (DMO/VOS) with special thanks to: B.Vandame, C.Rite, J.Retzlaff, R.Slijkhuis, M.Nonino Faculty Meeting - Dec 13, 2006

2. The VO System Dept. @ESO was created in Nov 2004 with the aim of transforming the SAF into a powerful scientific resource for the ESO and world-wide community: 1. Powerful and user-friendly Archive Interface 2. Populate the SAF with a large number of Advanced Data Products, i.e. science ready calibrated data (both imaging and spectra), and a much larger number of intermediate- level data products 3. Make the ESO Archive compliant with VO standards to allow efficient flow of SAF content and data products from/to other VO facilities How to boost Science with the ESO Science Archive Facility (SAF)

3. ESO Science Archive: user top-wish-list Science related needs of Archive Users ESO Archive User Survey: ESO Messenger 125,41 (Sep 06) http://archive.eso.org/archive/stats/survey/survey_results.htmlhttp://archive.eso.org/archive/stats/survey/survey_results.html

4. Users who published based on ESO archival data ~>3000 programs ~100 progs ESO Science Archive: scientific exploitation (Source ESO Archive User Survey) Raw ADP Publication rate/potential of archival data for non PIs/coIs: nature of the data Similar results are found from HST data

5. 5 The Challenge of HLSDPs The creation of a large number High Level Science Data Products from ground-based Archives is very challenging: Photometric calibration, large number of instruments/modes, non-homogeneous observing conditions/strategies Even harder for MOS spectroscopy, but feasible for 3D spectroscopy On-going ground-based efforts deal solely with (public) surveys, e.g.: NOAO, CFHLS, UKIDSS, SLOAN (also spec),... For space observatories, the task of producing HLSDPs from generic archival programs is easier. On-going efforts include HST, ISO, Chandra, XMM, Spitzer, etc. with plans to publish data products with VO standards First efforts have focused on Large Programs and Surveys to face the complexity and volume of the data sets and to guarantee their legacy value

6. 6 GOODS-like (highest level) Data Products Hierarchy @ESO ADP Raw Data Advanced Stacks Selected QC products 10 4 10 2 Scientific Value Processing Level SAF 10 Ingestion of UVES DPs first Gradually all other QC products

7. ESO provides pipeline “recipes” for all VLT instruments They remove the instrumental signature and are used for Quality Control at ESO and distributed to the community The Quality Control group processes of all pipeline-supported modes for the VLT/VLTI Service Mode stream (virtually 100% of the data volume) to ensure that science data can be calibrated to a known and documented level These reduced data are distributed to PIs and soon will be archived for general consumption from the community at large Ditto for HARPS and FEROS on La Silla and APEX QC products (from Martino)

8. 8 GOODS-like (highest level) Data Products Hierarchy @ESO ADP Raw Data Advanced Stacks Selected QC products Advanced Data Products Selected GO Products Large Programmes >P75 (e.g. z-Cosmos) Public Surveys (VISTA & VST) ESO/MVM products Special cases (HARPS) 10 4 10 2 Scientific Value Processing Level SAF Outsourced to the community ESO product 10

9. Creating Advanced Data Products with ESO/MVM ESO/MVM (Multi-Vision Model)ESO/MVM (Multi-Vision Model): end-to-end, high-throughput image reduction system, originally developed by B.Vandame as part of the EIS project Seamless processing of optical & near-IR data (WFI, ISAAC, SOFI, VIMOS, FORS2,..) ➡ the work of months is typically reduced to one day ! Major intellectual investment over 10 yrs: 171,000 lines of code (SExtractor: 30,000) Photometric Calibration (mostly a manual process for complex data sets!) SA F SA F SAF Prog-ID Field Definition Sci+ Calib ESO/MVM (Alambic) Data organization (calib, std, sci/filter) Instrum.signature removal Pixel masking/flagging Astrometric registration Stacking on a common astrometric grid Mosaicing Fully processed, science-ready mosaics (with photometric & astrometric calibration) J H K

10. GOODS/ISAAC ADP Data Release (v1.5 June 05, v2.0 Jan 07) Part of the GOODS/ESO Public Programme in CDFS (PI: Cesarsky) New v2.0 final release soon. Deepest large nearIR field carried out to date 160 arcmin 2 in J,H,K, 127 arcmin 2 H (J,H,K: 25.5, 25, 25 AB mag lim at 5  ) Fully processed and calibrated mosaics with ESO/MVM data reduction system (~50,000 raw frames  160 final science-ready frames). A few % accuracy of photometric calibration Intensive use by the worldwide community: it complements HST and Spitzer data for galaxy evolution studies out to z~7 Large number of papers/projects based on this high-quality data set UDF K s - 30h integration - to be released in Q1 07

11. 30 Doradus 2x2 WFI B, V, OIII, H a FoV: 1 deg 2 Input images: 300 No Zero Points Release date: Dec 18, 06

12. SWIRE-South field SOFI, Ks FoV: 1 deg 2 Input images: 3307 Release date: Q1 07

13. 13 Questions for the audience How should we strike a balance between level of data products (photometric calibration accuracy) and their number ? say, tens high-level (few % accuracy), ~100 with 10% accuracy, or thousands of DPs with no or poor photo-calibration? ESO/MVM could be used to scan the entire Archive (imaging instruments) and gradually produce Advanced Stacks: combined images on a common astrometric grid per night/runID/ target/filter + reduced standards, but with no zero point...or should we produce HLSDPs only from a limited sample of programs? which selection criteria should we adopt in such a case?...or should we decide not to produce HLSDPs from the Archive, with obvious repercussions on scientific exploitation of the SAF and the VO ?

14. 14 VOS people