1. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 1 Data Processing Day 9:00 Data Flow System Deliverables and Integration 10:00 Coffee Break 10:15 The ESO Common Pipeline Library 11:00 The ESO-Reflex Environment 12:00 Lunch break 13:00 CPL Tutorial 15:00 Coffee Break / End of Workshop

2. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 2 SDD / Pipeline Systems Department P.Ballester K.Banse S.Castro L. de Bilbao A. Gabasch E. Garcia C. Izzo Y. Jung J. Larsen H. Lorch L. Lundin A. Modigliani R. Palsa D. Petry K. Shabun J. Vinther

3. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 3 Data Flow System Deliverables and Integration

4. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 4 DFS Deliverables (1): Data Reduction Library PSD Paranal Science Operations INS Commissioning Team Data Flow Operations DICB/Archive Public Release User Community

5. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 5 DFS Deliverables (2): ETC/ Observation Preparation Tools

6. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 6 Talk Outline Data Reduction Library Calibration cascade Modular design Data Products Science-grade data products Simulations and validation Project Organisation Project phases and timeline DFS Integration Acceptance tests DFS infrastructure

7. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 7 Data Reduction Library Calibration Cascade Modular Design

8. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 8 Data Reduction Correct for detector and instrument effects Correct for atmospheric effects Separate science data from noise and background Astronomical Calibrators (Position, Spectral flux, Diameter, etc..) Instrumental Calibrators (Internal sources)

9. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 9 Calibration Cascade Detector Bias Instrument and Detector Sensitivity Photometry Standard Star

10. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 10 Shift & Add: HAWK-I Modular Design -Independent modular recipes -Integrated pipeline recipe

11. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 11 Shift & Add: HAWK-I Reflex Workflow

12. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 12 Shift & Add: Demonstration 1. Raw image (sky = 10,000 star = 1) 2. Dome flat 3. Estimating the sky from N jittered exposures 4. Subtracting the sky and correcting for flat-field 5. Co-adding the sky corrected images 6. Astrometry + Photometry !!

13. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 13 Science-Grade Data Products Accuracy Recipes Design Robustness, Fault tolerance Validation

14. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 14 Science-Grade Data Products Calibrated in physical units with error estimates No residual systematic error S/N close to the achievable optimum (e.g. ETC prediction) OB combination E.g. fully calibrated and mosaiced images with error bars Standard data formats SGDPs are independent of the science goals

15. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 15 Understanding the Instrument Signature (1) UVES Adaptive Optimal Extraction UVES archive data reprocessing: Robust to CCD defects S/N adaptive optimal extraction Analytic profile for low S/N Non-parametric profile for high S/N Ripple scale QC parameters

16. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 16 Understanding the Instrument Signature (2) MIDI Visibility Spectra

17. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 17 Designing Pipeline Recipes Image Model Optical/Detector model Assumptions on stability/reproducibility Input Data Signal-to-noise Missing data Contamination Stability (Atm., Ins.) Multiplex/Volume Reduced Data Random error estimates Systematic errors Inverse Solution Objective function No assumptions on scientific program Optimization strategy Computational complexity Robustness, Fault tolerance Pipeline Recipe Default parameters Fixed calibration data Accuracy Throughput Robustness

18. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 18 Ar lamp - 1” slit - from P. Spano et al. in Merate, July 2007 First light in the lab (VIS arm)

19. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 19 Using Simulations Ar lamp - 1” slit - simulated by P. Bristow (ESO)

20. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 20 State-of-the-art DRS pipeline Simulated image, multi-pinhole VIS: curved orders, line tilt varying along order

21. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 21 Robustness, Fault Tolerance Physical Model Pattern Matching

22. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 22 Project Organisation Scheduling Validation and Acceptance

23. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 23 4 Phases Phase A to FDR Documentation/ Prototyping2-3 years FDR to PAE Development/ Simulation2-3 years PAE to SV and SOP Deployment 6-12 months Operational PhaseMaintenance10 years

24. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 24 1618 Document and Technical Documentation 1618 Document (currently v.2.0) CPL Documentation DICB / VLTI DICD Gasgano Reflex

25. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 25 DFS Deliverables: 1618 Template Schedule

26. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 26 DFS Deliverables: 1618 Template Schedule (cont’d)

27. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 27 From Phase A to FDR Phase A and PDR Preparation Reuse existing documents as templates Identify extra needs: observation preparation, visualisation, link to the data analysis? Learning CPL before the FDR First recipes in CPL, coding standards, memory management, … Review existing pipelines, develop prototypes Coding starts officially only after the FDR FDR Preparation Prototype data reduction algorithms (test and simulated data) New document: Validation and Test Plan Observatory Pipeline is template based —Further processing requirements ? —Data combination, interactive processing

28. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 28 From FDR to Science Verification FDR to PAE Plan more import validation effort for the first recipes, In general, plan enough time for testing and finalising Regular intermediate software releases (3 to 6 months), including test reports Keep improving simulated or adapted data, in sync with instrument schedule PAE Plan for the complete set of recipes to be ready at PAE Validated data reduction algorithms (using laboratory and simulated data) Have ready a few alternative calibration/reduction methods Commissioning to Science Verification Test and validate on instrument and sky data Identify and solve the unexpected problems

29. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 29 DICB Approval DPR Keyword Values Ideally, classification rules are only based on DPR keywords Valid values are listed in the DICB 4.0 documents Submit proposal to DICB by FDR before using new values DICB Validation No redefinition of existing keywords (database at www.eso.org/dicb)www.eso.org/dicb Keyword names should not be too long Avoid underscores and special characters Multi-HDU files Always the same structure of data for a given data type Data extensions come first, auxiliary and optional tables afterward

30. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 30 Acceptance Tests Usage of CPL recipe template Following CPL coding standard Usage of external libraries Namespace protection Execution Tests Completeness of the set of recipes and DRL functions Availability/representativity of test data Proper execution of recipes Generation of products Memory leaks Unit tests Documentation Detailed Validation Correctness of results Validation of input FITS compliance User-friendly documentation Data reduction cascade Unit tests Performance and Portability Execution speed Standard platforms: Scientific Linux

31. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 31 DFS Integration Observatory Pipeline Quality Control Pipeline Desktop science-grade data reduction Maintenance

32. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 32 Data reduction environments Observatory Pipeline On-the-fly data processing (event driven) Template-based processing Static calibration database (only certified products are used) Quality Control Pipeline Batch processing of complete data sets (all science and calibration data produced by one ESO instrument in one night) Best available calibrations are used => data must be organized according to the Calibration Cascade Desktop science-grade data reduction Modular and additional recipes are avilable Several front-ends for scripting (esorex), browsing (Gasgano), interactive (Reflex)

33. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 33 Observatory Pipeline Instrument Raw Data Archive PIPELINEOFFLINE FurtherAnalysis Raw data Processed data On-Line Archive Sytem Shipping Shipping

34. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 34 Observatory Pipeline: Workstation Pipeline workstation Data Organizer Reduction Block Scheduler Configuration files Calibration database Data Reduction System PRODUCTS Instrument Package Common Pipeline Library Archive Quality control Data arriving from the instrument Reduction Block  List of the raw frames  List of the calibration data  Name of the products  DRS recipe to apply

35. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 35 OCA Rules if DPR.CATG=="CALIB" and DPR.TYPE=="LAMP,WAVE" then { DO.CLASS = "ARC_SPECTRUM"; RAW.TYPE = "WAVE"; } 1.C LASSIFICATION 2.O RGANIZATION select execute(GI_WAVE_CALIBRATION) from inputFiles where RAW.TYPE=="WAVE“ group by TPL.START 3.A SSOCIATION action GI_WAVE_CALIBRATION { select file as MASTER_BIAS from calibFiles where PRO.CATG=="MASTER_BIAS" and inputFile.DET.WIN1.BINX==DET.WIN1.BINX; select file as GRATING_DATA from calibFiles where PRO.CATG=="GRATING_DATA" and inputFile.INS.GRAT.NAME==INS.GRAT.NAME; recipe giwavecalibration { } product mflat { PRO.CATG="MASTER_FLAT"; } }

36. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 36 Quality Control Log Files 12:21:51>-START GROUP / Start [AMBER] 12:21:51> ARCFILE = 'AMBER.2007-03-21T12:16:08.099.fits' [AMBER] 12:21:51> TELESCOP = 'NOT_SPECIFIED' / Telescope [AMBER] 12:21:51> INSTRUME = 'AMBER' / Instrument name [AMBER] 12:21:51> OBSERVER = 'UNKNOWN' / Observer name [AMBER] 12:21:51> PIPEFILE = 'p2vm.fits' / Filename of data product [AMBER] 12:21:51> INS GRAT1 NAME = 'GHR' / Grating common name. [AMBER] 12:21:51> INS GRAT1 RESOL = 527.778; / Encoder resolution [Enc/deg]. [AMBER] 12:21:51> INS GRAT1 WLEN = 2364.972; / Grating central wavelength [nm]. [AMBER] 12:21:51> INS GRAT1 ZORDER = 40319; / Grating zero order position [Enc]. [AMBER] 12:21:51> INS GRIS1 NAME = 'NAR_SLT' / OPTIi name. [AMBER] 12:21:51> INS GRIS2 NAME = '3T_K' / OPTIi name. [AMBER] 12:21:51> INS MODE = '3Tstd_High_K_1_2.365' / Instrument mode used. [AMBER] 12:21:51> PRO DID = 'ESO-VLT-DIC.PRO-1.15' / Data dictionary for PRO [AMBER] 12:21:51> PRO CATG = 'P2VM_REDUCED' / pipeline product category [AMBER] 12:21:51> PRO TYPE = 'REDUCED' / Product type [AMBER] 12:21:51> PRO REC1 ID = 'amber_p2vm' / Pipeline recipe (unique) identifier [AMBER] 12:21:51> PRO REC1 DRS ID = 'cpl-3.0' / Data Reduction System identifier [AMBER] 12:21:51> PRO REC1 PIPE ID = 'AMBER/2.3.2' / Pipeline (unique) identifier [AMBER] 12:21:51> PRO REC1 RAW1 NAME = 'AMBER.2007-03-21T12:13:38.479.fits' / File name of raw frame [AMBER] 12:21:51> PRO REC1 RAW1 CATG = 'AMBER_3WAVE' / Frame category of raw frame [AMBER] 12:21:51> PRO DATANCOM = 14; / Number of frames combined [AMBER] 12:21:51> PRO REC1 CAL1 NAME = 'FlatFieldMap.fits' / File name of calibration frame [AMBER] 12:21:51> PRO REC1 CAL1 CATG = 'AMBER_FLATFIELD' / Frame category of calibration frame [AMBER] 12:21:51> PRO REC1 PARAM1 NAME = 'dummy' / Name of recipe parameter [AMBER] 12:21:51> DET NTEL = 3; / Number of telescopes [AMBER] 12:21:51> QC P1 OFFSETY = 0.02; / Offset wavelength calibration [AMBER] 12:21:51> QC P2 OFFSETY = 0.05; / Offset wavelength calibration [AMBER] 12:21:51> QC P3 OFFSETY = 0.03; / Offset wavelength calibration [AMBER] 12:21:51>-STOP GROUP / Stop [AMBER]

37. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 37 Pipeline @ Garching: QC Main Tasks Create master calibrations Derive and trend QC parameters Create science products (Service Mode) Prepare data packages (Service Mode) Pipelines & QC parameters: requirements & testing Customers Paranal Science Operations ESO community (PIs, archive users) Home Page: www.eso.org/qc

38. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 38 Information by Instrument Similar pages for every VLT/VLTI instrument

39. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 39 Public Releases www.eso.org/pipelines Linked from all ESO instrument pages Release package, documentation, demonstration data 13

40. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 40 Desktop Data Reduction: Gasgano VLT interactive data organisation tool – FITS file browsing – Grouping – Classification Interactive front-end – Interface to CPL plugins – Interface to vizualisation tools Features – Java language – FITS format

41. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 41 Desktop Data Reduction: Reflex -CPL recipes -External tools -Python scripts -Visualisation -Beta users and internal evaluation

42. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 42 Pipeline Maintenance DFS Tickets Issued by PSO, DFO, INS, and the user community Pipeline priority meetings Bi-yearly meetings with representation of INS, DFO, PSO, SDD General issues and for each instrument closed, in process, open tickets Priority setting for further pipeline development Instrument Evolution Commissioning of new modes Detector upgrades

43. SDD/PSD P.Ballester INS Software Workshop - 10 Oct 2008 43