1. ESO Reflex A Graphical Workflow Engine for Data Reduction Richard Hook Euro VO Data Centres Alliance Theory & Grid Workshop, Garching, April 2008

2. 11th April 2008Euro VO DCA Workshop2 Overview The Sampo Project The ESO data reduction context ESO Reflex Examples Future plans Apology: this talk is not really about grid activities… (and definitely not about theory)

3. 11th April 2008Euro VO DCA Workshop3 Sampo overview: As part of Finland’s joining fee for ESO a contribution “in kind” of computer scientist staff was made available and named “Sampo”. Sampo started in January 2005 and ended in January 2008 The aim of the project was to assess the requirements for ESO data reduction and analysis software infrastructure in the medium term and perform a series of pilot projects to assess different options and produce useful tools The project was managed at ESO Garching with the team based in Helsinki, Finland

4. 11th April 2008Euro VO DCA Workshop4 The Sampo priorities Enabling and facilitating science-grade reduction of ESO data from the La Silla Paranal Observatory was identified as its primary goal of the Sampo project Long-standing request from the community Input from Instrument Scientists in Garching and Chile Sampo SAC Sampo has concentrated on developing ESO Reflex, a graphical user interface to run ESO data reduction recipes Other sub-projects conducted by Sampo PyMidas: a Python interface to Midas VODA: a pilot project addressing the integration of data analysis environments and the Virtual Observatory (cancelled)

5. 11th April 2008Euro VO DCA Workshop5 ESO cannot reduce all of the data its telescopes and instruments produce to a level where their full scientific potential will be exploited The responsibility for the quality of the scientific reduction of the data can only rest with the individual users The users are, then, faced with the challenge of a timely and accurate data reduction Proprietary and/or archival data from several instruments often need to be combined As volume and complexity of data increase, a detailed knowledge of the instrument, data format and header content is essential to fully exploit the data Science-quality pipelines are increasingly needed also for Quality Control General-purpose tools like IRAF and ESO-MIDAS are inadequate for the task and instrument specific software, implementing carefully tuned algorithms is essential The data reduction challenge

6. 11th April 2008Euro VO DCA Workshop6 Data reduction by individual users: the context ESO provides pipeline “recipes” for all VLT/VLTI instruments They remove the instrumental signature and are used for quality control at ESO and distributed to the community In some cases the data products are adequate for scientific analysis, but this is generally not yet the case Offline tools (Gasgano and EsoRex) are available to call the pipelines, but lack some of the functionalities needed by the community AND Many older general purpose reduction and analysis systems remain in wide use (MIDAS/IRAF etc.) as they contain valuable algorithms Many instrument-specific packages have been developed in the community (e.g., Euro3D tools, VIPGI etc) Greater use will be made of remote data resources and the Virtual Observatory (VO)

7. 11th April 2008Euro VO DCA Workshop7 Introducing ESO Reflex: a graphical data reduction environment A data reduction system for the end user requires: Modular recipes to provide access to intermediate products Interactive tools, defined or customized by the user, to analyze intermediate and final data products A user-friendly, intuitive and flexible interface The ESO Reflex tool, addresses the interface issue, with a focus on the use case of ESO data: Dedicated invoker for CPL-based recipes General invoker for Python scripts (hence PyRAF & PyMidas) General invoker for IDL scripts Invoker for the command line Many other Taverna features for free

8. 11th April 2008Euro VO DCA Workshop8 ESO Reflex: look and feel ESO Reflex is based on Taverna, a popular open source Java workflow engine

9. 11th April 2008Euro VO DCA Workshop9 Main features of ESO Reflex (over and above what Taverna offers) In interactive mode the user can make changes to input data and parameters during execution Errors on during recipe execution are detected by ESO Reflex and appropriate action can be taken (Some) flow control: looping, skipping, conditional statements, etc. Parallel execution: full advantage on multi-processor or multi- core machines Customisability: workflows are easily modified, Python and IDL interfaces, system commands can be invoked, easy access to VO and other web services FITS file handling (using Gasgano code): data organization, tagging, selection

10. 11th April 2008Euro VO DCA Workshop10 ESO Reflex in action: FORS2 MXU and a Python-based GUI tool

11. 11th April 2008Euro VO DCA Workshop11 ESO Reflex in action: a FORS Calibration workflow.

12. 11th April 2008Euro VO DCA Workshop12 Reflex/Taverna works very well with distributed resources and web services Reflex also supports the PLASTIC protocol for passing information between client-side VO tools As a small test project a workflow has been developed that finds VO image data, passes it to a remote processing server and uses PLASTIC tools running locally Reduction of local ESO data and access to remote VO facilities within same environment ESO Reflex and VO services

13. 11th April 2008Euro VO DCA Workshop13

14. 11th April 2008Euro VO DCA Workshop14 Potential shortcomings of ESO Reflex Yet another tool to learn, support etc. Not as powerful as a script Flow control limited (loops not easy) Not suitable for very complex workflows …

15. 11th April 2008Euro VO DCA Workshop15 The bottom line on ESO Reflex The Sampo project has enabled ESO to explore options for future data reduction systems The ESO Reflex model is cost effective because it capitalises on ESO’s investment in CPL-based algorithms Offering ESO pipelines to the community in a more flexible way Stimulating more scientific feedback from the community and, therefore, steering algorithm development towards more science- grade applications It exploits existing systems to provide those facilities that CPL is not designed to offer (e.g., graphics, interactivity, etc.) It avoids the need to develop from scratch a multi-purpose, and expensive, full-fledged data reduction environment

16. 11th April 2008Euro VO DCA Workshop16 The future of ESO Reflex and pipelines The Sampo project itself ended in January 2008 ESO has examined the outcome of Sampo and concluded that ESO Reflex is a viable way of addressing some of the mid-term needs of the user community The continued development of science-grade CPL-based data reduction recipes is vital The development of interactive tools is also needed

17. 11th April 2008Euro VO DCA Workshop17 Status and plans ESO Reflex V1.1 is currently available as a beta-test version on request Current version is built on Taverna 1.5 - conversion to Taverna 2.0 will be needed, and involves significant changes: API changes Use of Maven Some UI refactoring A public release is planned for late-2008/early-2009