1. Lesson 1 Lesson 2: Association Lesson 1: dfos - vision and architecture Directory structure Installation Tips and tricks Lesson 3: Operations www.eso.org/~qc/dfos/ dfos: tutorial

2. Lesson 1 Complexity...... comes by itself: Simplification...... needs efforts The answer: dfos = Data Flow Operations System www.eso.org/~qc/dfos/ Vision

3. Lesson 1 To obtain … common understanding of tools and workflows contributions from everyone possible easier multi-tasking cross-instruments training possible Principle: all workflow steps in common dfos tools all instr-specific info in config files dfos regime user regime config files dfos tools Vision

4. Lesson 1 Architecture Tools for two workflows: –daily workflow –packing workflow Level 1 tools: called by user Level 2 tools: called by level 1 tools Interaction: –command line –graphical user interfaces: DFO monitor; SM monitor (self-managing HTML pages)

5. Lesson 1 Architecture Inter-tool “communication”: status flags –DFO_status; SM_status etc. (read by monitors) –possible to be read by wrapper tools $DFO_MON_DIR/DFO_STATUS:

6. Lesson 1 Architecture central element of dfos: the Association Block AB (--> Lesson 2)

7. Lesson 1 Architecture daily workflow sketch:

8. Lesson 1 Directory structure dfos needs well-defined directory structure environment variables many have DATE substructure, with DATE as 2005-01-14: –e.g. $DFO_RAW_DIR others are flat –e.g. $DFO_MON_DIR one is more complex: –$DFO_SRV_DIR/ / /raw etc. –$DFO_SRV_DIR/ / /sci_raw etc.

9. Lesson 1 Installation Get a complete installation of dfos (daily workflow) in four steps: 1. Getting started 2. Initial tool set 3. Creating ABs 4. Finish installation this is where you can get without ABs

10. Lesson 1 Step 1: Getting started Obtain a dfos installation: prepare the initialization file.dfosrc prepare the dfos directory structure download dfos tools prepare the configuration files 1. Prepare your account and install the initialization file.dfosrc download.dfosrc into $HOME edit as appropriate create directory structure as defined make sure.dfosrc is sourced properly 4x : ok, continue

11. Lesson 1 Step 1: Getting started 2. Install dfosExplorer (by hand): download and untar execute 2x : ok, continue 3. tune your browser to become more interactive (tune your browser) 4. Read the documentation (‘help’ and ‘dfos’)

12. Lesson 1 dfosExplorer use action button to download edit config file

13. Lesson 1 Step 2: Initial tool set 5. Install dfosInstall (using dfosExplorer) 6. Install: –dfosLog –dvdMonitor and dfosCron dfosInstall: new installation/upgrade –new installation: installs everything, incl. template config file (old one saved!) –upgrade: config file is installed in $DFO_CONFIG_DIR/template (no overwrite!) config files: always come as template; some need no editing at all, others need heavy editing. Always read documentation, all configurable parameters are documented! Remember: config files are your contribution and your responsibility.

14. Lesson 1 Step 2: Initial tool set 7. Install: dataclient, fitsreport, createReport, checkDownloads –these tools define the pre-processing part of dfos –they don’t need ABs –dataclient: the tool for header/fits file downloads implicit downloads (specify type, date, mode): dataclient -t h -d 2005-01-31: database headers dataclient -t r -m all -d 2005-01-31: all raw fits files explicit downloads (specify ARCFILE names): dataclient -t r -m l -l my_list.txt -o /data03/data/my_outpath/ –fitsreport: the workhorse for fits key reading used by various different tools flexible (formatted/unformatted), performant –createReport: general reporting tool (dfo reports, user reports)

15. Lesson 1 Step 3: Creating ABs We are here: Next: AB creation, processing workflow Today: no details about association, just assume you can do it... ABs: text file with all information about a set of raw frames necessary to: process and/or pack --> Even without a pipeline, you want to work with ABs.

16. Lesson 1 Step 3: Creating ABs 8. Install: createAB (comes with: helpAB, listMcalib) –it uses fitsreport to create lists: raw_list, vcalib_list, mcalib_list (all under $DFO_MON_DIR) 9., 10. create the config files (Lesson 2) –prepare: know your raw data types (DPR keys) and product types (PRO.CATG) know the match keys (‘setups’) know your pipeline (recipe names) know your DRS sketch your calibration map (piece of paper is fine)

17. Lesson 1 Step 4: Finish installation 11. Install: createJob, processAB; getStatusAB –supported DRS: esorex, RBS, REI 12. Install: measureQuality, certifyProducts –prepare your QC procedures (MIDAS prog is attached as example) –QC procedures are as instr-specific as pipelines, no attempt made to include them into dfos –measureQ is a wrapper to call these procedures –certifyP is also a wrapper around a user-provided procedure to call the QC1 plots (example provided) –if you have no procedures yet: skip that part

18. Lesson 1 Step 4: Finish installation 13. Install: moveProducts, renameProducts, updateAB –moveProducts calls renameProducts, updateAB, listMcalib –listMcalib updates the mcalib list, using fitsreport –cdbIngest is not yet called by moveProducts, TBD 14. Install: updateDP, finishNight –updateDP reads ABs for packing –supports: DATE and OB scheme –has a graphical monitor –also needed if pipeline support is lacking –finishNight: cleans up (almost) everything

19. Lesson 1 Step 4: Finish installation updateDP:

20. Lesson 1 Tips and tricks Tune your browser –follow the description to make your browser an interactive GUI: launch shell scripts by clicking on a button open an editor to edit your config files with the browser used by the dfos monitor tools: dvdMonitor, dfosExplorer, dfoMonitor, updateDP, smMonitor, relMonitor –after tuning, mozilla will execute any script with extension.ash or.esh just by clicking

21. Lesson 1 Tips and tricks Cronjobs –delegate some work to cronjobs: dvdMonitor, header download, createReport, dfosExplorer –use dfosCron to call these tools –test the cronjob and the way it sources.dfosrc! FAQ –read that web page before asking FAQs...