1. Lesson 2: association Lesson 2: Association Lesson 1: dfos - vision and architecture Directory structure Installation Tips and tricks Lesson 3: Operations www.eso.org/~qc/dfos/ dfos: tutorial

2. Lesson 2: association Note added in proof: active buttons dfosExplorer: –use active buttons (‘refresh’, red button, config file editor) dfoMonitor: –compare web version (passive) and local (active) version –http://www.eso.org/~qc/monitor/DFO/

3. Lesson 2: association Association: principles daily workflow: –processing –packing (increm.) Both need: association (organization, classification, association = OCA) Where: $DFO_RAW_DIR/$DATE What? CALIB, FLAT, 1x1 binning, 4port readout, V_BESS filter With whom? Other raw FLATs, a master_bias

4. Lesson 2: association Association: principles Principles: follow rules and properties (rather than: file names, nightlog entries, …) –createAB: reads rules from config files, knows about properties from property files: raw_list, mcalib_list, vcalib_list) work on a data set –a DFO night (normally) use time-match rule to associate calib products –CLOSEST: smallest |  t| (default) –NEXT: smallest positive  t (CALIBs later than SCIENCE) –ATTACHED: same OBS_ID + smallest |  t| SCIENCE CALIB C A N

5. Lesson 2: association Association: principles Use match rules: –common property object1 has same property as object2: SCIENCE to find a MASTER_FLAT with the same filter –fixed-property object1 has some property, match it with object2 having always a specified property: SCIENCE to find a MASTER_DARK with DIT = 2.0 sec

6. Lesson 2: association Association: definitions Basic data sets –data set which does not need another data set (cannot split further) –synonyms: setting, setup, configuration –example: all VIMOS files with filter = U_BESS, 2x2 bin, fast readout, same quadrant U_BESS, 2x2, fast, Q1U_BESS, 2x2, fast, Q2V_BESS, 2x2, fast, Q1U_BESS, 1x1, fast, Q1

7. Lesson 2: association Association: definitions match keys –set of parameters to define a basic data set (INS, DET; OBS) raw-raw raw-product product-product packing –example: ins.grat.wlen + ins.grat.order + ins.grat.name + ins.slit.name –concept known already from the mcalib names!

8. Lesson 2: association Calibration map … is: –a 2D map of mutual interdependencies of raw and product files (CALIB and SCIENCE) –a user-friendly but not yet machine-readable representation of association rules –translation into config files needed sketch --> calibration map --> config files

9. Lesson 2: association Association Block –Text file with QUALIFIERCONTENT –Generalized Reduction Block (RB) or Set of Frames (SOF) –holds all association info (for processing and packing) –for tradition, it contains the RB and SOF explicitely –name: [_tpl].ab –stored: in $DFO_AB_DIR, later moved to $DFO_LOG_DIR/

10. Lesson 2: association Cascaded and virtual calibrations –Columns in calib map: connected by nodes –horizontal connections: required calib products –if not existing at runtime of createAB: virtual, otherwise real –virtual mcalibs = predicted names of mcalibs, required to exist at AB execution time (but not earlier) FLAT gimasterflat BIAS gimasterbias –key concept to facilitate batch processing of complete nights and batches of nights in case of complex calibration maps –of course: if an AB crashes, all cascaded ABs will also crash! –Hence this works fine in a stable environment master Bias

11. Lesson 2: association dfos tools for association AB creation/editing: –createAB: AB creation –processAB: execute AB processing –updateAB: update AB content after product certification and distribution AB reading: –createJob: create AB execution file –updateDP: read ABs and create package increments –getStatusAB: provide AB status infrastructure: –listMcalib (mcalib list); extractAB (gets all associated files); helpAB

12. Lesson 2: association Association config files –config files: structured into three parts config.assoc (‘the wholy gral’) specific rule files fitsreport_assoclists.cfg: fitsreport configuration rawtype_1 (BIAS)... rawtype_2 (FLAT)... rawtype_3 (STD)... BIAS.rules config.assoc FLAT.rules STD.rules fitsreport_assoclists.cfg

13. Lesson 2: association config.assoc –config.assoc has: classification definitions of grouping criteria: DPR, TPL keys DO_CLASS: expected by recipes, not under your control! RAW_TYPE: short form of DO_CLASS, under your control (must be uniquely ‘greppable’: e.g. IFLAT and DFLAT is fine, but not IFLAT and FLAT!) PACK_DIR: string defining the name of the final package directory (OB scheme) –pipeline info: recipe names cascade sequence event –packing info: additional file types to pack (e.g. gif, ps, paf) –plus tool configuration (N_MCAL_LIST etc.)

14. Lesson 2: association Configuration: rules files –Rules files: one per RAW_TYPE (e.g. BIAS.rules, FLAT.rules etc.) has all information for a column of the association map –associated files: [raw for processing; need not be defined] raw_assoc (for packing) mcalib (processing) mcal_assoc (packing) –match keys for these file types RAW_MATCH, RASSOC_MATCH, CALIB_MATCH, MASSOC_MATCH –optional recipe parameters –cascade info: pro_catg to be used for other data sets –trigger, waitfor events: used for scheduling DRS (REI)

15. Lesson 2: association Configuration: fitsreport…cfg –fitsreport_assoclists.cfg –fitsreport is used by createAB and listMcalib to create parameter lists raw files: raw_list (filled by createAB, from scratch) virtual mcalibs: vcalib_list (new entry for every predicted product under CASC in rule files; incremental fill/deplete) real mcalibs: mcalib_list (created by listMcalib, from scratch) –used in technical (unformatted) mode, includes file names and path –has structure from fitsreport, plus some additional info

16. Lesson 2: association Demo –Demo: run listMcalib and create mcalib_list delete vcalib_list and observe it growing run createAB and view raw_list –use dummy config.assoc with BIAS only; use 2005-01-10; check ABs –add FFLATs –add WAVE –check fresh ABs: SOF part, RB part –check processed, updated ABs: status