ES 3 Takes the Provenance Challenge J AMES F REW Donald Bren School of Environmental Science and Management University of California, Santa Barbara

Provenance Challenge 2 Frew / Provenance Challenge Workshop / DC / SCA fraction, Sierra Nevada, 07 Mar 2004

Provenance Challenge 3 Frew / Provenance Challenge Workshop / DC / Model structure: MODIS SCA / albedo

Provenance Challenge 4 Frew / Provenance Challenge Workshop / DC / From Wrappers to Probulators Wrappers: Active Lineage  +  Complete control over what gets recorded  Single language/API for all wrapped events  Not tied to execution –You can even lie about what happened  -  Must explicitly script everything  Scripts can drift from reality –You can even lie about what happened

Provenance Challenge 5 Frew / Provenance Challenge Workshop / DC / From Wrappers to Probulators Probulators: Passive Lineage  +  Record what actually happened –Not just what you think happened –Not what didn’t happen  Automatic: don’t have to write new scripts for everything  -  Different flavors for different environments –Can’t just do everything in {…insert favorite language here…}

Provenance Challenge 6 Frew / Provenance Challenge Workshop / DC / Probulator flavors  Instrumentation  Insert lineage capture instructions directly into science codes –e.g. “I just created file ‘foo’”  Typical implementation: preprocessor/precompiler  Overriding  Replace standard routines/libraries with lineage-capturing versions –e.g. open(…) → snoopy_open(…)  Typical implementation: modify execution environment –environment variables –configuration files  Passive monitoring  Trace program execution –e.g. “called open() with args = foo, bar, …”  Typical implementation: strace’d shell

Provenance Challenge 7 Frew / Provenance Challenge Workshop / DC / ES 3 Lineage Architecture

Provenance Challenge 8 Frew / Provenance Challenge Workshop / DC / Probulating IDL: Instrumenting the code ;edit pro modscag_cleanse,prefix=prefix,ns=ns,nl=nl HELP, NAMES="*", OUTPUT=ES3_ENVIROMENT & ES3_LOG, $ ENTER="modscag_cleanse", ENVIROMENT=ES3_ENVIROMENT ;clean up {under,over}flow of MODSCAG run ; ;Input:prefix = prefix for all of the MODSCAG output filenames ;ns = number of samples ;nl = number of lines ;Output:rewrite of the MODSCAG files ; ;t.h.painter / ; open snow file ES3_openr,1,string(prefix,'snow.pic') snow=fltarr(ns,nl) readu,1,snow [ blah blah blah ] HELP, NAMES="*", OUTPUT=ES3_ENVIROMENT & ES3_LOG, LEAVE="modscag_cleanse", $ ENVIROMENT=ES3_ENVIROMENT END ; modscag_cleanse

Provenance Challenge 9 Frew / Provenance Challenge Workshop / DC / Probulating IDL: Results <init time=" T234606Z” pid="31002" stime=" T234604Z" pstime=" T234256Z" ppid="30920" language="idl" user="haavar" hostname="spitting-duck.bren.ucsb.edu"> <variable name="!PATH" value="/home/haavar/probulator//idl: /home/rsi/idl_6.1/lib/hook: […] /home/rsi […] /home/haavar/painter/data/tillsnow.pic ]

Provenance Challenge 10 Frew / Provenance Challenge Workshop / DC / Probulating a shell script: Example 