Data Flows in ACTRIS: Considerations for Planning the Future Markus Fiebig, and the EBAS team NILU - Norwegian Institute for Air Research

Requirements for a Research Infrastructure Workflow I Operational infrastructure sets different demands than scientific work: Maintenance: Easy, Fast, Cost Efficient Minimise number of points of failure Ensure fast access to components to be maintained. Homogeneous data processing Ensure / prove that all data are comparable, i.e. processed in exactly the same way. Document what has been done “Where does this point in the IPCC come from?” Every data (pre-)product and processing tool must be identified, versioned, and archived. Consequence: You don’t set up the same service several times unless you have a very good reason. Conclusion may vary between NRT and manually QAed best quality data.

Requirements for a Research Infrastructure Workflow II Data flows need to be well-defined, i.e. doesn’t contain ambiguities. All steps in a data processing / curation work flow need to be defined, and implementation quality assured. All steps in data processing need to be documented. The provenance (which processing steps has the data seen?) has to follow with the data. The roles in executing the workflow need to be defined (Who is doing what and when?).

Current ACTRIS Surface In-Situ NRT Workflow Sub-network data centre: auto-creates hourly data files (level 0). initiates auto-upload to NRT server. FTP transfer to data centre Station: collects raw data in custom format transfer Data Centre: check for correct data format (level 0). check whether data stays within specified boun- daries (sanity check). FTP transfer to data centre Not well-defined Products and actions aren’t properly separated. Doesn’t use correct symbols. Station: auto-creates hourly data files (level 0). initiates auto-upload to NRT server. automatic feedback automatic feedback User access (restricted) via web-interface: ebas.nilu.no Processing to level 1.5 Processing to level 1 EBAS database User access via machine-to-machine web-service Hourly level 1 data file Hourly level 1.5 data file

Future ACTRIS Surface In-Situ NRT Workflow Level 0 data Raw as provided by instrument Instrument specific Metadata attached Manual Processing: Manual assignment of flags Manual calibration correction. Automatic Processing: automatic assignment of flags No calibration correction Level 0a data Raw as provided by instrument Instrument specific Metadata attached Manual corr. / flagging applied Level 0b data Raw as provided by instrument Instrument specific Metadata attached Automatic flagging applied Reg NRT

Future ACTRIS Surface In-Situ NRT Workflow Cont’d Reg NRT Identical algorithm Raw Processing: Calculation of targeted property. Remove instrument parameters and instrument failure periods Raw Processing: Calculation of targeted property. Remove instrument parameters and instrument failure periods Level 1a data Final targeted property Only valid data Original time res. Manual corr. / flagging applied Level 1b data Final targeted property Only valid data Original time res. Automatic flagging applied Reg NRT

Future ACTRIS Surface In-Situ NRT Workflow Cont’d Reg NRT Identical algorithm Time averaging: Calculate hourly means. Disregard invalid data Add coverage flags. Copy env. cond. flags occuring in ave. period. Time averaging: Calculate hourly means. Disregard invalid data Add coverage flags. Copy env. cond. flags occuring in ave. period. Level 2 data Final targeted property Hourly averaged Coverage & env. cond. flags Manual corr. / flagging applied Level 1.5 data Final targeted property Hourly averaged Coverage & env. cond. flags Automatic flagging applied

How to Use Workflow to Meet Requirements Use Persistent Identifiers (PIDs) to tag data pre-products and processing software (each version). Use DOIs to identify final data products. Have versioned archives for both, data (pre-)products and software for data processing. If same processing is done in different locations, guarantee that result is identical to ensure homogeneity. Include provenance information in metadata, i.e. metadata states data pre- products and software version used for processing. This applies to ALL data and software used ANYWHERE in the infrastructure. When setting up / updating workflows and their diagrams: Distinguish between data(sub)products, processing steps, and decisions. Use correct diagram symbols. Remove ambiguities from workflows. Discussion: Distribute roles in infrastructure as to achieve these aims most efficiently.