The Process of Data Ingestion in ÆKOS Andrew Graham and Matt Schneider TERN Ecoinformatics Data Analysts Logos used with consent. Content of this presentation except logos is released under TERN Attribution Licence Data Licence v1.0

Introduction The Data Analyst Role with TERN Ecoinformatics Analysis of source data and methods ÆKOS system development and domain modelling Contextual description of the data Publication of data into ÆKOS

The AEKOS Framework 1.Upper Context: Party, Project, Scope etc 2.Domain Model (Ontology): Observed entities, their features and relationships 3.Description Model: Methods and definitions 4.Indexing Model: Search and federation

Upper Context Provides context for Datasets: Contact details High level objectives of program Licensing details and conditions of use Statement of scope Alignment with national metadata standards (ANDS) Statement of curation processes applied to data

Understanding Field Sampling Schematic view of sampling configuration

Methodological work-flow Study Location Selection Study Location Visit Study Location Establishment Sampling Unit Selection Vegetation Assessment Physical Assessment Landscape Assessment Soil Assessment Fire Evidence Surface Cover Disturbance Evidence Vertebrate Evidence Climate Evidence Species Assessment Species Life Stage Vegetation Assemblage Voucher Collection Canopy Age-class Canopy Assessment Structural Formation Overstorey Measurement

Authored Method Descriptions Start with published method manuals Enrich existing method descriptions (protocols) with external web links and other resources Clarify questions about methods Divide the protocol into smaller method descriptions

Authored Method Descriptions Use a consistent format across datasets to allow comparison Direct linkage between the data value and the specific method of measurement Allows rapid assessment of suitability of data for re-use Eventually a method catalogue for researchers

Definition of source datasets Analysis and definition of source data types: Observation data Taxonomic concepts (a specific type of ref. data) Reference data (i.e. Lookup tables) Images and other artefacts.

Mapping to the ÆKOS Domain Model Study Location Sampling Unit Study Location Visit Spatial Poin t mudmap comment visit date observers disturbance datum x coord y coord identifier marker type Species Organism Group Voucher Specimen determined identity accession No. determiner field identity life form cover/abundance life stage phenology dominance Landscape slope aspect landform pattern selects represents contains represented by

Indexing Enrichment of data with common indexes: Project level traits Data management traits Ecological process traits (disturbance and land-use) Measurement details Species taxonomy Vegetation Assemblage (e.g. NVIS Major Veg. Groups) Jurisdictional and Bio-geographic boundaries Spatially derived features (e.g. distance from road, slope, aspect, etc.)

Federated Taxonomy

The AEKOS Ingestion DSL Screen cap of Eclipse... Source data query Vocabulary management Method description Mapping to the common model Populate indexes Upper context authoring Sandbox testing Source data query Vocabulary management Method description Mapping to the common model Populate indexes Upper context authoring Sandbox testing

Data Work-flow Point of truth is always the source database Data values are not changed Data issues fed back to Data Providers Automatic data refresh mechanism developed Corrections made in source database and fed back to AEKOS on next push Just new records and edits after the first load Update frequency defined for each dataset

Quality Assurance ÆKOS QA and review: Team review domain modelling of every dataset ingested Sandbox test ingestion before publishing to ÆKOS Review of method description by other team members Internal code validation and error checking

Quality Assurance Data Providers QA: Review method descriptions Review upper context Portal feedback: Review data content in the portal Use the portal and suggest enhancements and changes Look and feel Index traits Data accuracy and representation Feedback survey and facility on portal

Thank you Contact Details Data Analyst – Matt Schneider Data Analyst – Andrew Graham Website