The Unified Access Framework (UAF) Philosophy, progress, and plans DAARWG Meeting, Seattle, Nov Steve Hankin (PMEL), Kevin O’Brien (PMEL/JISAO), and the NOAA UAF team Rich Signell )

Review … continuing into new material Nov NOAA/UAF 2

Nov GEO-IDE a strategy for enterprise- wide integration of NOAA data (i.e. adopting standards and practices to achieve data interoperability) NOAA/UAF pgs (by “DMIT”) Why is this a hard problem?

Nov NOAA-world NOAA/UAF weather forecast (time critical) fisheries management (regulatory concerns) nautical charting climate, ocean, atmosphere research response and restoration … the list goes on … … the list goes on …

Nov NOAA/UAF Different disciplines have different concepts of ‘data’. Each develops solutions that make sense to them.  Getting people (and organizations) to change habits is difficult! (and data management has often been an after thought.)

Nov The accepted approach: build a “system of systems” Wrap existing systems with loosely coupled, standardized services  a Service Oriented Architecture NOAA/UAF GEO-IDE Con-ops outlines such a plan. How to build it with a largely volunteer team?

Nov Generate use cases 2.Define requirements 3.Write a Concept of Operations 4. … and an Implementation Plan 5.Assemble (volunteer) teams to implement NOT ! NOT ! NOAA/UAF Tried and true approach …

Nov An alternative (‘agile’) approach Don't Solve Problems -- Copy Success -- NOAA/UAF

Nov Why ‘agile’ is attractive Because inevitably: funding is much smaller than needed; collaborations are more difficult than anticipated; infrastructure is being built on a background of rapidly evolving technology NOAA/UAF

Nov Why ‘agile’ is attractive NOAA/UAF infrastructure is being built on a background of rapidly evolving technology Change equals risk. Mitigate risk by following a strategy of incremental change that serves users (increasingly) well at every stage of evolution. “Visualize a decade away. Build things that work today.”

Agile Principles (condensed from the ‘Agile Manifesto’) Working software is the meaningful measure of progress: ensure early, continuous, frequent releases Emphasize simplicity - maximize the work not done Build projects around motivated individuals. Give them the support they need and entrust them The best architectures, requirements, and designs emerge from self-organizing teams Welcome changing requirements Scientists ( ‘business people’ ) and developers must work together on a daily basis Nov NOAA/UAF 11

Nov NOAA/UAF 12 Lemma “Don’t let perfect be the enemy of good”

What “success” did UAF chose to copy? Year 1 focused on gridded datasets. Nov Projects: (too many to name) Data formats: netCDFGRIBHDF Service stack: netCDF-CF-DAP-THREDDS (WMS) Applications: MatlabArcGISFerret GrADS Google Earth IDV LAS ERDDAP… Users: (too many to name) … NOAA/UAF

Nov Who is providing data this way? Modelers –AR4&5, GFDL, NCAR, … Satellite programs –GHRSST, PathFinder, CoastWatch, … NCEP weather and ocean forecasts –GRIB files served via NOMADS Coastal (“HF”) radar A growing list of observations programs –Argo, OceanSites, tide gauges, … Adoption by OGC is well underway NOAA/UAF

Nov How to reach users? (without downloading files) NOAA/UAF Through their preferred tools

Nov Model 1: UMASS-ECOM Model 2: UMAINE-POM Data: SST 2008-Sep-08 07:32 Desktop access in Matlab NOAA/UAF

Nov Access in ArcGIS using the Environmental Data Connector (EDC) NOAA/UAF

Nov Desktop access in Ferret NOAA/UAF

Nov Desktop access in GrADS NOAA/UAF

Nov NOAA/UAF 20 UAF home page Instructions for end users: how to access data through their favorite applications UAF is experimenting to see how documentation may be shared by other projects…

Under the hood Projects with data to provide –make their data available as netCDF-CF (or other TDS-compatible format) –host a THREDDS / OPeNDAP server Lets look at the tools we have to link projects together … Nov NOAA/UAF 21

UAF “network topology” a tree defined in THREDDS (XML) Nov NOAA/UAF 22 NOMADS GEO-IDE/UAF NOAANOAA Affiliated NMFSOARNWSNESDIS NODC NGDC GFDL PMEL AOML OCO PFEG NDBC ESRL Coastwatch IOOS National Partners IOOS Regional Partners NAVO AOOS NANOOS CENCOOS SCCOOS PACIOOS GLOS NERACOOS MACOORA SECOORA CARICOOS GCOOS

Nov Not so fast, kiddo! compliance with CF conventions is inconsistent files commonly are not aggregated into logical datasets metadata are often in need of enhancement NOAA/UAF Perspective – this is not an unusual situation: Standards compliance problems are *the norm* Divergent dialects often pile up (e.g. GRIB, BUFR) UAF tools offer a solution …

Nov ‘NcML’ can be used to repair problems (*) e.g. Improve CF compliance by adding “standard_name” attribute to GRIB data The file, itself, is untouched. The ‘virtual file’ seen through the services conforms to standards. (*) IOServiceprovider modules also important … not discussed here NOAA/UAF

Nov NcML for aggregation e.g. Aggregate three 1-year files of the same (say) model run </aggregation> A long time series ‘virtual file’ is seen through the services. NOAA/UAF

UAF Team members are helping data managers, person-to-person, to improve the data services from their projects (a gradual, but important process) (a gradual, but important process) in parallel … in parallel … Nov NOAA/UAF 26

Developing the UAF Catalog Cleaner (a ‘web crawler’) Nov NOAA/UAF 27 NOMADS UAF ‘RAW’ catalog NOAANOAA Affiliated NMFSOARNWSNESDIS NODC NGDC GFDL PMEL AOML OCO PFEG NDBC ESRL Coastwatch IOOS National Partners IOOS Regional Partners NAVO AOOS NANOOS CENCOOS SCCOOS PACIOOS GLOS NERACOOS MACOORA SECOORA CARICOOS GCOOS NOMADS UAF ‘CLEAN’ catalog NOAANOAA Affiliated NMFSOARNWSNESDIS NODC NGDC GFDL PMEL AOML OCO PFEG NDBC ESRL Coastwatch IOOS National Partners IOOS Regional Partners NAVO AOOS NANOOS CENCOOS SCCOOS PACIOOS GLOS NERACOOS MACOORA SECOORA CARICOOS GCOOS ‘RAW’ ‘CLEAN’

The Catalog Cleaner 1. Crawl the raw catalog 2. Extracts metadata from the files, themselves, and from THREDDS into a relational database 3. Process the data base to detect aggregations, etc. 4. Create new THREDDS XML that is aggregated and metadata-cleaned Nov NOAA/UAF 28

‘09 Carbon Tracker files from ESRL Nov NOAA/UAF 29 ‘raw catalog’ snippit Optional documentation (beyond what is inside the file) Services: Usually OPeNDAP. Often WMS. Sometimes more. Optional viewers

Nov NOAA/UAF 30 ‘clean catalog’ -- same data, but … augmented with Uniform services Uniform viewers improved metadata uniform services uniform viewers

Nov NOAA/UAF 31 Uniform services ISO-standard metadata metadata quality assessment Simple interface to get a subset

Nov NOAA/UAF 32

Nov NOAA/UAF 33

Nov NOAA/UAF 34 Godiva2 uses the WMS map service (as do other GIS apps)

Nov NOAA/UAF Google Earth Property- property plots Sections and Hofmullers Vector plots Segue to desktop tools Line plots Animations Differencing Analyses

Nov and ERDDAP provides … NOAA/UAF

Nov mat REST URL access to data subsets in several formats (accessible through home-grown scripting of many types).kml.mat.nc NOAA/UAF

Nov NOAA/UAF 38 ISO-standard metadata Data Discovery Crawl the clean catalog. Create ISO metadata

Agile principle: “Maximize the work not done” Who has already built reasonably mature discovery portals (preferably free) ? Unidata’s RAMADDAUnidata’s RAMADDA GI-CAT (able to crawl THREDDS catalog)GI-CAT (able to crawl THREDDS catalog) Geoportal (able to harvest ISO metadata)Geoportal (able to harvest ISO metadata) Nov NOAA/UAF 39

Data Discovery using ESRI Geoportal NODC is leading the UAF Geoportal investigations

Enter Search terms Data Discovery using ESRI Geoportal

Search Results Data Discovery using ESRI Geoportal

Refined Search New Results Data Discovery using ESRI Geoportal

Expand Selection Data Discovery using ESRI Geoportal

Available Services Data Discovery using ESRI Geoportal

Available Services Data Discovery using ESRI Geoportal

View metadata details about the dataset

Available Services Data Discovery using ESRI Geoportal

Direct link into the THREDDS Data Server

Available Services Data Discovery using ESRI Geoportal

This seque to LAS demonstrates how GeoPortal can be trained to recognize a uniform set of UAF viewers

Nov NOAA/UAF 52 Full disclosure: Geoportal is very powerful, but … difficult to set up immature user interface

Nov In situ observations NOAA/UAF are characterized by their ‘feature type’ (a time series, a track line, a vertical profile, …) individual feature instances are degenerate (1D) grids the UAF framework just described already handles individual 1D features many data providers are creating such files (Argo, OceanSites, …). Growth is robust. but users want to interact with collections of features standards and practices are in flux

Nov Collections of in situ observations NOAA/UAF Candidates: 1. CF ‘Discrete Geometries’ Chapter 9 of CF 1.6, May 2011 the ‘natural’ (?) extension for UAF but … a new API yet to be released (TDS 4.3) Java support only few tools available

Nov Collections of in situ observations NOAA/UAF Candidates: 2.‘DAPPER’ (OPeNDAP Sequences) solid technology good prototype tools and examples already years ago but … poor standardization of semantics (conventions for lat, long, time, …) flagging momentum …

Nov Collections of in situ observations NOAA/UAF Candidates: 3.OGC Web Feature Service (WFS) much momentum in the GIS community but … by design little standardization of semantics communities must develop ‘application schemas’ success rates poor, momentum flagging …

Nov Collections of in situ observations NOAA/UAF Candidates: 4.OGC Sensor Observation Service (SOS) significant momentum in the earth science community but … only the request protocol is standardized many choices for semantics: “SWE common”, KML, custom ASCII/JSON/etc. few tools, interoperability concerns

Nov Collections of in situ observations NOAA/UAF What to do? Agile philosophy: Maximize the work not done Don’t let perfect be the enemy of good UAF isn’t resourced to undertake R&D ($ + risk)

Nov Collections of in situ observations NOAA/UAF UAF approach: 1.Explore the potentials of ERDDAP. It provides IOOS-compatible SOS ‘TableDAP’ and other R.E.S.T. outputs commitment to support many protocols 2.Where data are in databases: install ERDDAP e.g. OSMC (in situ ocean obs) 3.Prototype TDS 4.3 / CF Discrete Geometries Utilize from ERDDAP and LAS (ArcGIS soon after?)

Nov Collections of in situ observations NOAA/UAF UAF approach: 4.Vigorously promote 1D CF features Here’s why: Available today, understood by data providers, momentum exists Compatible with UAF tools and practices Provides immediate value to users

Nov Collections of in situ observations NOAA/UAF UAF approach: 4.Vigorously promote 1D CF features  Efforts will pay off (low risk) Gets the data ‘out there’ (open, available) Metadata-rich Discoverable through standard frameworks Future TDS versions will offer aggregation  miracles will happen for free: SOS, improved discovery, etc.

To increase data integration, what question should we be asking? Is it: “How do I build a system that will […]?” Nov NOAA/UAF 62 Or is it: “How do we most rapidly narrow the gap between what we know technology is capable of delivering, and what our actual practices are today?” What is the right balance of strategies: ‘Design and build’ vs. ‘Agile’ ? ‘Design and build’ vs. ‘Agile’ ?

Nov Discussion NOAA/UAF