Presentation is loading. Please wait.

Presentation is loading. Please wait.

DMAC The Data Management and Communications Subsystem of the U.S. Integrated Ocean Observing System (IOOS) Steve Hankin NOAA/Pacific Marine Environmental.

Similar presentations

Presentation on theme: "DMAC The Data Management and Communications Subsystem of the U.S. Integrated Ocean Observing System (IOOS) Steve Hankin NOAA/Pacific Marine Environmental."— Presentation transcript:

1 DMAC The Data Management and Communications Subsystem of the U.S. Integrated Ocean Observing System (IOOS) Steve Hankin NOAA/Pacific Marine Environmental Laboratory DMAC Steering Team DMAC Modelers’ Caucus

2 Nov ‘06TECO-WIS 2 GOOS - Global Ocean Observations GTSPP, ARGO, Satellites, VOS, others Why not do something comparable for the coastal ocean?

3 Nov ‘06TECO-WIS 3 US IOOS Coastal System: “National Backbone” Regional Associations

4 Nov ‘06TECO-WIS 4 Satellites Aircraft Fixed Platforms Ships Drifters & Floats AUVs Observing Subsystem Integrated Ocean Observing System Metadata standards Data discovery Data transport Online browse Data archival DMAC Subsystem Modeling & Analysis Subsystem Safe & Efficient Maritime Operations Homeland Security Natural Hazards Climate Change Public Health Ecosystem Health Living Marine Resources Societal Goals

5 Nov ‘06TECO-WIS 5 Many Variables – the IOOS Core Set Physical –Salinity, temperature, air-sea heat flux –Sea and water level, water depth (Shore line) –Surface waves (height, direction, freq), currents & sea ice Chemical –Contaminants & dissolved nutrients –Dissolved oxygen Biological –Fish species & abundance –Chlorophyll, ocean color & phytoplankton abundance/species –Zooplankton abundance/species –Pathogens Interdisciplinary –Optical properties (Atmospheric & in-water) –Bottom character and benthic habitats

6 Nov ‘06TECO-WIS 6 The Governance Challenge Many independent observing system activities –federal, state, county, municipal, academic & private players Independent funding paths Widely varying program missions A planning office -- Ocean.US -- established to coordinate the efforts Effective governance depends upon a workable data management and communications (DMAC) plan

7 Nov ‘06TECO-WIS 7 …competing characteristics ruled out the most straightforward solutions for IOOS

8 Nov ‘06TECO-WIS 8 No simple data standard can be designed that all ocean data providers will utilize

9 Nov ‘06TECO-WIS 9 No management structure exists to enforce adoption of a complex, data management solution

10 Nov ‘06TECO-WIS 10 Data cannot be centrally managed at a single location

11 Nov ‘06TECO-WIS 11 The DMAC Plan – a community effort Brought together … 6 Federal agencies 6 Universities 4 Regional/State agencies 3 Private sector orgs. Steering Committee guided 2 Outreach Teams, 4 Expert Teams Data Facilities Outreach User Outreach Data Discovery & Metadata Data Transport Data Archive & Access Applications & Products Steering Comm. convened June 2002 Public drafts circulated Dec. 2003

12 The DMAC Plan – a community effort Four levels of review 1.Solicited expert review 2.Public workshop 3.Public email-based review 4.Federal Registry comments Drafts broadly circulated ExCom, NORLC, IWG, USGST, Ocean.US Workshops, Regional Summits, Professional Society newsletters, email lists and websites 150 reviewers 6 Federal Agencies 22 Universities & Institutes 13 Private Sector 13 Regional/State agencies 5 International An implementation plan (a roadmap to detailed specification) Published March 2005

13 Nov ‘06TECO-WIS 13 The DMAC Plan is a framework for interoperability among independent, heterogeneous systems, large and small Independence: Organizations will manage their own data Standards: enabling innovation! “Web services”: the common language to be spoken by pre- existing systems

14 Nov ‘06TECO-WIS 14 How DMAC selects standards “Adapt, adopt and only as a last resort develop” Classified by IOOS maturity levels 1.R&D 2.Pilot 3.Pre-operational 4.operational Strong preference for open standards

15 Nov ‘06TECO-WIS 15 DMAC planning process continues … Steering Team Expert Teams Transport and Access Metadata and Data Discovery Archive Caucuses Private sector International Modeling Regional Working Group Systems Engineering Interagency Oversight Working Group

16 Nov ‘06TECO-WIS 16 Data Transport Harmonize developments from 2 communities 1.GIS – utilizing OGC W*F standards 2.FES * – utilizing OPeNDAP “Adapt, adopt and only as a last resort develop” * FES = fluid earth systems Expert Team -- 10 members academia (4), federal (4), private sector (2)

17 Nov ‘06TECO-WIS 17 Digression on OPeNDAP … ”DAP” == Data Access Protocol Since 1992 - a “web service” before the term existed “Files” vanish  “datasets”, instead On the server –netCDF, GRIB, HDF, IEEE, … or relational databases –aggregations (multi-terabyte datasets) On the client –“open” and read subsets –any netCDF application is a web client (may need to relink) –Matlab, IDL, IDV, GrADS, Ferret, CDAT, … Widely used in research modeling communities

18 Nov ‘06TECO-WIS 18 OGCOPeNDAPGateway Grids 2 - WCS3 - CFGALEON Time series 1 - WFS/GML2 - DAPPER Profiles 1 - WFS/GML2 - DAPPER Tracks 2 - WFS/GML1 - DAPPER Points 3 - WFS/GML1 - (CDM)...... …… IOOS maturity levels 1.R&D 2.Pilot 3.Pre-operational 4.operational Data Transport

19 Nov ‘06TECO-WIS 19 Metadata and Data Discovery Team approach –Coordination with organizations/programs such as MMI *, ISO, FGDC *, GCMD *, ORION * and QARTOD * –Engage data providers in the process Determine marine metadata content requirements –data lifecycle: identify, assess, access, utilize and archive –adaptive to data type and method of data collection Compare metadata elements across multiple standards –FGDC, ISO, DIF, EML, ESML, SensorML, MarineXML, etc. MMI = Marine Metadata Interoperability project FGDC = US Federal Geographic Data Committee GCMD = NASA Global Change Master Directory ORION = US NSF-funded ocean research initiative QARTOD = Quality Assurance Real Time Ocean Data “Adapt, adopt and only as a last resort develop”

20 Nov ‘06TECO-WIS 20 Starting point: “Guide for IOOS Data Providers, v1.0” Initial guidance contains: –Required vocabularies IOOS keywords, core variables, national backbone programs –Recommended vocabularies ISO 19115 topic categories, GCMD keywords, IHO codes, OBIS species, CF standard names –Metadata standards ISO 19115, FGDC CSDGM, GCMD DIF, OBIS –Minimal list of elements Discovery, access, data transport, archive, consumer use, data management –Data discovery systems NSDI Clearinghouse, Geospatial OneStop, GCMD, OBIS Metadata and Data Discovery

21 DMAC Modelers’ Caucus – Challenges 1.Advancement and use of CF (netCDF & OPeNDAP) Further work: a.unstructured grids b.Geospatial datums c.Nested grids 2.Downscaling and splicing of neighboring domains 3.Rich model metadata 4.OSSEs – couple modeling and observing system design 5.Model-data validation and model-model comparison Modeling and Analysis Steering Team (MAST) >> Convened Oct. 25, 2006 << Community Modeling Networks (CMNs) to follow

22 Nov ‘06TECO-WIS 22 “ Community Standards for Unstructured Grids” UCAR/Unidata, Boulder CO -- >> October 16-18 << NOAA-IOOS funded Standards for unstructured coastal model outputs –A community process –Standardize file contents, formats –  Standardize network access protocols A demonstration testbed will be created Technical discussions will be open –on a moderated web site

23 Nov ‘06TECO-WIS 23 DMAC and WIS – Closely related challenges Both seek to Link organizationally-independent data suppliers Address a broad range of user needs Integrate heterogeneous data types Merge real-time (push) and delayed mode (pull) Build upon standards WORLD METEOROLOGICAL ORGANIZATION Weather Climate Water

24 Nov ‘06TECO-WIS 24 DMAC areas in need of particular attention (to what degree can WIS solutions be adopted?) Real time (push) ( IDD/LDM? ) Access control Service registries Data assembly (regional and national) WORLD METEOROLOGICAL ORGANIZATION Weather Climate Water DMAC and WIS – Closely coupled solutions

25 Nov ‘06TECO-WIS 25 kahm-sa-hahm-ni-da (Thank you)

Download ppt "DMAC The Data Management and Communications Subsystem of the U.S. Integrated Ocean Observing System (IOOS) Steve Hankin NOAA/Pacific Marine Environmental."

Similar presentations

Ads by Google