1. NOAA Data Management Activities Deirdre Jones, EDMC Chair Jeff de La Beaujardière, DM Architect Prepared for DAARWG 2011-11-15 1

2. Outline Motivation Recent EDMC Accomplishments EDMC FY2012 Plans DM Framework in NEO Strategy Data catalog approaches 2

3. Motivation NOAA Strategic Plan calls for: – Improved data interoperability and usability through application and use of common data management standards – Enhanced access and use of environmental data through data storage and access solutions, integration of systems, and long-term stewardship – Increased volume and diversity of data and information effectively integrated into models 3

4. New EDMC Procedural Directives Data Management Planning Directs managers of all projects and systems that produce data to write DM Plans Data Documentation – Directs NOAA programs to provide data documentation (metadata) Data Sharing by NOAA Grantees – Directs NOAA grantees to make their data publicly available All 3 are agenda topics for tomorrow 4

5. EDMC Plans for FY2012 (1/2) Implement approved procedural directives – EDMC developing detailed work plan – Further discussion tomorrow Begin to develop additional Procedural Directives – Data Access and Discovery Goal: Enable users to find and retrieve NOAA data Goal: Automate publication of NOAA data to data.gov and GEOSS – Data Citation Goal: Enable datasets to be referenced by unique identifier to provide credit, enable usage metrics, and distinguish duplicates 5

6. EDMC Plans for FY2012 (2/2) Hold 3rd annual NOAA‐wide EDM Conference – To engage stakeholders Host OGC Workshop – Coordination on data access standards Support DAARWG Meetings (twice annually) – To receive guidance from advisory board Support development of Archive Concept of Operations – Called for in CLASS External Review – Briefing after lunch today 6

7. Data Management Framework from National Earth Observations (NEO) Strategy, ch. 4 (inter-agency draft) Jeff de La Beaujardière, PhD NOAA DM Architect 7

8. Data Management Framework Principles Governance Architecture Standards Assessment Data Lifecycle Principles Full and Open Access Preservation Information Quality Ease of Use 8 Data Lifecycle from National Earth Observations (NEO) Strategy - Data Management Chapter (in preparation 2011)

9. Data Lifecycle Planning and Production Activities Data Management Activities Usage Activities 9 from National Earth Observations (NEO) Strategy - Data Management Chapter (in preparation 2011)

10. Data Lifecycle Usage Activities Data Management Activities Planning and Production Activities Collection Processing Quality Control Documentation Cataloging Dissemination Preservation Stewardship Usage Tracking Final Disposition Discovery Reception Analysis Product Generation User Feedback Citation Tagging Gap Assessment Requirements Definition Planning Development Deployment Operations 10 from NEO Strategy - DM Chapter (in prep. 2011)

11. Data Lifecycle Usage Activities Data Management Activities Planning and Production Activities Collection Processing Quality Control Documentation Cataloging Dissemination Preservation Stewardship Usage Tracking Final Disposition Discovery Reception Analysis Product Generation User Feedback Citation Tagging Gap Assessment Requirements Definition Planning Development Deployment Operations 11 Data Documentation DM Planning Data Sharing What-to-Archive Applicability of EDMC Directives Cataloging Data Citation Data Services

12. Data Lifecycle Usage Activities Data Management Activities Planning and Production Activities Collection Processing Quality Control Documentation Cataloging Dissemination Preservation Stewardship Usage Tracking Final Disposition Discovery Reception Analysis Product Generation User Feedback Citation Tagging Gap Assessment Requirements Definition Planning Development Deployment Operations 12 Some of the possible feedback loops in the Data Lifecycle

13. (proposed) NOAA Data Catalog Approach Jeff de La Beaujardière, PhD NOAA DM Architect 13

14. Catalog Goals Users can find NOAA data for desired phenomenon, location and time – Without knowing Office/Program structure – Single starting to point to find the data that is accessible via web services and well documented Data providers can register their services once, in a community catalog – And have their data be visible in a master catalog NOAA leadership can see improvements in NOAA data discovery & access 14

15. Some Existing Community-Specific Catalogs 15 IOOS Catalog Data UAF Catalog Services NGDC Geoportal NODC Geoportal CWIC CLASS Catalog GeoPlatform (ArcGIS.com Portal) NCDC Geoportal

16. Conceptual NOAA Distributed Catalog Architecture Data NOAA Master Catalog NOAA Web Site UI Community Catalogs data.gov API GEOSS API federated search (or scheduled harvest) NCDCNODCIOOSUAF Users & Clients 16 Analysis Tools API Services NGDC others...

17. (possibly colocated) Archive ConOps Data Management Overview Graphic: Connections and Information Flow 17 DM Plan* Data Documentation* (Metadata) Archive Decision* Data Access Service OAIS Reference Model Data Inventory Metrics Dashboard Catalog Service ID Tools Result paper decision policy response ID create write assess preserve guide add publish* understand get find register compile measure analyze use Data Producer Data User cite *topic of current EDMC Directive publish Archive [OV-2] (Note: Not all activities illustrated) Requirements Gap Assessment assess guide NOAA Leadership assess

18. BACKUP SLIDES 18

19. DM Principles from NEO Strategy Principles Full and Open Access: Earth observations should be made fully and openly available to all users promptly, in a non-discriminatory manner, and free of charge. Preservation: Earth observations should be managed as an asset and preserved for future use. Information Quality: Earth observations should be of known quality and fully documented. Ease of Use: Earth observations should be easily discoverable and accessible online using interoperable services and standardized formats that encourage the broadest possible use. 19 from National Earth Observations (NEO) Strategy - Data Management Chapter (in preparation 2011)

20. Procedural Directive Data Management Planning (DMP) Summary – Directs managers of all projects and systems that produce data to write DM Plans Provides guidance on content of DM Plans, including: – General description of the data – Data documentation and standards – Data access methods – Initial data storage and long-term preservation – Provides a DMP template and FAQs Feedback – Hundreds of comments through briefings, workshops, and meetings shaped principles, concepts and final text. – 117 comments received during official 30-day comment period EDMC approval was unanimous 20

21. Procedural Directive Data Documentation Summary: – Directs NOAA programs to provide data documentation (metadata) – Requires use of ISO 19115/19139 Provides guidance on metadata content, including: – Metadata for Discovery – Metadata for Use – Metadata and Documentation for Understanding – Documentation of Collections – Documentation of Datasets – Documentation of Services Highlights metadata resources, tools and challenges EDMC approval was unanimous 21

22. Procedural Directive: Data Sharing by NOAA Grantees Summary – Directs NOAA grantees to make their environmental data publicly available – Requires data sharing plan to be provided with new proposals and published at award – Data must be shared in a "timely" fashion but no later than two years after collection – Exceptions or extensions granted for legal reasons or on a case-by-case basis upon request Provides guidance on data sharing plans – Includes metadata – FAQs and template Feedback – EDMC approval – Feedback from Cooperative Institutes and Sea Grant Program 22

23. Good Data Management supports NOAA Leadership Priorities 23 NOAA Data Good Documentation Data Inventory Metrics Dashboard Data Catalog ______ Standardized Services + enable Ability to find, access, understand NOAA data Visibility in data.gov and GEOSS enables selected NOAA Leadership Priorities for NOAA data +

24. NOAA Master Catalog metadata record Tag Database metadata record metadata record metadata record metadata record metadata record metadata record metadata record DWH data.gov GEOSS CORE GEOSS StP Purpose E Purpose F DWH Response data.gov GEOSS Data CORE External Catalogs or Portals other portal Tags are not inserted into metadata records by data providers. Instead, the Catalog adds tags to indicate datasets relevant to a particular purpose. Datasets with a relevant tag are recorded by external catalogs. Tagging Concept 24

25. Potential Relationship of GeoPlatform to NOAA Master Catalog B) GeoPlatform is Master Catalog Community Catalogs Cat. 1Cat. 2 GeoPlatform Map & Data Svcs Cat. N D) Master Catalog feeds GeoPlatform Community Catalogs Cat. 1 Cat. 2 Master Catalog Map & Data Svcs Cat. N GeoPlatform Map Svcs Only C) GeoPlatform feeds Master Catalog Cat. 1 Cat. 2 Master Catalog Map & Data Svcs GeoPlatform Map Svcs Only Community Catalogs 25 A) No relation Master Catalog Cat. 1 Cat. 2 GeoPlatform Map Svcs Only WMS 1 WMS 2 Community Catalogs Map Services

26. GeoPlatform and Master Catalog working together NOAA Master Catalog (Geoportal or t.b.d.) Web-based Map Viewer UI data service Catalog 1 Cat. 2 Catalog 3 26 GeoPlatform (ArcGIS.com Portal) data.gov CS/W GEOSS Other API other catalog GCMD WAF List of WMS List of manual registrations ArcGIS server Shapefile KML Manual registration Some datasets might be registered directly in GeoPlatform

27. gridded data gridded data gridded data UAF Distributed Catalog Architecture Project Data & Services Unified Access Framework (UAF) Catalog Project Catalogs DAP THREDDS Catalog DAP THREDDS Catalog THREDDS Catalog DAP Analysis Tools 27 Matlab API IDVArcGISERDDAP Community Catalog

28. Use Google instead of a Dedicated Catalog? Project Data & Services Google & other search engine crawlers NOAA Web Site ? Community Catalogs data.govGEOSS agreed convention to identify geodata servers (e.g., /geodata.xml ) data service Users & Clients 28 ??

29. Probably want both formal catalog & search engine support Project Data & Services NOAA Master Catalog (machine API, spatial & temporal queries, controlled vocabularies) NOAA Web Site UI Community Catalogs external catalogs API general users simple search data service Geoportal Server GeoNetwor k WAF THREDDS Catalog Users & Clients 29 Google (free-text search)