1. 1 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Addressing NOAA’s Data Management Challenges Presentation to the World Data Center for Glaciology 30th Anniversary Workshop Dr. Christopher G. Fox Director NOAA National Geophysical Data Center

2. 2 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 NOAA Encompasses a Challenging Diversity NOAA currently manages >90 environmental observing systems, some with hundreds of stations: including land-, sea-, air, and space-based observing platforms These systems gather >300 diverse environmental parameters (e.g. marine biological health, economic fisheries data, physical and chemical state of the atmosphere and ocean, paleoclimate proxy data, geodetic survey points, etc.) NOAA also requires other national, international and commercial data in its operations (some in real-time) NOAA data management infrastructure includes numerous significant stovepipe systems Future observing systems will produce vastly increased data volumes that will need to be archived and efficiently accessed by an expanding number of users NOAA is migrating from this current stovepipe environment to an information enterprise

3. 3 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 NOAA’ Top Ten Challenges 1 10) Alphabet Soup 9) Stove Pipes 8) Integration 7) Architecture 6) Data Sharing 5) User Needs 4) Maximizing Benefits 3) Communication 2) Data Management 1) Execution 1.Vice Admiral Lautenbacher in address to American Meteorological Society Annual Meeting, January 30, 2006

4. 4 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Data Management - a top priority Improving data management is among the highest priority challenges facing NOAA

5. 5 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Response - NOAA’s GEO-IDE Global Earth Observation Integrated Data Environment Scope – NOAA-wide architecture development to integrate legacy systems and guide development of future NOAA environmental data management systems Vision – NOAA’s GEO-IDE is envisioned as a “system of systems” – a framework that provides effective and efficient integration of NOAA’s many quasi-independent systems Approach – evolution of existing systems into a service-oriented architecture built upon agreed standards, principles and guidelines Result – a single system of systems (user perspective) to access the data sets needed to address significant societal questions

6. 6 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Project Management Undersecretary for Oceans and Atmosphere DMIT Data Management Integration Team NOSC NOAA Observing System Council DMC NOAA Data Management Committee All NOAA GoalsAll NOAA Line Offices

7. 7 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Scope Concerned with environmental and geospatial data and information obtained or generated from worldwide sources to support NOAA's mission Does not consider administrative support systems such as finance, personnel, acquisition or facilities management Includes all aspects of data management, including data acquisition, ingest, data processing, archive and access (CLASS)

8. 8 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Vision “System of systems” – a framework to effectively and efficiently integrate NOAA’s many systems Minimize impact on legacy systems Utilize standards Work towards a service-oriented architecture

9. 9 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Approach Each NOAA LO/program/project continues to manage its data independently Standards –Adopt, adapt and only as a last resort, create –Open, inclusive process for adoption –Inclusive not exclusive use of standards Service Oriented Architecture Reference: Federal CIO Council, Jan ’06 "Services and Components Based Architectures: A Strategic Guide for Implementing Distributed and Reusable Components and Services in the Federal Government"

10. 10 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Standards Standard names and terminology Metadata standards –e.g. FGDC and ISO 19115 w/ remote sensing extensions Standard formats for delivery of data/products –WMO, NetCDF, HDF, GeoTIF, JPEG, etc. Web Services Standards –World Wide Web Consortium –OGC (Maps, Features, Coverage, GML) –Community Standards: OPeNDAP, Unidata’s Common Data Model (CDM),COARDS, CF –REST /SOAP / UDDI / WSDL where appropriate

11. 11 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 NOAA GEO-IDE Standards Process Submitted Standard – Standards can be submitted by anyone in NOAA for evaluation to see if they address needs and could be applicable to NOAA Proposed NOAA Standard – Can be provisionally used within NOAA for technical evaluation Recommended NOAA Standard – NOAA data systems should consider supporting the standard wherever applicable for evaluation in real-world NOAA systems NOAA Standard – Approved and mandated where appropriate Phased approach of submission, evaluation and adoption

12. 12 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Fast-track submission An initial set of well known and/or widely used standards are being considered as an initial set of “submitted” standards Discovery Metadata: FGDC, ISO 19115 (w/ extensions), OBIS Keyword Lexicon: Start with Global Change Master Directory Metadata Exchange: XML compliant with FGDC CSDGM and OBIS Catalog Search: Compatible with Geospatial One Stop specifications (currently Z39.50 or OAI-PMH) File Transfer: FTP and HTTP Data Base Access: ODBC and JDBC API and Web Services: OpenDAP and OGC service specifications Data and product formats: The usual suspects…

13. 13 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Service-Oriented Architecture Under an SOA, capabilities are built one at a time to create “Web Services” The fabric of the SOA is built upon standards for: –discovery (e.g. FGDC, ISO, DIF,CSW) –transport (e.g. HTTP, FTP, OPeNDAP) –use (e.g. netCDF, HTML, etc.) Can be SOAP, REST, or some other standard Define core common services and build Service layer agreements across NOAA

14. 14 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 SOA (continued) Operational Public Access Services: for public access to data, products and information services Operational Services: where security, timeliness, and reliability are paramount Scientific Services: where efficient and flexible discovery and access to data sets are required Commercial value-added services Four general classes of web services are anticipated for NOAA:

15. 15 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Key Development Strategies Coordinate activities through Communities of Interest organized by “Data Types” –Grids, time-series, moving-sensor multi-dimensional, profiles, trajectories, geospatial framework, point data and metadata Evolutionary development through pilot projects Minimize impact on legacy systems Develop Services and Component based Architecture (SCBA) using a top-down, bottom- up iterative software development process (Federal CIO Council)

16. 16 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Communities of interest based upon data types An initial list Grids (e.g. model output, gridded data products) Moving-sensor multidimensional fields (e.g. satellite swaths, side-scan sonar, weather radar) Time series (e.g. fish landings, sun spot activity, climate data, paleo-records) Profiles (e.g. atmospheric soundings, ocean casts, profiling floats) Trajectories (e.g. underway ship measurements, aircraft track data, ocean surface drifters) Geospatial Framework Data (e.g. shorelines, fault lines, marine boundaries, map annotations) Point data (e.g. tsunami or seismic occurrences, geodetic control) Metadata - information needed for the use and interpretation of data

17. 17 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Future Direction - Priorities FY07 Work with scientists/data system managers to assess requirements and systems –Develop enterprise architecture and GEO-IDE Implementation Plan –Implement standards process –Active out-reach activities FY08/09 Incrementally execute work packages –Adopt/Adapt/Develop data standards and interoperability mechanisms, e.g., translators and directory services –Direct, test and evaluate changes being made to data management systems FY10/11 Re-evaluate architecture related to new data systems (across NOAA & with national & international partners)

18. 18 WDC for Glaciology 30 th Anniversary Workshop 25 October 2006 Happy Anniversary NSIDC We at NGDC and NESDIS look forward to working with you to integrate your snow and ice data archives into NOAA’s GEO- IDE framework… Questions?