Thoughts on Stewardship, Archive, and Access to the National Multi- Model Ensemble (NMME) Prediction System Data Sets John Bates, Chief Remote Sensing Applications Division NCDC/NESDIS/NOAA

Outline Climate Information Stewardship Value Chain and Architecture How to Archive Trade Space Defining an Architecture for NOAA’s Climate Service Modeling Archive and Access Conclusions

Draft End-to-End NCEP-NCDC CFSRR Information Stewardship Flow End-to-End Climate Activities NCDC-NCEP Models - Mapping, Responsibilities, and Funding Risks (stoplights) OBSERVING SYSTEMS INPUT MODEL- ASSIMILATION VERSION RAW DATA RECORDS CALIBRATION & VALIDATIONCSF DATA RECORDS ARCHIVE DEVELOPMENT, ARCHIVE O&M, LONG-TERM STEWARDSHIP CFS VERIFICATION AND VALIDATION AGAINST ECVS CLIMATE MODEL DATA PORTAL USER COMMUNITY Specific CFS Model and Assimilation Version - NCEP Lead NESS ESPC; CS NCDC CDR; NESS JPSS-IDPSNESS STARNCEPCS NCDC CS NCDC CDR PROGRAM CS NCDC CDMP PROGRAM CLIMATE SERVICE PORTAL DATA TAB Climate information stewardship is a concept that combines the numerous iterative steps involved in the creation of climate data records including; instrument testing on the ground, calibration and validation of the instrument and products, archival and preservation of relevant data and provenance of the data flow, comparisons and assessments of the products. All of this information must be gathered, organized, and preserved for future generations A similar approach is required to identify information stewardship steps required by NOAA climate modeling activities

Satellite & In Situ Observations Satellite & In Situ Observations Sustained Climate Information Flow Notional Architecture Satellite data Environmental Data Records Long-term Information Preservation Interim Climate Data Records (ICDRs) Essential Climate Variables Climate Information Records Short scale physical phenomena monitoring Operational Climate Monitoring supporting Climate Services Longer term climate variability & climate change analysis Adaptation + mitigation planning (decision making) Sustained Applications Major model-based Reanalysis Near Real Time Re-calibration Inter-calibration Reprocessing Observing system performance Monitoring and automated corrections Data conversion User Services Archives Inter-calibration Sustained Coordinated Processing SCOPE-CM Thematical Climate Data Records (TCDRs) Fundamental Climate Data Records (FCDRs)

CDR Systems Engineering Architecture

Satellite & In Situ Observations Satellite & In Situ Observations We Need to Map a NMME-NCDC Architecture Against a Continuum of Latency-Accuracy-Cost Space Satellite data Environmental Data Records Long-term Information Preservation Interim Climate Data Records (ICDRs) Essential Climate Variables Climate Information Records Short scale physical phenomena monitoring Operational Climate Monitoring supporting Climate Services Longer term climate variability & climate change analysis Adaptation + mitigation planning (decision making) Sustained Applications Major model-based Reanalysis Near Real Time Re-calibration Inter-calibration Reprocessing Observing system performance Monitoring and automated corrections Data conversion User Services Archives Inter-calibration Sustained Coordinated Processing SCOPE-CM Thematical Climate Data Records (TCDRs) Fundamental Climate Data Records (FCDRs) CFS Vers ion X CFS Vers ion Y CFS Vers ion Z

Defining a NOAA Archive Architecture Differing information producers and consumers reflect the wide diversity of NOAA’s products and services – How is information produced? – How is information consumed – How can societal needs be best served in connecting producers and consumers in the most cost effective manner? Need a flexible and re- usable digital archive architecture that balances all these needs Information Producers Information Consumers Present and Future Societal Benefits Digital Archive Architecture

Defining Enterprise Archive Solution Space Drivers Represent Enterprise Archive in terms of three separate dimensions: – Producer-Archive Needs – an agreed to architecture and maturity for products and services – Long-term Preservation and Information Stewardship - a set of measures of the long-term sustainability of the information in a data set – Archive-Consumer Needs - a set of metrics that assess the potential and actual value of a data set for societal needs Producer-Archive Needs Archive- Consumer Needs Long-term Preservation & Information Stewardship Optimal Archive Solution for A Given Producer-Consumer CMIP Lesson – Define a common format (NetCDF-4), variable names (CF names) and coordinate system

Defining an Architecture for NOAA’s Climate Service Modeling Archive and Access – Trade Space Ideas CMIP-5 Approach – Federated Archive and Access using Earth System Grid (ESG) – Pros – Cost effective and responsive stewardship – Cons – Each center must set up and host a site NCDC Archive – Single archive using known NOAA archive processes – Pros – Archive, stewardship, and access routes are well established and proven – Cons – Difficult and cost prohibitive to transfer many PBs of data Modified NCDC/CMIP Approach – Only model data used in official U.S., International assessments are archived at NCDC; Other model data are made available on a select number of federated ESG sites – Pros – Critical data are archived for official use and other less mature data are broadly available – Cons – This approach still requires resources and considerable coordination

Conclusions For NMME products to be more valuable to users, they need to have documentation of all steps in the stewardship value chain CMIP is an excellent example of how agreement on data formats, names, and coordinate systems lead to widespread ease of use of model data NCDC would like to engage early efforts that might be put into operations to help find the most effective and efficient ways to preserve and distribute that information