NOAA OneStop and the Cloud Kenneth Casey, PhD Deputy Director, Data Stewardship Division NOAA/NESDIS National Centers for Environmental Information (NCEI) Tuesday, 08 October 2019 WGISS-48 Hanoi, Vietnam

Cloud is a Key NESDIS Strategic Objective The NESDIS Cloud Project!

Business Readiness: People, Process, and Technology

Initiation & Assessment NESDIS Cloud Roadmap Experimentation Migration Transformation Cloud Migration Leverage Cloud Cloud Optimization Initiation & Assessment Proof of Concept Staging & Implementation Develop phasing management plan Prepare cloud environment Orchestrate cloud implementation Organize and consolidate Migrate data to the cloud Migrate services to the cloud Automate cloud services Monitor and manage cloud environment Optimize services, architecture, and cost components Evolve architecture based on new cloud service offerings Develop NESDIS Cloud Strategy Organize IPT(s) Kick-off Business Readiness Assessment Develop ConOps Design and build Pilot Projects Update the organization WE ARE HERE TODAY

No More Stovepipes! Link to NESDIS Cloud Strategy: https://docs.google.com/document/d/1yv8E1jsXH4hcLLL5MO2pp1IfwS4AA5oXW_tD9XQd1ug/edit

NESDIS Enterprise Cloud Vision Science and Development Sandbox Storage Product Generation Consolidated Ingest Distribution and Access Users Configuration and Program Office Software and Release Management Developers Metadata Catalog Continuous Integration Continuous Deployment Data Metadata Metadata Query

Functional Focus Areas Functional Areas Scope Security in the Commercial Cloud Develop NESDIS Authority to Operate (ATO) methodology to meet security standards in the commercial FedRAMP Moderate cloud by tailoring up security controls Ingest, Cataloging, and Storage Extend NESDIS common data ingest, cataloging, and storage services and demonstrate this capability in the commercial cloud Product Generation and Science Development Demonstrate that industry approaches for compute and science development can be used in a common architecture to meet all current NESDIS system needs in the commercial cloud Dissemination and Access Prototype a NESDIS common architecture for cloud-based data distribution that meets NESDIS requirements

Functional Focus Areas Functional Areas Example Activities Security in the Commercial Cloud Create the NESDIS Cloud ATO process and checklist Document cloud security guidance Ingest, Cataloging, and Storage Implement an Ingest function that builds on the Secure Ingest Gateway Pilot (SIGP) and is source agnostic Demonstrate full data lifecycle Catalog and Storage services Product Generation and Science Development Demonstrate 10 science algorithms can be run as containers in the Product Generation environment For those 10, show that the science teams can work with the algorithms in the Science Development Sandbox Dissemination and Access Demonstrate ability to serve data to simulated partners Demonstrate OneStop interface for data discovery/access, along with a tailored interface for internal science users

Multiple Products Generated Multiple Data Streams being Ingested Progress to Date Multiple Products Generated Multiple Data Streams being Ingested All Available for Discovery and Access Monitoring Dashboards

Ongoing Work Training Cost Modeling Communications

Focus in on OneStop

OneStop in a Nutshell To improve the discovery of, access to, and usability of NOAA’s vast and diverse collection of big data Modern, user-friendly discovery interface Open infrastructure, standards, and best practices

What Have We Achieved? > 378 collections made OneStop-ready in 15 groups Tiers of OneStop readiness (Bronze-Silver-Gold) established OneStop metadata and data format best practices documented Data Stewardship Maturity Model assessments conducted at scale, incorporated into ISO metadata, and used as basis for streamlined Data Stewardship Maturity Questionnaire (DSMQ) ISO-lite granule metadata spec with millions generated

What Have We Achieved?

What Have We Achieved? NOAA OneStop https://www.ncei.noaa.gov/onestop

What Have We Achieved? Goal: 66% of users prefer OneStop UI over existing NOAA Data Catalog Achieved: 83%

What Have We Achieved? Initial Common Metadata Editor Tool (CoMET) deployed and available for stakeholder testing Incorporated DSMQ into CoMET Collection Manager supporting tools (Docucomp, CEdit, Metaview, Metaserver, Pipelines, WAFs) operationalized Training and user feedback collected at Sept 2019 NOAA Environmental Data Management Workshop

What Have We Achieved? User’s Guide and DSMQ glossary published to NOAA EDM Wiki MediaCurrent Third-Party Usability Testing Completed a second round of UI testing in late 2018 OneStop 2.1.2 user interface incorporates much of the professional feedback and is live CoMET testing completed as well, with feedback being incorporated into latest release (focus on “golden paths”!) Established initial framework for the Open Data Framework we have been calling the Mission Data Management System….

NCEI’s Open Data Framework30,000 ft Data Producers and Operators Data Managers and Data Scientists Data Consumers and the Public Data Ingest Services Metadata Management Services (Inventory Manager) Search and Discovery Services (OneStop) Archival Storage and Access Services

NCEI’s Open Data Framework10,000 ft Data Producers and Operators Data Managers and Data Scientists Data Consumers and the Public Data Ingest Services Metadata Management Services Search and Discovery Services Common Ingest S2N ATRAC OneStop UI and Thin Portals OneStop API Collection Manager Inventory Manager Archival Storage Services Tape Disk Cloud Order Fulfillment + Access Services (via Mission Science Network)

NCEI’s Open Data Framework2,000 ft Collection Manager OneStop API/ Search Engine Index External Systems CEdit Rubrics CoMET Metaserver Data Managers OneStop UI EMMA CoMET Collection M/D DocuComp Thin Portals Public C G Common Ingest Data Streams Inventory Manager workflows Order Fulfillment + Access Services Storage Services M/D Event Log Tape Disk Cloud S2N + ATRAC C is for Collections, G is for Granules Data Producers and Operators Ingest Services MD Services Search Services Storage/Access Services

Summary and Next Steps OneStop’s Open Data Framework components at different stages of maturity, some fully operational other just getting started… Integrate the components to improve operational robustness Next UI release will include login.gov capability Improve CoMET tools through user testing and iterative feedback Continue Inventory Manager development Iteratively improve the OneStop discovery UI Working with the NESDIS Cloud IPT to stay in sync where possible and prepare for eventual cloud migration

Questions?