1. NOAA OneStop and the Cloud
Kenneth Casey, PhD
Deputy Director, Data Stewardship Division
NOAA/NESDIS National Centers for Environmental Information (NCEI)
Tuesday, 08 October 2019
WGISS-48
Hanoi, Vietnam

2. Cloud is a Key NESDIS Strategic Objective
The NESDIS Cloud Project!

3. Business Readiness: People, Process, and Technology

4. Initiation & Assessment
NESDIS Cloud Roadmap
Experimentation
Migration
Transformation
Cloud Migration
Leverage Cloud
Cloud
Optimization
Initiation & Assessment
Proof of Concept
Staging &
Implementation
Develop phasing management plan
Prepare cloud environment
Orchestrate cloud implementation
Organize and consolidate
Migrate data to the cloud
Migrate services to the cloud
Automate cloud services
Monitor and manage cloud environment
Optimize services, architecture, and cost components
Evolve architecture based on new cloud service offerings
Develop NESDIS Cloud Strategy
Organize IPT(s)
Kick-off Business Readiness Assessment
Develop ConOps
Design and build Pilot Projects
Update the organization
WE ARE HERE TODAY

5. No More Stovepipes!
Link to NESDIS Cloud Strategy:

6. NESDIS Enterprise Cloud Vision
Science and Development Sandbox
Storage
Product Generation
Consolidated Ingest
Distribution and Access
Users
Configuration and Program Office
Software and Release Management
Developers
Metadata Catalog
Continuous Integration
Continuous Deployment
Data
Metadata
Metadata Query

7. Functional Focus Areas
Functional Areas
Scope
Security in the Commercial Cloud
Develop NESDIS Authority to Operate (ATO) methodology to meet security standards in the commercial FedRAMP Moderate cloud by tailoring up security controls
Ingest, Cataloging, and Storage
Extend NESDIS common data ingest, cataloging, and storage services and demonstrate this capability in the commercial cloud
Product Generation and Science Development
Demonstrate that industry approaches for compute and science development can be used in a common architecture to meet all current NESDIS system needs in the commercial cloud
Dissemination and Access
Prototype a NESDIS common architecture for cloud-based data distribution that meets NESDIS requirements

8. Functional Focus Areas
Functional Areas
Example Activities
Security in the Commercial Cloud
Create the NESDIS Cloud ATO process and checklist
Document cloud security guidance
Ingest, Cataloging, and Storage
Implement an Ingest function that builds on the Secure Ingest Gateway Pilot (SIGP) and is source agnostic
Demonstrate full data lifecycle Catalog and Storage services
Product Generation and Science Development
Demonstrate 10 science algorithms can be run as containers in the Product Generation environment
For those 10, show that the science teams can work with the algorithms in the Science Development Sandbox
Dissemination and Access
Demonstrate ability to serve data to simulated partners
Demonstrate OneStop interface for data discovery/access, along with a tailored interface for internal science users

9. Multiple Products Generated Multiple Data Streams being Ingested
Progress to Date
Multiple Products Generated
Multiple Data Streams being Ingested
All Available for
Discovery and Access
Monitoring
Dashboards

10. Ongoing Work
Training
Cost
Modeling
Communications

11. Focus in on OneStop

12. OneStop in a Nutshell
To improve the discovery of, access to, and usability of NOAA’s vast and diverse collection of big data
Modern, user-friendly discovery interface
Open infrastructure, standards, and best practices

13. What Have We Achieved?
> 378 collections made OneStop-ready in 15 groups
Tiers of OneStop readiness (Bronze-Silver-Gold) established
OneStop metadata and data format best practices documented
Data Stewardship Maturity Model assessments conducted at scale, incorporated into ISO metadata, and used as basis for streamlined Data Stewardship Maturity Questionnaire (DSMQ)
ISO-lite granule metadata spec with millions generated

14. What Have We Achieved?

15. What Have We Achieved?
NOAA OneStop

16. What Have We Achieved?
Goal: 66% of users prefer OneStop UI over existing NOAA Data Catalog
Achieved:
83%

17. What Have We Achieved?
Initial Common Metadata Editor Tool (CoMET) deployed and available for stakeholder testing
Incorporated DSMQ into CoMET
Collection Manager supporting tools (Docucomp, CEdit, Metaview, Metaserver, Pipelines, WAFs) operationalized
Training and user feedback collected at Sept 2019 NOAA Environmental Data Management Workshop

18. What Have We Achieved?
User’s Guide and DSMQ glossary published to NOAA EDM Wiki
MediaCurrent Third-Party Usability Testing
Completed a second round of UI testing in late 2018
OneStop user interface incorporates much of the professional feedback and is live
CoMET testing completed as well, with feedback being incorporated into latest release (focus on “golden paths”!)
Established initial framework for the Open Data Framework we have been calling the Mission Data Management System….

19. NCEI’s Open Data Framework30,000 ft
Data Producers and Operators
Data Managers and Data Scientists
Data Consumers and the Public
Data Ingest Services
Metadata Management Services
(Inventory Manager)
Search and Discovery Services
(OneStop)
Archival Storage and Access Services

20. NCEI’s Open Data Framework10,000 ft
Data Producers and Operators
Data Managers and Data Scientists
Data Consumers and the Public
Data Ingest Services
Metadata Management Services
Search and Discovery Services
Common Ingest
S2N
ATRAC
OneStop UI and Thin Portals
OneStop API
Collection Manager
Inventory Manager
Archival Storage Services
Tape
Disk
Cloud
Order Fulfillment + Access Services
(via Mission Science Network)

21. NCEI’s Open Data Framework2,000 ft
Collection Manager
OneStop API/ Search Engine
Index
External Systems
CEdit
Rubrics
CoMET
Metaserver
Data Managers
OneStop UI
EMMA
CoMET
Collection M/D
DocuComp
Thin Portals
Public C G
Common Ingest
Data Streams
Inventory Manager
workflows
Order Fulfillment + Access Services
Storage Services
M/D Event Log
Tape
Disk
Cloud
S2N + ATRAC
C is for Collections, G is for Granules
Data Producers and Operators
Ingest Services
MD Services
Search Services
Storage/Access Services

22. Summary and Next Steps
OneStop’s Open Data Framework components at different stages of maturity, some fully operational other just getting started…
Integrate the components to improve operational robustness
Next UI release will include login.gov capability
Improve CoMET tools through user testing and iterative feedback
Continue Inventory Manager development
Iteratively improve the OneStop discovery UI
Working with the NESDIS Cloud IPT to stay in sync where possible and prepare for eventual cloud migration

23. Questions?