Presentation is loading. Please wait.

Presentation is loading. Please wait.

Comprehensive Large Array-data Stewardship System Status Presented to DAARWG Kern Witcher CLASS Program Manager November 8, 2012.

Similar presentations


Presentation on theme: "Comprehensive Large Array-data Stewardship System Status Presented to DAARWG Kern Witcher CLASS Program Manager November 8, 2012."— Presentation transcript:

1 Comprehensive Large Array-data Stewardship System Status Presented to DAARWG
Kern Witcher CLASS Program Manager November 8, 2012

2 Agenda CLASS Summary Program Overview Data Center Migration
CLASS Capabilities CLASS System Evolution Other Initiatives Challenges CLASS and DAARWG

3 CLASS Summary CLASS meets original intended purpose (Large Volume Data) Suomi-NPP Volume Ingested – 1.32Pb Volume delivered-3.53Pb Files Delivered- over 34M 22 major datasets included AVHRR, GOES, IASI, CFS-R 2.27 Pb, safely stored at 2 sites CLASS must evolve to become a sustainable NOAA enterprise NOAA’s desire for CLASS to be the Enterprise Archive Significant challenges must be addressed

4 Program Overview

5   Program Transition  Program/ Contract Management was transitioned from OSD to NCDC in May of 2011 New WBS and program management staff established Increased communications between CLASS development and Data Centers CY3* extension in Fall of 2011 to end of FY12 CY4 extension though Q1 of FY13 CY5 to begin Q2 of FY13. CY5 delayed to allow for CLASS response to emerging Data Center consolidation data migration activities begin “pivot” or evolution of CLASS to the Enterprise Archive System Effort is dependent on final funding allocations for FY13 Contract may reach ceiling as early as FY15 Acquisition planning needs to begin by January 2013 *- Contract Year 5

6 Organization NGDC NODC CLASS Operations Planning Board NESDIS CIO
COWG NCDC Other NESDIS Contractor Legend Supervisory Reporting Remote Sensing Applications Division CLASS Program Manager – 100% (Kern Witcher) IT Specialist/Systems Engineer - 100% (Jay Morris ) Support Services Division Budget Analyst– 5% (L. Cholid) Inventory Spec – NCDC – 10% (A. Annis, acting ) OMB Exhibit 300 – 30% (T. Cohen) Budget Execution - 5% (T. Leary) COR / Proj Engineer + CWIP – 100% (Jim Goudouros ) ISSO – 100% (Scott Koger) Alt COR – 5% (J. Niemiec ) Government Task Monitor – 30% (N. Ritchey) Operations Manager – 100% (D. Carter)) NESDIS CIO Total of 5 dedicated personnel

7 Current Program Milestones
Notes: CDR: Critical Design Review FOR: Flight Operations Review PDR: Preliminary Design Review SDR: Systems Definition Review SRR: System Requirements Review ORR: Operational Readiness Review NCT: NPP Connectivity Test ICD: Interface Control Document Q3 Q4 Q1 Q2 CLASS GOES-R Campaign NPP Campaign Climate Model Data Nexrad Campaign NDE Campaign Jason-3 Campaign Data Center Migration CLASS Releases FY15 FY16 FY17 Fiscal Year FY09 FY10 FY11 FY12 FY13 FY14 Development Construction Integration/ Test Operations ORR Archive & Access PDR SRR J-1 CDR J-1 SRR J-1 PDR 6/15 CDR v1 CDR v2 PDR Charter ICD CDR 10/11 SDR 2/12 9/12 6/12 7/11 Launch 10/16 4/13 10/15 4/14 10/14 4/15 4/16 R5.2 R5.3 R5.4 R5.5 Current Milestone Completed Milestone SDS v1 R3 R4 Interface PDR Access CDR Interface CDR FOR NCT3 NCT4 NEAAT 1 JPSS-1 R6.0 NPP Launch NEAAT 2 NEAAT 3 NEAAT 4 IDPS Blk 1.5 CDR IDPS Blk 2.0 CDR IDPS Blk 1.5 Ops IDPS Blk 2.0 Ops Currently focused on NESDIS and NWS data sets 7

8 Ongoing Challenges Architectural Boundary Responsibilities
Common vs. Unique Data Ingest Data Center Migration Data Center Consolidation Governance Budget Transition to New Architecture 8

9 CLASS Definition From CLASS Level 1 Requirements (preliminary) signed 2008 Purpose: Support long-term, secure storage of NOAA-approved data, information, and metadata and enable access to these holdings through both human and machine-to-machine interfaces. Capabilities will be provided in 3 primary functional areas (Open Archive Information System Reference Model, OAIS-RM components): Ingest - provide mechanisms by which data, information, and metadata are transferred to and organized within the storage system. Issue: Feasibility – Cannot afford/maintain unique solutions 9

10 CLASS Definition (con’t)
Archival Storage - provide common enterprise means for data, information, and metadata to be stored by the system and the capability to refresh, migrate, transform, update, and otherwise manage these holdings as part of the preservation process. Access - provide common enterprise access capability enabling users to identify, find, and retrieve the data and information of particular interest to the user. Issue: Open to interpretation – IT systems versus Data Center stewardship responsibilities Optimal data access varies by observations (vertical profiles vs. spatial patterns, Gridded vs. swaths, Time series vs. synoptic) 1010

11 CLASS Goals Goals: As an enterprise solution, CLASS will reduce anticipated cost growth associated with storing environmental datasets by: Providing common services for acquisition, security, and project management for the IT system supporting NOAA Archives Consolidating stove-pipe, legacy archival storage* systems Relieving data owners of archival storage-related system development and operations issues Archival storage provides the services and functions for the storage, maintenance and retrieval of archival information packets. Archival storage functions include receiving archival information packets from ingest and adding them to permanent storage, managing the storage hierarchy, refreshing the media in which archive holdings are stored, performing routine and special error checking, providing disaster recovery capabilities, and providing archival information packets to access to fulfill orders 1111

12 High Level Requirements for CLASS
From CLASS Level 1 Requirements (Preliminary 2008) 5.1 Core Mission Requirements 5.1.1 CLASS shall provide defined and documented human and machine-to-machine interlaces by which archives may securely store, maintain, and provide access to their data, information, and metadata holdings for indefinite periods. 5.1.2 CLASS shall ingest, provide long-term, secure storage, and provide access to baseline information holdings 5.1.3 CLASS shall provide long-term, secure storage of and common access to information pertaining to processing of CLASS maintained information holdings, including documentation, processing algorithms, and procedures. 5.1.4 CLASS shall comply with applicable National Archives and Records Administration (NARA) regulations. 5.1.5 CLASS shall initiate pilot programs with the GEO-IDE project to support risk reducing development and phased integration of standards for metadata, machine-to-machine interlaces, and archives. Proposed NOAA Action: Finalize CLASS Level 1 Requirements 12

13 Current CLASS Governance
The CLASS Program has numerous oversight and approval authorities From the CLASS Level I Requirements CLASS is designated a "major" NOAA system in accordance with criteria in NOAA's Administrative Order (NAO) NOAA Observing Systems Council (NOSC) is the designated oversight council and will review the CLASS project at each Key Decision Point (KDP). The project will be reviewed as a major project by the Program Management Council (PMC), unless delegated. As an IT project, the CLASS project will be reviewed by the NOAA Information Technology Review Board (NITRB) and by the Commerce Information Technology Review Board (CITRB). Changes to project baseline, scope, and direction shall be approved by the Deputy Undersecretary for Oceans and Atmosphere Proposed NOAA Action: Provide Clear Lines of Authority and a well defined Governance Structure 13

14 Data Center Migration

15 Data Center Migration Status
Goal, Objective, and Outcomes from NOAA Data Centers Data Migration Plan v1.2 signed May 2011 Goal: Prior to the end of FY 2015, the CLASS Operational System will be the primary safe storage/access capability for all environmental data holdings under the auspices of NOAA’s Data Centers. Objectives: The objective of this plan is migration of all historical environmental data holdings on current Data Center storage systems to CLASS, in addition to the ingest into CLASS [of] near real-time environmental data streams currently received by the Data Centers. This objective involves archival storage as well as elements of ingest, data access, data management, and preservation planning.

16 Data Center Migration Status(con’t)
Starting point Incoming Data Streams already migrated to CLASS include POES, GOES, Jason-2, MetOp Incoming Data Streams solely in CLASS Suomi-NPP, GCOM-W, (future GOES-R, future JPSS, future Jason-3)

17 Data Center Migration Status(con’t)
Data Center Status Each Data Center developing its own individual plans including schedule, metrics and milestones CLASS PM included several pilot activities in CLASS’ Contract Year 4 (CY4) Cloud access pilot (NODC, NCDC) - ongoing NEXRAD pilot (NCDC) - completed NCDC in situ migration prototype activity - completed Next steps to be included in CLASS’ CY5 activities

18 Data Center Migration Status(con’t)
CLASS accomplishments in FY12 for Data Center Migration Aerospace completed Phase I (“as is” analysis) of Data Center Con Ops Completed Data Center Migration Requirements Documents Completed Data Center Migration Interface Control Documents Migrated NCDC Historical Data Set Completed NexRad Pilot 2 week migration test Initiated Cloud Pilot Project

19 Data Center Migration Status(con’t)
Uncertainty regarding FY15 timeline Reductions in CLASS funding available for migration activity could delay implementation Increasing comms costs, operational costs FY13 President’s Budget has increased ORF to solve this Data Center consolidation activities Individual Data Center plans not yet finalized Several possible technological solutions still being evaluated with Data Centers and CLASS

20 Data Center Migration Status(con’t)
Summary The Data Centers have an overall plan in place for the migration of their historical environmental data to CLASS by FY15. Many historical data sets already migrated – 60% by volume – but many smaller sets still need to be completed. Several large data streams already migrated, but many incoming data streams still need to be transitioned to CLASS storage. Each Data Center working on their individual plans and coordinating with CLASS PM. CLASS PM will address next steps in FY13/CY5, contingent upon available funds for migration activities.

21 Current Capabilities

22 Replication via NOAA Science Network(N-WAVE) Direct Connectivity to:
Current CLASS Assets CLASS NGDC, Boulder CO CLASS NSOF, Suitland MD CLASS NCDC, Asheville NC Replication via NOAA Science Network(N-WAVE) Users Development Team Fairmont, WV Functions Operations Ingest Storage (Disk & Tape) Public Access Test and Integration Environment Satellite Landing Zone Development Environment Direct Connectivity to: ESPC- NOAA Environmental Satellite Processing Center National Ice Center NOAA Coast Watch JPSS Interface Data Processing Segment (IDPS) Key Capabilities: Tape Library Capacity – ,000 tape robotic libraries with a total storage capacity of 15.5 Pb (LT04 Tapes/ Native) Spinning Disk Capacity- 2.1 Pb (NSOF, NCDC, NGDC) 10 Gb/sec Internal Network Backbone Redhat Linux OS Server Count- 48 (576 processors) 10 Gb/Sec WAN (N-Wave)

23 CLASS interfaces to NPP and JPSS-1
CURRENT IDPS Block 1.2 NPP CLASS NGDC NCDC STAR NSOF SDS IDPS GRAVITE NWAVE 10Gb/s C3S FUTURE IDPS Block 1.5/2.0 NPP, J-1, J-2 OPS RN I&T RN NWAVE OPS FSN Boulder Asheville = Receipt Node – Data delivery to archive = Full Service Node (Ingest), Archive, and User Access Primary JPSS IDPS Backup IDPS at CBU NSOF Fairmont = CLASS backbone WAN GRAVITE IFCs Backup system for COOP Event

24 CLASS Interface to GOES-R
(In Development) 24

25 Submit service request
  Data Access from CLASS Preservation Planning Administration Ingest Access Data Management Archival Storage PRODUCER CONSUMER requests results Present CLASS Search and find info Submit order Submit service request Submit ad-hoc order Submit standing order Consumer 25

26 NPP Data Dissemination
CLASS to SDS Near-instantaneous subscription delivery (~3.3 TB/day) CLASS to GRAVITE Near-instantaneous subscription delivery (~3.3 TB/day) SDS GRAVITE Public Subscriptions Near-instantaneous delivery Svalbard, Norway CLASS NGDC ~4.7TB/day C3S IDPS CLASS NSOF Public Subscriptions Replication ~20-min IDPS to CLASS 6-hour delay NWAVE 10Gb/s CLASS NCDC 6-hour Delay imposed by CLASS Minimizes retransmissions requests Ensures control of limited distribution files. ~4.7TB/day Public Ad hoc orders STAR Public Ad Hoc Block Orders (up to 18 Tb/day) 50% complete within 0 to 6 hours 27% complete within 6 to 12hours 22% complete within 12 to 72 hours 1% complete over 72 hours Public Ad Hoc Bulk Orders Up to 7 days Process Duration IDPS to CLASS ~ 6 hours File Ingest & catalog populated ~ 20 min NCDC/NGDC Node Replication ~ 22 min Total ~ 6:42 CLASS to STAR Near-instantaneous subscription delivery (~3.3 TB/day) Note: Near- Instantaneous subscription delivery is limited by network bandwidth

27 Data Currently Archived in CLASS
Represent 95% of the Archival Holdings by Volume Dataset Name Period of Record Date First Archived into CLASS Total Volume (TB) Source Suomi National Polar-orbiting Partnership (S-NPP) 10/2011 to present 10/1/2011 NASA/NOAA Geostationary Operational Environmental Satellite (GOES) 1/1979 to present 12/5/2003 310.00 NOAA Climate Forecast System - Reanalysis (CFS-R) 1/1979 to 3/2011 1/20/2010 294.56 Advanced Very High Resolution Radiometer (AVHRR) 10/1978 to present 11/1/2001 99.49 Infrared Atmospheric Sounding Interferometer (IASI) 4/2007 to present 5/1/2007 93.84 EUMETSAT Defense Meteorological Satellite Program (DMSP) 7/1987 to 2/1997 26.53 DoD Advanced Clear-Sky Products over Oceans (ACSPO) 5/2008 to present 5/1/2011 19.28 Coast Watch 2/1989 to present 11/15/2005 15.70 Advanced Scatterometer Level 1B (ASCAT) 2/2007 to present 2/1/2007 10.29 NASA Synthetic Aperture Radar (SAR) 6/1992 to 3/2011 4/30/2003 9.03 Continuously Operating Reference Stations (CORS) 11/2010 to present 10/7/2011 7.79 Jason-2 6/2008 to present 6/22/2008 6.68 Microwave Integrated Retrieval System (MIRS) 8/2007 to present 8/30/2007 6.45 Ocean Color Data Products 5/2000 to present 5/7/2003 6.18 Tiros Operational Vertical Sounder (TOVS) 1/1988 to 12/1995 5.74 Global Nav Satellite Receiver for Atmospheric Sounding Level 1B (GRAS) 9/2007 to present 9/14/2007 2.93 Microwave Surface and Precipitation Products System (MSPPS) 7/2003 to present 7/20/2003 2.09 Aerosol Optical Thickness (100 KM) (AERO100) 11/1998 to present 0.98 Solar Backscatter Ultraviolet Spectral Radiometer Version 2 3/1985 to present 5/23/2006 0.25 Global Change Observation Mission 1 - Water (GCOM-W1) 8/2012 to present 8/22/2012 0.04 JAXA Coral Bleaching Monitoring Products (CORBL) 6/2007 to present 6/6/2007 0.00 NCEP Weather Analyses and Forecast Charts 3/1999 to 5/2012 5/2/2012 Total (Single Node) 27

28 Customer Groupings that Access CLASS
CLASS Users by Domain TB 2012

29   Current Constraints Access limited through CLASS Web site New data sets to archive require significant SW development efforts Implementation of Trusted Internet Connection (TIC) will effect throughput and system performance Access to data holdings have limited on-line disk capability Manual vs. automated operations (ex. Load balancing) CLASS is addressing these constraints though several initiatives and standard software releases that will be discussed as part of the enterprise discussion 29

30 System Evolution Ongoing Initiatives

31 CLASS Projected Archive Volumes

32 NOAA Enterprise Archive System (CLASS)
Cumulative Total Volume by Data Type

33 CLASS Projected Ingest and Delivery

34 CLASS Evolution to an Enterprise Archival System
Evolve the existing CLASS hardware and software infrastructure into a distributed, modular, service-oriented architecture Allow greater flexibility in supporting, to the maximum extent possible, not only large data arrays from satellite programs but also provide additional archival storage services for all of NOAA’s environmental data that has been approved for archive . Working with the NOAA National Data Centers, the new Enterprise Archival Storage architecture will consist of: Generic ingest services for flexible data acquisition, Flexible access services using existing community standard, open-source and emerging technologies such as cloud services, Standardized metadata repository to support a variety of search and discovery services and Long-term, secure archival storage and data management capabilities. .....continued…

35 CLASS Evolution to an Enterprise Archival System(con’t)
Preservation Planning Administration Ingest Access Data Management Archival Storage PRODUCER CONSUMER requests results Present CLASS Satellite NOAA Enterprise Archival Storage System Ingest Interface Interface Diss. Access Interface Administration and Preservation Climate. Gov Ordering Discovery Commercial Cloud Services Dissemination Services Access Services Model Insitu Pre-ingest Services Future CLASS (in green) Data Gateway Private Cloud Services Web Accessible Folders Direct Delivery Admin. Interface CLASS provides components of the OAIS-RM as services and interfaces. Additional systems and services implemented by the data centers. 35

36 Ongoing Initiatives Leading toward the Enterprise Archival System
CLASS pivot points Pre-Ingest Services (Receipt Node/Gateway) Common Access Interface (M2M) Core System Upgrades Data Managers’ Toolkit Rules Base Middleware Common Storage Service (Cloud Pilot Project) Common Interface Definition Satellite NOAA Enterprise Archival Storage System Ingest Interface Interface Diss. Access Interface Administration and Preservation Climate. Gov Ordering Discovery Commercial Cloud Services Dissemination Services Access Services Model Insitu Pre-ingest Services Data Gateway Private Cloud Services Web Accessible Folders Direct Delivery Admin. Interface Receipt Node/ Gateway Performs Data Integrity Checks Provides Temporary Data Storage Provide translation of data if necessary Provides a static XML schema for passing the elements to CLASS Assigns UUID for preservation in CLASS Passes metadata information to search and discovery services 36

37 Ongoing Initiatives Leading toward the Enterprise Archival System
CLASS pivot points Common Ingest Interface (Gateway) Common Access Interface (M2M) Core System Upgrades Data Managers’ Toolkit Rules Base Middleware Common Storage Service (Cloud Pilot Project) Common Interface Definition Satellite NOAA Enterprise Archival Storage System Ingest Interface Interface Diss. Access Interface Administration and Preservation Climate. Gov Ordering Discovery Commercial Cloud Services Dissemination Services Access Services Model Insitu Pre-ingest Services Data Gateway Private Cloud Services Web Accessible Folders Direct Delivery Admin. Interface M2M (Common Access API) Search, order, and holdings information RESTful, Asynchronous, interaction with external systems Returns all data submitted via Common Ingest Interface Metadata publishing to data center catalogs 37

38 Ongoing Initiatives Leading toward the Enterprise Archival System
CLASS pivot points Common Ingest Interface (Gateway) Common Access Interface (M2M) Core System Upgrades Data Managers’ Toolkit Rules Base Middleware Common Storage Service (Cloud Pilot Project) Common Interface Definition Satellite NOAA Enterprise Archival Storage System Ingest Interface Interface Diss. Access Interface Administration and Preservation Climate. Gov Ordering Discovery Commercial Cloud Services Dissemination Services Access Services Model Insitu Pre-ingest Services Data Gateway Private Cloud Services Web Accessible Folders Direct Delivery Admin. Interface Common Storage Service Access area for newly arrived and often used data sets Single interface for all access systems Push once, read many model Utilizes Cloud Services and Cloud technologies in hybrid model Extends CLASS cache out to the Enterprise 38

39 Ongoing Initiatives Leading toward the Enterprise Archival System
CLASS pivot points Common Ingest Interface (Gateway) Common Access Interface (M2M) Core System Upgrades Data Managers’ Toolkit Rules Base Middleware Common Storage Service (Cloud Pilot Project) Common Interface Definition Satellite NOAA Enterprise Archival Storage System Ingest Interface Interface Diss. Access Interface Administration and Preservation Climate. Gov Ordering Discovery Commercial Cloud Services Dissemination Services Access Services Model Insitu Pre-ingest Services Data Gateway Private Cloud Services Web Accessible Folders Direct Delivery Admin. Interface Hardware/OS refresh All CLASS software migrated to Linux on new hardware Significant improvements in performance HPSS migration will provide more reliability and flexibility in configuration options

40 Ongoing Initiatives Leading toward the Enterprise Archival System
CLASS pivot points Common Ingest Interface (Gateway) Common Access Interface (M2M) Core System Upgrades Data Managers’ Toolkit Rules Base Middleware Common Storage Service (Cloud Pilot Project) Common Interface Definition Satellite NOAA Enterprise Archival Storage System Ingest Interface Interface Diss. Access Interface Administration and Preservation Climate. Gov Ordering Discovery Commercial Cloud Services Dissemination Services Access Services Model Insitu Pre-ingest Services Data Gateway Private Cloud Services Web Accessible Folders Direct Delivery Admin. Interface Common Administration Interface Single interface data managers supports: Stewardship tools Metadata updating and versioning Data holdings monitoring and statistics

41 Ongoing Initiatives Leading toward the Enterprise Archival System
CLASS pivot points Common Ingest Interface (Gateway) Common Access Interface (M2M) Core System Upgrades Data Managers’ Toolkit Rules Base Middleware Common Storage Service (Cloud Pilot Project) Common Interface Definition Satellite NOAA Enterprise Archival Storage System Ingest Interface Interface Diss. Access Interface Administration and Preservation Climate. Gov Ordering Discovery Commercial Cloud Services Dissemination Services Access Services Model Insitu Pre-ingest Services Data Gateway Private Cloud Services Web Accessible Folders Direct Delivery Admin. Interface Rules based middleware Data Transport Moves data objects between systems Data Information Sharing Synchronizes metadata between systems Holds data location information Orchestration Implements Enterprise process flow and routes products through Enterprise Controlled by data managers and stewards 41

42 Ongoing Initiatives Leading toward the Enterprise Archival System
CLASS pivot points Common Ingest Interface (Gateway) Common Access Interface (M2M) Core System Upgrades Data Managers’ Toolkit Rules Base Middleware Common Storage Service (Cloud Pilot Project) Common Interface Definition Satellite NOAA Enterprise Archival Storage System Ingest Interface Interface Diss. Access Interface Administration and Preservation Climate. Gov Ordering Discovery Commercial Cloud Services Dissemination Services Access Services Model Insitu Pre-ingest Services Data Gateway Private Cloud Services Web Accessible Folders Direct Delivery Admin. Interface Common Interface Definition Supports interaction between the components in the OAIS-RM Granules Collections Browse Images Static and well documented Common Set of Elements Common Schema Unique Identifiers 42

43 Evolution Roadmap (preliminary)
FY2012 FY2013 FY2014 FY2015 FY2016 NODC NCDC NGDC Phase I Phase II Phase III Cloud Pilot Access Path Metadata Data Net IRODS Concurrent CLASS Initiatives Archive Path Data Center Migration NPP GCOM-W Jason On-Hold Programs Access Dissemination Stewardship Staging Archive Storage Service JPSS GOES-R M2M HPSS MOB

44 CLASS Evolution to an Enterprise Archival System
CLASS will evolve to: Implement well-defined interfaces based on industry-standard protocols and best practices. Reduce the need for costly custom software development and allow the flexibility to use a multitude of different COTS discovery tools such as rules-based middleware for data sharing, search and discovery. Be scalable to support growth in volume and is extensible Add functionality and services as new technologies mature and enterprise needs surface. Provide a significant and rapid return on investment for NOAA and the Nation Enable the efficient and inexpensive archive of many NOAA products that are currently awaiting archival services Enables the Data Centers to provide new services and products to their customers. . Allow the efficient leveraging of other elements of the NOAA Enterprise Ground system, processing centers, distribution capabilities through generic services and interfaces

45 Other Initiatives that Intersect with CLASS

46 Notional Fairmont Consolidation Plan
Federal Data Center Consolidation    Notional Fairmont Consolidation Plan System Configuration NOAA Enterprise Archival System, Fairmont, WV NSOF, Suitland MD NOAA Science Network(N-WAVE) Users Development Team Fairmont, WV Functions Operations Ingest Storage (Disk & Tape) Public Access Test and Integration Environment Satellite Receipt Node/ Gateway Development Environment Direct Connectivity to: Environmental Satellite Processing Center( ESPC) Product Distribution and Access (PDA) National Ice Center NOAA Coastal Watch JPSS Interface Data Processing Segment (IDPS) GOES-R Product Distribution (PD) Direct Connectivity to: Environmental Satellite Processing Center( ESPC) Product Distribution and Access (PDA) Backup JPSS Interface Data Processing Segment (IDPS) Backup GOES-R Product Distribution (PD) Backup Notional Commercial Cloud Services Off-site Backup Facility Data Centers, NOAA Gateways Fairmont Consolidation Approach Phase I – Migrate Development, Test and Integration Environments Phase II- Establish backup Ingest Capabilities for JPSS, GOES-R, PDA Phase III- Migrate NGDC assets to Fairmont Phase IV- Consolidate NCDC assets into Fairmont

47 NESDIS Data Center Consolidation
Implementation team established Begin in 2015 Budget? Complete by 2023? Three Data Centers into One? Final Organizational Structure Plan not yet approved, in preliminary planning phases but is a popular idea within NOAA

48 NESDIS Enterprise Ground System
NOAA/NESDIS assembling EGS to meet its mission Planning phases Technical Reference Model PDA feasibility Study Common Storage Various approaches under consideration include archive, dissemination Driving question- Does CLASS to become the Enterprise Archive part of the EGS? 48

49 CLASS Programmatic Challenges
Support for the Revision and Rebaseline of Level 1 Requirements Analysis of Alternatives (Exhibit 300) Update Full Lifecycle Cost Estimate (GAO Report) Clear program governance structure Adequate Funding for Operations Rebalance of Requirements/Cost/ Schedule for GOES-R and JPSS Budget profile currently does not align with requirements or delivery schedule Funding Demarcation Where does the Program’s funding responsibility end? Post Launch Support Operations Access

50 CLASS Technical Challenges
MetaData- Need to define common elements and format (ISO, ECHO, Etc.) for ingest and access Interoperability- Date Information Exchange Language ( can include meta-data and data formats) for ingest and access Catalog- Search and discovery services outside of CLASS domain (Catalog Services for the Web) Data harvester Storage- External storage for dissemination and access outside the CLASS domain Con Ops- CLASS has a different role for satellite data and insitu data

51 Overlap with DAARWG Issues
Priority Order from a CLASS Prospective: 1. What to archive (Not CLASS but has significant implications) What to Archive needs to be coupled with the Budget to Archive 2. Data format (integration and interoperability)- netCDF4 3. Metadata- Elements and Schema  4. Access Storage – Cloud for access (and dissemination) Access (They do NOT mention Catalogs – maybe that is GEO-IDE???? 5. GEO-IDE  Is it still an active viable program


Download ppt "Comprehensive Large Array-data Stewardship System Status Presented to DAARWG Kern Witcher CLASS Program Manager November 8, 2012."

Similar presentations


Ads by Google