December 10, 2010

National Climatic Data Center Background System Development Operational Integration Recent Accomplishments Future Direction Next Steps December 10, 2010 DAARWG Meeting: CLASS Update 2

National Climatic Data Center No place to put mountain of data. Manage rapidly growing data volume of major observing and modeling systems Every program solving the same problem. Eliminate various "stove-pipe” systems and produce a unified "enterprise” data access system to reduce IT cost Satellite Active Archive (SAA) GOES Active Archive (GAA) Earth Observing System (EOS) Archive Can’t access the data we have. Centralize NOAA’s numerous data systems for environmental data access-- create a single portal Don’t break anything. Retain, as much as possible, portions and modules of existing legacy systems December 10, 2010 DAARWG Meeting: CLASS Update 3

National Climatic Data Center Scope. Enterprise-wide IT system supporting long- term, secure storage of and common access to environmental datasets and information stewarded by NOAA’s Archives. 1.Large Data Campaigns. Satellites (NPP/JPSS, GOES, POES, DMSP, MetOp), Radar (NEXRAD), Models 2.Enterprise Approach Providing common services for development and operation of IT systems supporting NOAA Archives Consolidating legacy archival storage systems Relieving data producers of responsibility for archival December 10, 2010 DAARWG Meeting: CLASS Update 4

National Climatic Data Center The CLASS support contract was awarded competitively on June 20, 2008 to Diversified Global Partners (DGP) JV LLC Small business set-aside – 8(a) mentor- protégé program Protégé Company: DB Consulting Group Mentor Company: Global Science and Technology (GST) Inc. Potential nine-year period of performance – Now in year 3: Base Year Four (4) one-year Option Periods Four (4) one-year Award Term Option Periods Indefinite Delivery/Indefinite Quantity (IDIQ) contract Maximum Ordering Volume of $200M (maximum of nine years) Cumulative Tasking to date valued at $42.0M December 10, 2010 DAARWG Meeting: CLASS Update 5 NGDC (Ops) Boulder, CO NCDC (Ops) Asheville, NC Fairmont, WV (Devel) NSOF (Devel/Ingest) Suitland, MD

National Climatic Data Center Systems Software Data Integration December 10, 2010 DAARWG Meeting: CLASS Update 6

National Climatic Data Center CLASS Software Evolution (CLASS-SE). Five-year project to establish configurable ingest NOAA Enterprise Archive Access Tool (NEAAT). Enterprise Application Program Interface (API) December 10, 2010 DAARWG Meeting: CLASS Update 7 Stakeholders CLASS System Producer Consumer Ingest Preservation Planning Administration Data Management Archival Storage Access System Administrators Data Stewards OAIS-RM

National Climatic Data Center CLASS-SE provides configurable ingest capability Configurability reduces development costs for storage and access of new data types Applications, or services, are ‘data agnostic’ and may be applied against selected data types Workflow engine supports both Collective system operations & resource allocation Allocates services from 1 to N instances of CLASS nodes Individual Nodes may perform all or sub-set of system capabilities December 10, 2010 DAARWG Meeting: CLASS Update 8

National Climatic Data Center Enterprise access to all NOAA archive storage systems Satisfies L1RD Requirements CLASS Access Interface Support to GEO/IDE Interface to Legacy Systems Supports both data access and stewardship applications Service-Oriented Architecture (SOA) Middleware Simple plugin adaptor in integration layer provides interface to NEAAT Support for open source tool kits (i.e., OGC) OPeNDAP protoype provides access to climate model data data through NOMADS (National Operational Model Archive & Distribution System) December 10, 2010 DAARWG Meeting: CLASS Update 9 Legacy Systems NEAAT Plugin Satellites CFSR CLASS NARR NOMADS NCDC HDSS NCEP Models ESG INE IDEAS ESSE ESG INE IDEAS ESSE SPIDR Plugin Standard Protocols (OPeNDAP) Standard Protocols (OPeNDAP) Customer Application Customer Application Data Migration

National Climatic Data CenterDecember 10, 2010 DAARWG Meeting: CLASS Update 10 Data SetPhaseStatus GOEScompleteoperational POEScompleteoperational DMSPcompleteoperational MetOpcompleteoperational EOS MODIS-canceled Jason 1&2completeoperational Jason 3not startedundefined, expected volume & complexity NPPin progresssee next slide JPSS (NPOESS)planningrequirement added to JPSS L1RD GOES-Rin progress Completed Systems Definition Review & Systems Requirement Review, preparing for Preliminary Design Review NEXRADplanning NCEP ModelsplanningCFSR complete, need to refine requirement

National Climatic Data CenterDecember 10, 2010 DAARWG Meeting: CLASS Update 11

National Climatic Data Center First climate model data archived on CLASS; access through NOMADS 245 TB of CFS reanalysis in tape archive 100 TB of “high-priority” data available on disk for rapid access Just began ingest of reforecast NCDC-NCEP-CLASS Project partnership Reuse of existing spinning disk Agile Software Development; rapid integration of open source access protocol One of NCEPs most important data sets Significant jump in NCDC data access December 10, 2010 DAARWG Meeting: CLASS Update 12

National Climatic Data Center First NOS data set in CLASS Complete: CORS File Naming Convention document signed May 2010 NGDC has established a centralized ingest interface to CLASS In-process: Interface Control Document CORS goals for CLASS version 5.4 release: archive of forward-looking RINEX files (3173 daily files) and metadata daily ingest ~ 49.5 GB/day Future CORS goals, pending success of CLASS archive: archive of forward-looking binary files archive of historical RINEX and binary files archive of NGS reanalysis data current NGDC archive total: ~69.0 TB December 10, 2010 DAARWG Meeting: CLASS Update 13

National Climatic Data CenterDecember 10, 2010 DAARWG Meeting: CLASS Update 14 Simple model based on preservation through two-site replication. Ingest Node Ingest Node NSOF Full Node Full Node NCDC Full Node Full Node NGDC Replication Archive Data Sources

National Climatic Data CenterDecember 10, 2010 DAARWG Meeting: CLASS Update 15 Data Stewardship Data Stewardship Sub-Node (Federated) Sub-Node (Federated) Center of Data Center of Data Processing Node Processing Node Data Processing Data Processing Ingest Node Ingest Node Data Producer Data Producer Full Node Full Node Data Center Data Center Cloud Access Increase distribution of nodes Federate with Centers of Data providing tiers of service Exploit cloud resources for faster access Becoming more H/W agnostic

National Climatic Data Center NCDC partnership with RENCI and DataNet Consortium Prototype “system of systems” framework; federation of NOAA data systems with NOAA archive using iRODS (Integrated Rule-Oriented Data System) Connectivity to data systems such as RENCI, ORNL, OOI, and Earth System Grid (ESG) Pilot Project Federate with RENCI to share 70TB of NEXRAD data Utilize highly distributed computing to derive climate- quality precipitation re-analysis, push data products to NCDC archive system (CLASS) Future Plans to Support Climate Assessments Federate with NOS systems via RENCI to integrate data from OOI with climate data at NCDC Federate with GFDL and ESG to integrate climate model data with in situ and satellite data at NCDC December 10, 2010 DAARWG Meeting: CLASS Update 16

National Climatic Data Center #1 Priority: NPP Operational Test & Evaluation Prepare FY13 Submission Release of 5.4.1; implement final version of NEAAT Complete Cloud Computing Study Establish Archive Architecture and ConOps Prepare for Transition to Climate Service Stand-up Project Management Staff Program Review Migrate data from legacy systems December 10, 2010 DAARWG Meeting: CLASS Update 17

National Climatic Data Center Need for NOAA-level focus on enterprise infrastructure (beyond “comm-lines”), i.e., NOAA Program for GEO-IDE development and fielding Need for NOAA-level policies, directives, and concepts to constraint operational practices and guide IT investments, i.e., CLASS December 10, 2010 DAARWG Meeting: CLASS Update 18

Scott Hausman Acting Director NOAA’s National Climatic Data Center (NCDC) 151 Patton Avenue, Room 557 Asheville, NC   