GODAE High Resolution Sea Surface Temperature Pilot Project GHRSST-PP and the NOAA-NASA Collaboration for Data Management and Reanalysis Kenneth Casey NOAA/NESDIS/NODC

NOAA-NASA GHRSST-PP Overview October 20, 2004 (2) GHRSST-PP Objectives To deliver a new generation of operational SST products To deliver a new generation of operational SST products To ensure that duplication of SST activities are minimized To ensure that duplication of SST activities are minimized To implement an operationally efficient methodology for real time fusion of SST data To implement an operationally efficient methodology for real time fusion of SST data To develop and foster considerable scientific and operational knowledge during the lifecycle of the GHRSST-PP To develop and foster considerable scientific and operational knowledge during the lifecycle of the GHRSST-PP

NOAA-NASA GHRSST-PP Overview October 20, 2004 (3) International Framework

NOAA-NASA GHRSST-PP Overview October 20, 2004 (4) International Framework Long Term Stewardship and Reanalysis Facility (delayed mode users) RDAC and GDAC (near real time users) RDACs

NOAA-NASA GHRSST-PP Overview October 20, 2004 (5) GHRSST-PP Science Team Craig Donlon (Chair), Hadley Centre for Climate Prediction and Research, Met Office UK Craig Donlon (Chair), Hadley Centre for Climate Prediction and Research, Met Office UK Bill Emery, University of Colorado, USA Bill Emery, University of Colorado, USA Chelle Gentemann, Remote Sensing Systems, USA Chelle Gentemann, Remote Sensing Systems, USA Chris Mutlow, Rutherford Appleton Laboratory, UK Chris Mutlow, Rutherford Appleton Laboratory, UK Doug May, Naval Oceanographic Office, USA Doug May, Naval Oceanographic Office, USA Gary Wick, NOAA/OAR ETL, USA Gary Wick, NOAA/OAR ETL, USA Ian Barton, CSIRO Marine Research, Australia Ian Barton, CSIRO Marine Research, Australia Ian Robinson, Southampton Oceanography Centre, UK Ian Robinson, Southampton Oceanography Centre, UK Jim Cummings, Naval Research Laboratory/US-GODAE, USA Jim Cummings, Naval Research Laboratory/US-GODAE, USA Richard Reynolds, NOAA/NESDIS NCDC, USA Richard Reynolds, NOAA/NESDIS NCDC, USA Neville Smith, BMRC, Australia Neville Smith, BMRC, Australia Hiroshi Kawamura NASDA/University of Tohoku, Japan Hiroshi Kawamura NASDA/University of Tohoku, Japan Nick Rayner, Hadley Centre for Climate Prediction and Research, Met Office UK Nick Rayner, Hadley Centre for Climate Prediction and Research, Met Office UK Peter Minnett, RSMAS, University of Miami, USA Peter Minnett, RSMAS, University of Miami, USA Bob Evans, RSMAS, University of Miami, USA Bob Evans, RSMAS, University of Miami, USA Pierre LeBorgne, Meteo France O&SI SAF, France Pierre LeBorgne, Meteo France O&SI SAF, France Andy Harris, NOAA/NESDIS ORA, USA Andy Harris, NOAA/NESDIS ORA, USA Ed Armstrong, JPL PO.DAAC, USA Ed Armstrong, JPL PO.DAAC, USA Ken Casey, NOAA/NESDIS NODC, USA Ken Casey, NOAA/NESDIS NODC, USA Jorge Vasquez, JPL, PO.DAAC, USA Jorge Vasquez, JPL, PO.DAAC, USA David Llewellyn-Jones, Univ. of Leicester, UK David Llewellyn-Jones, Univ. of Leicester, UK Andrew Bingham. JPL PO.DAAC, USA Andrew Bingham. JPL PO.DAAC, USA Jean-Francois Piolle, IFREMER, France Jean-Francois Piolle, IFREMER, France Chris Merchant, Southampton Oceanography Centre, UK Chris Merchant, Southampton Oceanography Centre, UK

NOAA-NASA GHRSST-PP Overview October 20, 2004 (6) GHRSST-PP Outputs An operational data stewardship, exchange and delivery system An operational data stewardship, exchange and delivery system Standardized, real-time L2P SST products Standardized, real-time L2P SST products Real time L4 SST (gap free, gridded) Real time L4 SST (gap free, gridded) Reprocessed L4 SST for Climate Data Records Reprocessed L4 SST for Climate Data Records High Resolution Diagnostic Data Sets (HR-DDS) High Resolution Diagnostic Data Sets (HR-DDS) Matchup Data Base (MDB) Matchup Data Base (MDB) Master Metadata Repository (MMR) Master Metadata Repository (MMR) L4 L2P

NOAA-NASA GHRSST-PP Overview October 20, 2004 (7) Users and Applications

NOAA-NASA GHRSST-PP Overview October 20, 2004 (8) Users and Applications Sampling from one RDACs User Requirement’s Document: MERCATOR MERCATOR MET OFFICE, UK, FORECASTING OCEAN ASSIMILATION MODEL (FOAM) MET OFFICE, UK, FORECASTING OCEAN ASSIMILATION MODEL (FOAM) MEDITERRANEAN OCEAN FORECASTING SYSTEM (MFS) MEDITERRANEAN OCEAN FORECASTING SYSTEM (MFS) EUROPEAN CENTRE FOR MEDIUM RANGE WEATHER FORECASTING EUROPEAN CENTRE FOR MEDIUM RANGE WEATHER FORECASTING TOWARDS AN OPERATIONAL PREDICTION SYSTEM FOR THE NORTH ATLANTIC EUROPEAN COASTAL ZONES (TOPAZ) TOWARDS AN OPERATIONAL PREDICTION SYSTEM FOR THE NORTH ATLANTIC EUROPEAN COASTAL ZONES (TOPAZ) BALTIC OPERATIONAL OCEANOGRAPHIC SYSTEM (BOOS) BALTIC OPERATIONAL OCEANOGRAPHIC SYSTEM (BOOS) NORWEGIAN METEOROLOGICAL INSTITUTE (MET.NO) NORWEGIAN METEOROLOGICAL INSTITUTE (MET.NO) DANISH METEOROLOGICAL INSTITUTE DANISH METEOROLOGICAL INSTITUTE ROYAL DANISH ADMINISTRATION OF NAVIGATION AND HYDROGRAPHY ROYAL DANISH ADMINISTRATION OF NAVIGATION AND HYDROGRAPHY NORWEGIAN POLAR INSTITUTE NORWEGIAN POLAR INSTITUTE NORWEGIAN MET. INSTITUTE’S MARINE FORECASTING CENTRE NORWEGIAN MET. INSTITUTE’S MARINE FORECASTING CENTRE BLACK SEA GOOS BLACK SEA GOOS WMO AFFILIATED INTERNATIONAL PROJECTS WMO AFFILIATED INTERNATIONAL PROJECTS PROUDMAN OCEANOGRAPHIC LABORATORY PROUDMAN OCEANOGRAPHIC LABORATORY

GHRSST Overview Talk A Collaborative Effort Between NOAA/NODC and NASA/PO.DAAC October 20 th, 2004 PO.DAAC Team Manager: Pat Liggett Deputy Manager: Robert Benada Sue Heinz: Raytheon Manager Jorge Vazquez Edward Armstrong Andrew Bingham Chris Finch Rosanna Sumagaysay

NOAA-NASA GHRSST-PP Overview October 20, 2004 (10) Outline GHRSST Processing System GHRSST Processing System –RDACs, GDAC, Archive –Data Products –Data Flow –Oceanids –Data Volumes GHRSST Data Management GHRSST Data Management –MMR –MDB –User Services NASA/NOAA Significance NASA/NOAA Significance

NOAA-NASA GHRSST-PP Overview October 20, 2004 (11) GHRSST Processing System RDACs, GDAC & Archive

NOAA-NASA GHRSST-PP Overview October 20, 2004 (12) GHRSST Processing System Data Products

NOAA-NASA GHRSST-PP Overview October 20, 2004 (13) Level 2 Preprocessed data (L2P) Level 2 Preprocessed data (L2P) –Level 2 data from each sensor in a standard netcdf format that has complete error statistics associated with each sensor. GHRSST Processing System Data Products High Resolution Diagnostic Data Sets (HR-DDS) High Resolution Diagnostic Data Sets (HR-DDS) –L2P data extracted in given predefined boxes to be used for validation studies. All data from different sensors co-located on common grid.

NOAA-NASA GHRSST-PP Overview October 20, 2004 (14) GHRSST Processing System Data Products Recommended data products for Near Real- time SST fields Recommended data products for Near Real- time SST fields –L2 SST, Cloud Fraction, Internal SST Error Estimate (rms uncertainty) Using AIRS Data Within the GHRSST-PP Recommended data products for GHRSST-PP Diagnostic Data Studies Recommended data products for GHRSST-PP Diagnostic Data Studies – IR Spectral Radiance and Cloud- Cleared Radiance L2, L3 SST – Boundary layer moisture/temperature (total column and layered) – Surface emissivity – Cloud fraction and cloud-top pressure – Cloud flags (including aerosol) Source: Denise Hagan, JPL

NOAA-NASA GHRSST-PP Overview October 20, 2004 (15) GHRSST Processing System Data Products Map of Global Sites for GHRSST Diagnostic Data Sets Example of Single Sensor Error Characterization for AIRS based on comparison with shipboard in situ MAERI data (Caribbean): Mean Bias (N=120) RMS (N=120) Source: Denise Hagan, JPL – AIRS Valuable Reference Sensor for GHRSST Diagnostic Data Sets and SST Error Assessment

NOAA-NASA GHRSST-PP Overview October 20, 2004 (16) GHRSST Processing System Data Products Level 4 Merged Analyzed SST products Level 4 Merged Analyzed SST products –10km global product (Remote Sensing Systems) –2 km very high resolution regional products, including areas such as the Mediterranean

NOAA-NASA GHRSST-PP Overview October 20, 2004 (17) GHRSST Processing System - OCEANIDS Autonomous data stream ingest, management and distribution Built-in redundancy GUI for monitoring data flows Provides instant anomaly messaging to DAAC operators Manage over 50 data streams (50,000 files/month – 1 TB/month) OCEANIDS is enabling the DAAC to work with near real time data streams and serve customers such as COOS and IOOS.

NOAA-NASA GHRSST-PP Overview October 20, 2004 (18) GHRSST Processing System – OCEANIDS – Data Stream Interfaces User Interface GDAC (JPL U.S.) RDACs OCEANIDS (Data Management) MMR/MDB L4 (MISST – U.S.) Archive (NODC – U.S.) Near Real Time Users Delayed mode Users Medspiration (Europe) MISST (U.S.) BlueLink (Australia) PO.DAAC (U.S.) NGSST (Japan) NAVOCEANO (U.S.) Data Tools

NOAA-NASA GHRSST-PP Overview October 20, 2004 (19) GHRSST Processing System – Data Volumes

NOAA-NASA GHRSST-PP Overview October 20, 2004 (20) GHRSST Data Management Master Metadata Repository Two types of metadata records based on NASA DIF format: Data Set Record (DSD): Data Set Record (DSD): –Information common to all files (i.e., same sensor, data center etc.) –Static: Generally prepared once for each data set File Record (FR): File Record (FR): –File specific information (e.g., parameter, time and space etc.) –Dynamic: Prepared on a file-by-file basis by each Regional Data Assembly Center (RDAC) Both types integrated into a relational database: the Master Metadata Repository (MMR). Both types integrated into a relational database: the Master Metadata Repository (MMR).

NOAA-NASA GHRSST-PP Overview October 20, 2004 (21) GHRSST Data Management Master Metadata Repository Parse meta [QA check] RDAC submission GDAC submission Create/update/modify mySQL tables [SQL] [Perl scripts] Pass QA Fail QA – return to source meta Metadata queries by external user

NOAA-NASA GHRSST-PP Overview October 20, 2004 (22) GHRSST Data Management Matchup Database (MDB) Record of in situ/satellite matchups from buoys and other in situ platforms Record of in situ/satellite matchups from buoys and other in situ platforms –Source for satellite sensor error statistics Not an explicit metadata record but captures actual data (5x5 grids of matchup information) Not an explicit metadata record but captures actual data (5x5 grids of matchup information) These databases will facilitate the data discovery of multiple satellite platforms and products for the customer These databases will facilitate the data discovery of multiple satellite platforms and products for the customer

NOAA-NASA GHRSST-PP Overview October 20, 2004 (23) GHRSST Data Management Matchup Database (MDB) Matchups from Pathfinder SST

NOAA-NASA GHRSST-PP Overview October 20, 2004 (24) GHRSST Data Management User Services GDAC Customer Relationship Management (CRM) Plan GDAC Customer Relationship Management (CRM) Plan –Goals –Target Audiences –Strategies & Activities –Evaluation Mechanisms

NOAA-NASA GHRSST-PP Overview October 20, 2004 (25) GHRSST Data Management User Services Goals Goals –Leverage existing PO.DAAC CRM infrastructure to ensure a high level of customer satisfaction –Fully integrate RDACs, GDAC and NODC into the Customer Knowledge Base for 24/7 support –Implement and distribute metrics developed by the GHRSST Project Office into the GDAC reporting system –Develop and maintain web and information access portal unique to GHRSST-PP

NOAA-NASA GHRSST-PP Overview October 20, 2004 (26) GHRSST Data Management User Services Target Audiences – Customer Base Target Audiences – Customer Base –GHRSST Project –RDACs –Management Stakeholders JPL, PO.DAAC, NODC, NOAA, NASA, JPL, PO.DAAC, NODC, NOAA, NASA, –L2P Data Operational Users –L4 Public Data Customers –NODC Long Term Archive

NOAA-NASA GHRSST-PP Overview October 20, 2004 (27) GHRSST Data Management User Services Strategies and Activities Strategies and Activities –GHRSST-PP Web Portal – data and information access –24/7 Knowledge Base Customer Support –Customer access to all communication channels: , telephone, fax, web. –Measures of Effectiveness-provide statistics report to stakeholders –Privacy policies –Operational communications –Solicit customer feedback and implement process improvement

NOAA-NASA GHRSST-PP Overview October 20, 2004 (28) GHRSST Data Management User Services Provide monthly analytic reports to stakeholders on customer statistics Provide monthly analytic reports to stakeholders on customer statistics –Number of Customers accessing data via electronic transfer, subsetting tools –Characterization of customers –Number of customer interactions –Results of closed end surveys to customers accessing data Provide monthly statistics on data access, availability and transfer: Provide monthly statistics on data access, availability and transfer: –Number of successful data transfers –Number of data files served (volume?) –FTP statistics Number of Users, Number of Users, Number of Files downloaded Number of Files downloaded Data volume transferred Data volume transferred Provide data quality metrics Provide data quality metrics –Quantify number of customers providing feedback for validation studies against climatologies, in- situ data, etc. Evaluation Mechanisms – monthly and quarterly analytic reports to Stakeholders

NOAA-NASA GHRSST-PP Overview October 20, 2004 (29) GHRSST Data Management User Services Regional Data Assembly Centers/ Value Added Providers/LTA  Medspiration  RSS  NAVOCEANO  Bluelink  JMA  MODIS  MISST  Naval Research Monterey  NODC Data User Communities Discovery Data Access Building KB Queries Applications NOAA NASA GHRSST- PP JPL ESA GDAC Customer Relationship GDAC CRM Overview

NOAA-NASA GHRSST-PP Overview October 20, 2004 (30) NOAA-NASA Significance Why is all of this important? SST fundamental parameter for understanding climate change. New generation of SSTs used in data assimilation must come with full error characterizations. SST fundamental parameter for understanding climate change. New generation of SSTs used in data assimilation must come with full error characterizations. Directly related to NASA’s Earth Science Enterprise Focus themes. Includes: Directly related to NASA’s Earth Science Enterprise Focus themes. Includes:  “How can predictions of climate variability and change be improved”?  “How can weather forecast duration and reliability be improved by new space-based observations, data assimilation, and modeling?”  “The NASA Earth Science Applications Theme was created to accelerate and expand the delivery of Earth system science research results to serve decision support for American citizens and the world.”

NOAA-NASA GHRSST-PP Overview October 20, 2004 (31) NOAA-NASA Significance Why is all of this important? Will also support themes dealing with Carbon Cycle, Coastal Management, Water and Energy Cycles Will also support themes dealing with Carbon Cycle, Coastal Management, Water and Energy Cycles Will support themes dealing with improved data management, including standardization of data formats, metadata, user support, and integration of new technologies for faster data transfer. Will support themes dealing with improved data management, including standardization of data formats, metadata, user support, and integration of new technologies for faster data transfer. Also addresses all four of NOAA’s Goals: Climate, Ecosystems, Weather and Water, Commerce and Transportation Also addresses all four of NOAA’s Goals: Climate, Ecosystems, Weather and Water, Commerce and Transportation And…GHRSST is IOOS-compliant!! And…GHRSST is IOOS-compliant!!

NOAA-NASA GHRSST-PP Overview October 20, 2004 (32) New Technologies Aspera: Enables large data set transfers over any network regardless of network conditions and distance. Aspera: Enables large data set transfers over any network regardless of network conditions and distance. RightNow: Enables user and RDAC access to knowledge base system for maximizing customer communication and satisfaction RightNow: Enables user and RDAC access to knowledge base system for maximizing customer communication and satisfaction

NOAA-NASA GHRSST-PP Overview October 20, 2004 (33) Conclusion… This is a great example of NOAA and NASA working together to better meet the needs of our shared community of users!