RAINEX Data Management UCAR Joint Office for Science Support José Meitín Jim Moore Dick Dirks UCAR Joint Office for Science Support José Meitín Jim Moore.

Slides:

Advertisements

Similar presentations

Southeast Climate Consortium Extension Program C. W. Fraisse, J. Bellow, N. Breuer, V. Cabrera, J. W. Jones, K. Ingram, and G. Hoogenboom.

Advertisements

José Meitín NOAA Office of Global Programs Steve Williams UCAR Joint Office for Science Support José Meitín NOAA Office of.

GOES Direct Services. Topics  Direct Broadcast Services  GOES Data Collection Services  Search and Rescue.

Support Considerations for the Houston Environmental Aerosol Thunderstorm (HEAT) Project Richard Dirks, Jim Moore, Steve Williams UCAR/Joint Office for.

SEA-COOS The Southeast Atlantic Coastal Ocean Observing System National IOOS Regional observing systems.

The Carolinas Coastal Ocean Observing and Prediction System (Caro-COOPS) Funded by the National Oceanic And Atmospheric Administration A partnership among.

GCOOS Data Portal GCOOS Board of Directors Meeting Corpus Christi, TX 19 August 2008 GCOOS Board of Directors Meeting Corpus Christi, TX 19 August 2008.

Integrated Ocean Observing System Data Management and Communications March 2004 The US Integrated Ocean Observing System (IOOS) Plan for Data Management.

Progress Towards a Regional Coastal Ocean Observing System for the Southeast (SEACOOS) Harvey Seim / University of North Carolina at Chapel Hill University.

RAINEX Real time Communications Overview NCAR, NOAA, Rest of the World Operational Products Satellite Imagery Model Products Surface Observations RECON.

Science Mission Directorate National Aeronautics and Space Administration transitioning unique NASA data and research technologies The SPoRT Center – Infusing.

October 16-18, Research Data Set Archives Steven Worley Scientific Computing Division Data Support Section.

Field Project Planning, Operations and Data Services Jim Moore, EOL Field Project Services (FPS) Mike Daniels, EOL Computing, Data and Software (CDS) Facility.

Coordinated Energy and water-cycle Observations Peroject A Well Organized Data Archive System Data Integrating/Archiving Center at University of Tokyo.

Research Data at NCAR 1 August, 2002 Steven Worley Scientific Computing Division Data Support Section.

DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.

NOAA PORTS ® Partnerships MARACOOS Annual Meeting 12/15/2011 Darren Wright Maritime Services Program Manager.

MADIS to LITTLE_R Converter MADIS observation types MADIS to LITTE_R converter Future work.

AON Data Questionnaire Results 21 Respondents Last Updated 27 March 2007 First AON PI Meeting Scot Loehrer, Jim Moore.

Real-Time Coastal Ocean Data Products for Tampa Bay, West Florida, and the Southeast US Mark E. Luther College of Marine Science University of South Florida.

MADIS – The Meteorological Assimilation Data Ingest System Current Operational Status and Future Plans Greg Pratt, Leon Benjamin, Thomas Kent, Gopa Padmanabhan,

JCOMM Data Buoy Cooperation Panel October 16, 2006 National Data Buoy Center 2006 Review: A Year of Growth Paul F. Moersdorf, PhD, Director.

April nd IBTrACS Workshop 1 Operational Procedures How can we build consistent, homogeneous, well- documented climate quality data?

Scientific Investigations; Support from Research Data Archives for Joint Office for Science Support 26 February, 2002 Steven Worley SCD/DSS.

IODE Ocean Data Portal - technological framework of new IODE system Dr. Sergey Belov, et al. Partnership Centre for the IODE Ocean Data Portal MINCyT,

Joint WMO-IOC Technical Commission for Oceanography and Marine Meteorology Contributions to WIGOS David Meldrum, vice chair, JCOMM OPA.

Technical Working Group, II Teruko Manabe Steven Worley Miroslaw Mietus Shawn Smith Simon Tett Volker Wagner Scott Woodruff David Berry Liz Kent.

Page 1 Pacific THORPEX Predictability, 6-7 June 2005© Crown copyright 2005 The THORPEX Interactive Grand Global Ensemble David Richardson Met Office, Exeter.

1 Global Systems Division (GSD) Earth System Research Laboratory (ESRL) NextGen Weather Data Cube Chris MacDermaid October, 2010.

Data Access to Marine Surface Observations and Products from COADS 29 January, 2002 Steven Worley National Center for Atmospheric Research.

1 NOAA IOOS Program Data Integration Framework (DIF) Project Overview Adapted from a brief to the NOAA Data Management Committee August 6, 2008 by Jeff.

The Arctic Observing Network (AON) Cooperative Arctic Data and Information Service (CADIS) Florence Fetterer,

Automated Weather Observations from Ships and Buoys: A Future Resource for Climatologists Shawn R. Smith Center for Ocean-Atmospheric Prediction Studies.

ARCSS Data Management Support Overview and Update James Moore Steve Williams NCAR Earth Observing Laboratory 3-5 October 2007.

Topics describe the Data Buoy Cooperation Panel (DBCP) –aims, –achievements and –Challenges network status developments of data buoy technology JCOMM.

Data Acquisition & Data Management Ronald Muzzi Electronics Engineer GLERL’s Marine Instrumentation Lab.

START-08/pre-HIPPO FIELD CATALOG AND DATA MANAGEMENT Steve Williams NCAR Earth Observing Laboratory (EOL) Boulder, Colorado START-08/pre-HIPPO Planning.

2015 GLM Annual Science Team Meeting: Cal/Val Tools Developers Forum 9-11 September, 2015 DATA MANAGEMENT For GLM Cal/Val Activities Helen Conover Information.

2013 NOAA MADIS Plans MADIS – The Meteorological Assimilation Data Ingest System Greg Pratt, Leon Benjamin, Thomas Kent, Gopa Padmanabhan, and Leigh Cheatwood.

Division of Nearshore Research Texas Coastal Ocean Observation Network Dr. Gary Jeffress Mr. James Rizzo October 2003.

MADIS Airlines for America Briefing Meteorological Assimilated Data Ingest System (MADIS) FPAW Briefing Steve Pritchett NWS Aircraft Based Observations.

IHOP_2002 DATA MANAGEMENT UPDATE Steve Williams UCAR/Joint Office for Science Support (JOSS) Boulder, Colorado 2 nd International IHOP_2002 Science Workshop.

November 16, 2009 Page 1 of 28 Data and Data Management: Introduction to the BCO-DMO Presented to Professor Keiichi Uchida November 16, 2009 Robert C.

IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,

An Introduction to the Argo Data Sytem South Pacific Workshop 11 – 14 October 2005 Mark Ignaszewski FNMOC.

Robert Grumbine National Weather Service RA-IV WIGOS Implementation Workshop (English), December, Willemstad Curaçao United.

Tools for Coordinating Aircraft During Hurricane Field Campaigns: Real Time Mission Monitor and Waypoint Planning Tool Richard Blakeslee / NASA Marshall.

RICO DATA MANAGEMENT STATUS UPDATE Steve Williams and Scot Loehrer UCAR/Joint Office for Science Support (JOSS) Boulder, Colorado RICO Planning Workshop.

DATA MANAGEMENT UPDATE Steve Williams, Linda Cully, and Scot Loehrer NCAR/Earth Observing Laboratory (EOL) NAME SWG (7.5) Meeting 28 October 2005.

CO-OPS Expands Meteorological Sensor Network and Quality Control Kathleen Egan, Tom Landon NOAA/NOS/Center for Operational Oceanographic Products and Services.

End-to-End Data Services A Few Personal Thoughts Unidata Staff Meeting 2 September 2009.

3-D rendering of jet stream with temperature on Earth’s surface ESIP Air Domain Overview The Air Domain encompasses a variety of topic areas, but its focus.

Access to IOOS Data Relevant to OOI Kathleen Bailey NOAA/NOS/IOOS January 6, 2016.

UCAR Office of Programs University Corporation for Atmospheric Research Overview of NAME Field Catalog NAME Special Session José Meitín NAME Project Office.

Where/how could we change the overall process of field project implementation to improve in our mission of answering key science questions? Are we open.

Data Management System to Collect, Quality Control, Distribute, and Archive Near Real-time Marine Data Jeremy J. Rolph, Jacob T. Rettig, Mark A. Bourassa,

SCD Research Data for Ocean Observatories Steering Committee June 18, 2001 Steven Worley Scientific Computing Division Data Support Section.

RICO DATA MANAGEMENT Steve Williams and Scot Loehrer UCAR/Joint Office for Science Support (JOSS) Boulder, Colorado RICO Workshop Boulder, CO February.

OFCM CEISC December 12, 2005 Non-Traditional and Non-WMO Observational Networks: Transitioning of the Meteorological Assimilation Data Ingest System (MADIS)

5-7 May 2003 SCD Exec_Retr 1 Research Data, May Archive Content New Archive Developments Archive Access and Provision.

National Data Buoy Center Ian Sears National Data Buoy Center An Overview of Quality Control Procedures for Buoy Data.

1. 2 NOAA’s Mission To describe and predict changes in the Earth’s environment. To conserve and manage the Nation’s coastal and marine resources to ensure.

Ocean Monitoring and Forecasting A Commercial Perspective Dr Ralph Rayner Ocean Numerics/Fugro GEOS.

IODE Ocean Data Portal - technological framework of new IODE system Dr. Sergey Belov, et al. Partnership Centre for the IODE Ocean Data Portal.

START08 DATA MANAGEMENT AND ARCHIVING UPDATE Janine Goldstein, Steve Williams NCAR Earth Observing Laboratory Boulder, Colorado

Global Precipitation Data Access, Value-added Services and Scientific Exploration Tools at NASA GES DISC Zhong Liu1,4, D. Ostrenga1,2, G. Leptoukh4, S.

José Meitín Overview of NAME Field Catalog NAME Project Office

José Meitín Overview of JOSS Field Catalog UCAR/JOSS Boulder, Colorado

Texas Coastal Ocean Observation Network

Presentation transcript:

RAINEX Data Management UCAR Joint Office for Science Support José Meitín Jim Moore Dick Dirks UCAR Joint Office for Science Support José Meitín Jim Moore Dick Dirks RAINEX Planning Meeting, Miami, FL

Early involvement in project planning Involvement with PIs to develop data management strategy Consistent implementation of strategy for lifetime of project and beyond Reliable and efficient archive and distribution system Easy and efficient access to datasets by broader community Early involvement in project planning Involvement with PIs to develop data management strategy Consistent implementation of strategy for lifetime of project and beyond Reliable and efficient archive and distribution system Easy and efficient access to datasets by broader community JOSS Data Management RAINEX Planning Meeting, Miami, FL UCAR Office Of Programs Joint Office for Science Support

Data Management Considerations Develop a consistent Data Policy Consider Data Types Formats and Documentation (metadata) Data Collection strategies (operations support, supplementary data) Real-time Requirements ( data calibration, intercomparison, and needs for operational decision making) Data Quality Control and Processing responsibilities Data Archive structure, functions and responsibilities Data Distribution protocols (user interface, metadata inventories, browse products) Collaboration with other programs and data centers (interoperability) Develop a consistent Data Policy Consider Data Types Formats and Documentation (metadata) Data Collection strategies (operations support, supplementary data) Real-time Requirements ( data calibration, intercomparison, and needs for operational decision making) Data Quality Control and Processing responsibilities Data Archive structure, functions and responsibilities Data Distribution protocols (user interface, metadata inventories, browse products) Collaboration with other programs and data centers (interoperability) RAINEX Planning Meeting, Miami, FL

Data Questionnaire Data Management Plan Project Web Pages Real-time Operations and Data Ingest Field Data Catalog Quality control and dataset development Interactive Data Archive and Distribution Special media products (e.g. CD-Roms) Data Questionnaire Data Management Plan Project Web Pages Real-time Operations and Data Ingest Field Data Catalog Quality control and dataset development Interactive Data Archive and Distribution Special media products (e.g. CD-Roms) JOSS Data Services RAINEX Planning Meeting, Miami, FL UCAR Office Of Programs Joint Office for Science Support

JOSS Data Services Data Questionnaire JOSS Data Services Data Questionnaire UCAR Office Of Programs Joint Office for Science Support RAINEX Planning Meeting, Miami, FL

Project Web Pages JOSS Data Services RAINEX Planning Meeting, Miami, FL

Real-time Operations Support and Data Ingest Satellite Ground Station Forecast Products Numerical Model Analyses Operational Data Streams Real-time Operations Support and Data Ingest Satellite Ground Station Forecast Products Numerical Model Analyses Operational Data Streams JOSS Data Services RAINEX Planning Meeting, Miami, FL UCAR Office Of Programs Joint Office for Science Support

RAINEX Planning Meeting, Miami, FL GTS METAR Observation Locations 6-10 November 2004 GTS METAR Observation Locations 6-10 November 2004

RAINEX Planning Meeting, Miami, FL GTS Ship and Buoy Observation Locations 6-10 November 2004 GTS Ship and Buoy Observation Locations 6-10 November 2004

RAINEX Planning Meeting, Miami, FL Coastal and Marine Observation Locations *NDBC BuoyoTCOON (TX) *CMAN*TABS (TX) oNWLON+LUMCON (LA) oPORTSoWAVCIS (LA) +COMPS (FL)

Land Observation Locations *ASOSoLAIS (LA) *AWOS+TX ET (TX) *AWOS modemoFAWN (FL) oGAEMN (GA)+TNRCC (TX) +GFC (GA)oALMESO (AL) RAINEX Planning Meeting, Miami, FL

JOSS Data Services Field Data Catalog ( WWW based ) Daily Mission Reports Operations Summary Status Reports Data Analysis Products Authoring Tools JOSS Data Services Field Data Catalog ( WWW based ) Daily Mission Reports Operations Summary Status Reports Data Analysis Products Authoring Tools UCAR Office Of Programs Joint Office for Science Support RAINEX Planning Meeting, Miami, FL

JOSS Data Services Quality control and dataset development Processing of research-quality, high resolution data Processing of standard operational data streams Compatible formats Composite datasets and data integration JOSS Data Services Quality control and dataset development Processing of research-quality, high resolution data Processing of standard operational data streams Compatible formats Composite datasets and data integration UCAR Office Of Programs Joint Office for Science Support RAINEX Planning Meeting, Miami, FL

Composite Data Sets at UCAR/JOSS A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does UCAR/JOSS develop composites? - Provides data in a uniform format with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort. A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does UCAR/JOSS develop composites? - Provides data in a uniform format with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort. RAINEX Planning Meeting, Miami, FL

ASOS AWOS ARM Others (7) OKMESO ABLE SCAN LTER HPCN WTXMESO NCAR (4) MADIS Ameriflux KVII GWMD NMSU CVT HRLY Visual MERGE Gross Limit Checks Horizontal QC Visual Hourly Surface Composite IHOP_2002 Hourly Surface Composite Development Examine Statistics CVT RAINEX Planning Meeting, Miami, FL

NCAR / RAF NetCDF NOAA / AOC “standard” format JOSS convert “type 5” to NetCDF JOSS RAINEX Archive Integrated RAINEX NetCDF Aircraft Flight Level Datasets

JOSS DATA MANAGEMENT TOOLS Field Catalog In-field tool to ingest and display operational and preliminary research data and project documentation for making real-time decisions and evaluating project progress Features: Daily Mission Reports Operations Summary Facility Status Reports Data Analysis Products Authoring Tools Web-based access Field Catalog In-field tool to ingest and display operational and preliminary research data and project documentation for making real-time decisions and evaluating project progress Features: Daily Mission Reports Operations Summary Facility Status Reports Data Analysis Products Authoring Tools Web-based access Interactive Data Archive/Access System Primary means for all project scientists and researchers to browse and retrieve data from any JOSS-supported projects Features: Long-term field project data archival and distribution Interactive data browsing, subsetting, and format conversion Remote Archive Access Value-added datasets Data documentation Interactive Data Archive/Access System Primary means for all project scientists and researchers to browse and retrieve data from any JOSS-supported projects Features: Long-term field project data archival and distribution Interactive data browsing, subsetting, and format conversion Remote Archive Access Value-added datasets Data documentation RAINEX Planning Meeting, Miami, FL

UCAR Office Of Programs Joint Office for Science Support JOSS Data Services Interactive Data Archive and Distribution Ease of Data Access (web based) Data available via FTP, magnetic tape, CD-Rom Metadata descriptions Data Browse, Subsetting, Format Translation Capabilities Quality Control Distributed Data Archives Customized data ordering and reformat JOSS Data Services Interactive Data Archive and Distribution Ease of Data Access (web based) Data available via FTP, magnetic tape, CD-Rom Metadata descriptions Data Browse, Subsetting, Format Translation Capabilities Quality Control Distributed Data Archives Customized data ordering and reformat RAINEX Planning Meeting, Miami, FL

RAINEX Data Management Issues: Data Policy Data Management Plan Real Time Data Needs (model, satellite) Distributed Archive Responsibilities Special Data Processing (surface, upper-air, airborne radar aircraft flight-level) RAINEX Data Management Issues: Data Policy Data Management Plan Real Time Data Needs (model, satellite) Distributed Archive Responsibilities Special Data Processing (surface, upper-air, airborne radar aircraft flight-level)

RAINEX Planning Meeting, Miami, FL UCAR Office Of Programs Joint Office for Science Support

PORTS – Physical Oceanographic Real-Time System NWLON – National Water Level Observation Network TCOON – Texas Coastal Ocean Observation Network (TAMU – Corpus) TABS – Texas Automated Buoy System (TAMU) LUMCON – Louisiana Universities Marine Consortium WAVCIS – Wave Current Surge Information System (LSU) COMPS – Coastal Ocean Monitoring and Prediction System (USF) GAEMN – Georgia Automated Environmental Monitoring Network (U of Georgia) GFC – Georgia Forestry Commission ALMESO – Alabama Mesonet (Auburn U) LAIS – Louisiana Agriclimatic Information System (LSU) TX ET – Texas Evapotranspiration Network (TAMU) FAWN – Florida Automated Weather Network (U of Florida) TNRCC – Texas Natural Resources Conservation Commission Acronyms