DATA MANAGEMENT UPDATE Steve Williams, Linda Cully, and Scot Loehrer NCAR/Earth Observing Laboratory (EOL) NAME SWG (7.5) Meeting 28 October 2005.

Slides:

Advertisements

Similar presentations

José Meitín NOAA Office of Global Programs Steve Williams UCAR Joint Office for Science Support José Meitín NOAA Office of.

Advertisements

NAME Enhanced Observation Period NAME Science Working Group April, 2003 NAME Homepage:

Report from the GCOS Archive/Analysis Center Matthew Menne NOAA/National Centers for Environmental Information Center for Weather and Climate (NCEI-Asheville)

Support Considerations for the Houston Environmental Aerosol Thunderstorm (HEAT) Project Richard Dirks, Jim Moore, Steve Williams UCAR/Joint Office for.

Systems Oceanography: Observing System Design. Why not hard-wire the system? Efficiency of interface management –Hard-wire when component number small,

The International Surface Pressure Databank (ISPD) and Twentieth Century Reanalysis at NCAR Thomas Cram - NCAR, Boulder, CO Gilbert Compo & Chesley McColl.

Comprehensive Large Array-data Stewardship System (CLASS) Web Site Tutorial Visit CLASS Site at

Temperature and Precipitation Data CBRFC Stakeholder Forum July 31, 2012.

Dr Mark Cresswell Model Assimilation 69EG6517 – Impacts & Models of Climate Change.

October 16-18, Research Data Set Archives Steven Worley Scientific Computing Division Data Support Section.

Field Project Planning, Operations and Data Services Jim Moore, EOL Field Project Services (FPS) Mike Daniels, EOL Computing, Data and Software (CDS) Facility.

VAMOS PROGRAMS PROJECT OFFICE C. B. Emmanuel, Director José Meitín, Deputy Director C. B.

ClimDB/HydroDB (ClimHy) Integration ClimHy has been migrated from AND to LNO and will remain status quo in 2011 – Public page (

Assessment of the NCEP data assimilation systems during the NAME04 EOP period Marco Carrera, Kingtse Mo, and Wayne Higgins CPC/NCEP/NWS/NOAA.

Plans for a Sounding Data Monitoring and Display Facility at the NAME Operations Center in Tucson Richard Johnson and Paul Ciesielski Department of Atmospheric.

Details for Today: DATE:18 th November 2004 BY:Mark Cresswell FOLLOWED BY:Literature exercise Model Assimilation 69EG3137 – Impacts & Models of Climate.

Research Data at NCAR 1 August, 2002 Steven Worley Scientific Computing Division Data Support Section.

DATA FORMATS AT EOL DATA FORMATS AT EOL Steve Williams, Chris Webster, and Dennis Flanigan EOL Computing, Data, and Software Facility Joint EOL/Unidata.

MADIS to LITTLE_R Converter MADIS observation types MADIS to LITTE_R converter Future work.

AON Data Questionnaire Results 21 Respondents Last Updated 27 March 2007 First AON PI Meeting Scot Loehrer, Jim Moore.

Workshop on QC in Derived Data Products, Las Cruces, NM, 31 January 2007 ClimDB/HydroDB Objectives Don Henshaw Improve access to long-term collections.

CC&E Best Data Management Practices, April 19, 2015 Please take the Workshop Survey 1.

Iceberg Detection Service Supports safe shipping and other offshore operations structures by providing near real- time (NRT) detection of icebergs based.

ARM Aerial Facilities BAECC Kickoff Meeting Feb Aircraft Data Local Share Drive ARM IOPshare (at Oak Ridge) ARM IOPshare (at Oak Ridge) Corrections.

Data Management Practices for Early Career Scientists: Closing Robert Cook Environmental Sciences Division Oak Ridge National Laboratory Oak Ridge, TN.

Integrated Grid workflow for mesoscale weather modeling and visualization Zhizhin, M., A. Polyakov, D. Medvedev, A. Poyda, S. Berezin Space Research Institute.

1 Mexico Regional AMDAR Workshop November 2011 Data Quality Monitoring and Control (QM / QC) Axel Hoff Convenor of WMO AMDAR Panel‘s Science and Technical.

ARCSS Data Management Support Overview and Update James Moore Steve Williams NCAR Earth Observing Laboratory 3-5 October 2007.

Operational Issues from NCDC Perspective Steve Del Greco, Brian Nelson, Dongsoo Kim NOAA/NESDIS/NCDC Dongjun Seo – NOAA/NWS/OHD 1 st Q2 Workshop Archive,

Meteorological Data Analysis Urban, Regional Modeling and Analysis Section Division of Air Resources New York State Department of Environmental Conservation.

START-08/pre-HIPPO FIELD CATALOG AND DATA MANAGEMENT Steve Williams NCAR Earth Observing Laboratory (EOL) Boulder, Colorado START-08/pre-HIPPO Planning.

RAINEX Data Management UCAR Joint Office for Science Support José Meitín Jim Moore Dick Dirks UCAR Joint Office for Science Support José Meitín Jim Moore.

NORTH AMERICAN MONSOON EXPERIMENT (NAME) An internationally coordinated, joint US-Mexico process study aimed at improving warm season precipitation prediction.

NAME : STATUS AND ISSUES NAME Homepage:

CPC Unified Precipitation Project Pingping Xie, Wei Shi, Mingyue Chen and Sid Katz NOAA’s Climate Prediction Center

MADIS Airlines for America Briefing Meteorological Assimilated Data Ingest System (MADIS) FPAW Briefing Steve Pritchett NWS Aircraft Based Observations.

IHOP_2002 DATA MANAGEMENT UPDATE Steve Williams UCAR/Joint Office for Science Support (JOSS) Boulder, Colorado 2 nd International IHOP_2002 Science Workshop.

IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein,

NAME 2004 Field Operations and Procedures NAME Special Session Field Operations Center Concept: Define requirements for, and conduct the day-to-day operations,

Legacy Data. From the 2011 EOL Science Review Team Report “The EOL effort to rescue and digitally archive historical field data that are presently stored.

Robert Grumbine National Weather Service RA-IV WIGOS Implementation Workshop (English), December, Willemstad Curaçao United.

RICO DATA MANAGEMENT STATUS UPDATE Steve Williams and Scot Loehrer UCAR/Joint Office for Science Support (JOSS) Boulder, Colorado RICO Planning Workshop.

NAME Enhanced Observation Period 5 th NAME Science Working Group Meeting November 5-7, 2003 NAME Homepage:

The Research Data Archive at NCAR: A System Designed to Handle Diverse Datasets Bob Dattore and Steven Worley National Center for Atmospheric Research.

SST and vegetation in modulating the diurnal cycle forcing of convection during the warm season Michael Douglas, NSSL Co-PI’s: Christopher Watts, Univ.

NAME Upper-Air Gridded Datasets: Description and Some Preliminary Results Paul E. Ciesielski Richard H. Johnson, Peter J.

UCAR Office of Programs University Corporation for Atmospheric Research Overview of NAME Field Catalog NAME Special Session José Meitín NAME Project Office.

Where/how could we change the overall process of field project implementation to improve in our mission of answering key science questions? Are we open.

Data Management System to Collect, Quality Control, Distribute, and Archive Near Real-time Marine Data Jeremy J. Rolph, Jacob T. Rettig, Mark A. Bourassa,

Review of Field Catalog and Data Archive Chin-Tzu Fong 馮欽賜.

Data Management Practices for Early Career Scientists: Closing Robert Cook Environmental Sciences Division Oak Ridge National Laboratory Oak Ridge, TN.

Instruments. In Situ In situ instruments measure what is occurring in their immediate proximity. E.g., a thermometer or a wind vane. Remote sensing uses.

SCD Research Data for Ocean Observatories Steering Committee June 18, 2001 Steven Worley Scientific Computing Division Data Support Section.

RICO DATA MANAGEMENT Steve Williams and Scot Loehrer UCAR/Joint Office for Science Support (JOSS) Boulder, Colorado RICO Workshop Boulder, CO February.

5-7 May 2003 SCD Exec_Retr 1 Research Data, May Archive Content New Archive Developments Archive Access and Provision.

NOAA National Climatic Data Center Dr. Karsten Shein Climatologist NOAA/NESDIS/NCDC 151 Patton Ave. Asheville, NC

A. FY12-13 GIMPAP Project Proposal Title Page version 26 October 2011 Title: WRF Cloud and Moisture Verification with GOES Status: New GOES Utilization.

START08 DATA MANAGEMENT AND ARCHIVING UPDATE Janine Goldstein, Steve Williams NCAR Earth Observing Laboratory Boulder, Colorado

Quality Control of OLYMPEX sounding data

Downloading Weather Observations

José Meitín Overview of NAME Field Catalog NAME Project Office

José Meitín Overview of JOSS Field Catalog UCAR/JOSS Boulder, Colorado

GODAE Quality Control Pilot Project

RICO DATA MANAGEMENT STATUS UPDATE

International Environmental Data Rescue Organization

NCAR’s Reanalysis Effort : Past, Present, Future

Robert Dattore and Steven Worley

Status of the Regional OSSE for Space-Based LIDAR Winds – Feb01

Comeaux and Worley, NSF/NCAR/SCD

Presentation transcript:

DATA MANAGEMENT UPDATE Steve Williams, Linda Cully, and Scot Loehrer NCAR/Earth Observing Laboratory (EOL) NAME SWG (7.5) Meeting 28 October 2005

NAME Data Management Timeline Finalize Operations Plan Complete Operations Summary (WWW) Operational Data Staged to CODIAC On-line Catalog Operational (CODIAC) Final Specs from PIs Field Ops and Data Collection QC’d data from PIs Final Data Staged Finalize Data Management Plan OPS Planning Meeting Planning Meeting Preliminary data stagedIntegrated Data Staged Draft DM Plan 2004 Data Analysis Workshop

NAME Data Flow Operational Data Ingest GTS Satellite Radar Model Special Research Data Ingest Aircraft Ships Profiler ISS NAME PI Data and Products NAME Online Field Catalog NAME Data Archive Center UCAR (JOSS) SMEX-04 Data Center (NSIDC) Distributed Data Centers CICESE IMTA NASA/DAACs NASA/GSFC NASA/JPL NASA/LaRC NOAA/ETL NOAA/NCDC NOAA/NODC NOAA/NSSL NSIDC SMN Univ. of AZ Delayed Operational Data “Final” PI Research Data Catalog Information “Preliminary” Data Sets “Preliminary” Products

EOL Data Archives metadata quality control special processing EOL Interactive Data System Data Sources Research Community Archive and Distribution

NAME DATA BASE STATUS (as of 26 October 2005) 239 of 266 Data sets (90%) have been submitted or linked. 11 Data sets are in progress Field Catalog (w/reports and “browse” products) is available on-line Precipitation (hourly and daily) and Upper Air “Composites” are under preparation Data Access:

NAME FIELD CATALOG SAMPLE PRODUCTS

NAME 2004 Field Catalog Statistics Reports/Summaries (Status, Mission, and Operations) 3,785 documents/imagery (0.27 GB) Research Platform Products (Aircraft, Surface, Upper Air, Ship) 17,606 Imagery files (1.25 GB) Operational Products (Satellite, Surface, Upper Air) 367,092 Imagery files (17.83 GB) Model Output Imagery (Analysis and Forecast Fields) 99,387 imagery files (6.76 GB) TOTALS: 487,870 Files (26.11 GB)

Composite Data Sets at NCAR/EOL A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does NCAR/EOL develop composites? - Provides data in a uniform format with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort.

“Composite” Data Sets Being Developed by NCAR/EOL for NAME Surface Meteorological – Datasets being converted Precipitation - Hourly, and Daily (June-Sept 2005) Soundings - Highest Res & 5 hPa (June – Sept 2005) Complete Station List for all Composites

NCAR/EOL Atmospheric Sounding Processing Procedures Observing System Output e.g. NWS Micro-ART Convert to JOSS Quality-control Calculate Derived Format (ASCII) Parameters Vertical Consistency Checks QC Flags Gross Limit Checks Automated ChecksQC Flags e.g., u, v, dz/dt Visual Examination Error Files High-resolution and/or Interpolated Vertical Compositee.g., 5 hPa Inventory, Archive, and Develop Metadata Data and Metadata Available Online

Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+) NAME TIER-1 EOL UPPER AIR “COMPOSITE” DATA

Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+) NAME TIER-2 EOL UPPER AIR “COMPOSITE” DATA

NAME TIER-3 EOL UPPER AIR “COMPOSITE” DATA Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+)

Belize soundings R/V Ulloa - 91 soundings R/V Altair soundings R/V El Puma - 32 tethersonde SMN soundings ISS soundings Loreto GLASS soundings NWS soundings Phoenix soundings San Jose soundings Yuma soundings pibal pibals NAME ARCHIVED SOUNDINGS BY SOURCE Total of 24,459 soundings, pibals, & tethersondes to be included in the Final “composite” data set for Tier 3*. * Need NSSL Tezopaco and Pibal data to complete composite

Enhanced sites: Number Type Number Estimate of percent of sites of sondes number missing missing 10 USA-NWS enhanced ISS, Loreto,2 ships SMN enhanced105730*2.8 1 Yuma (4 sites comb.) San Jose subtotal enhanced Operational sites: 33 USA-NWS operational SMN operational40257** Belize subtotal Total * 14 missed due to unpaid electric bill at Empalme ** 11 missed due to broken generator at Isla Socorro CSU UPPER AIR COMPOSITE DATASET (V2.0) (1 July - 15 August 2004)

Embedded Accumulative Precipitation

“Preliminary” Tier-1 Hourly Precipitation (872 stations) ALERT (+ 405) Army Range (o 40) AZMET (o 27) ISS (* 3) MADIS (+ 84) METAR (o 18) NCDC Coop (* 58) NCEP (* 59) NERN/SMN/Sonora (+ 119) NMSU (+ 59)

“Preliminary” Tier-2 Hourly Precipitation (3913 stations) ALERT (+ 414) Army Range (o 40) AZMET (o 28) ISS (* 3) MADIS (+ 891) METAR (o 327) NCDC Coop (* 736) NCEP (* 1163) NERN/SMN/Sonora (+ 182) NMSU (+ 129)

“Preliminary” Tier-3 Hourly Precipitation (10328 stations) ALERT (+ 414) Army Range (o 43) AZMET (o 28) ISS (* 3) MADIS (+ 2855) METAR (o 1089) NCDC Coop (* 2328) NCEP (* 3249) NERN/SMN/Sonora (+ 190) NMSU (+ 129)

“Preliminary” Tier-1 Daily Precipitation (2050 stations) GASIR (* 195) Hourly Stations (+ 872) NCDC Daily Coop (+ 208) NCEP (* 557) Simple Raingage (o 218)

“Preliminary” Tier-2 Daily Precipitation (11955 stations) GASIR (* 1072) Hourly Stations (+ 3913) NCDC Daily Coop (+ 2544) NCEP (* 4208) Simple Raingage (o 218)

“Preliminary” Tier-3 Daily Precipitation (30232 stations) GASIR (* 1106) Hourly Stations ( ) NCDC Daily Coop (+ 7326) NCEP (* 11254) Simple Raingage (o 218)

DATA ISSUES FOR DISCUSSION Mexican Navy SEMAR Data approval letter received; Request to collaborate with data processing. SMEX-04 Archives are being merged  Priorities? Are there any datasets not included in NAME Master List? Are there other “composites” that are needed? Should U.S. and Mexican Streamflow data be “composited”? Period of Record? Need re-organization (and cleanup) of NAME web pages. Need to establish “cutoff” date for datasets to be included in the “composites”. Others?