N ational C limatic D ata C enter Development of the Global Historical Climatology Network Sea Level Pressure Data Set (Version 2) David Wuertz, Physical.

Slides:



Advertisements
Similar presentations
World Meteorological OrganizationIntergovernmental Oceanographic Commission of UNESCO Ship Observations Team ~ integrating and coordinating international.
Advertisements

JMA 1 GCOS Surface Network Monitoring Centre Climate Prediction Division HIROSHI NAKAMIGAWA.
CTS130 Spreadsheet Lesson 20 Data Consolidation. Consolidation is a process in which data from multiple worksheets or workbooks is combined and summarized.
Konstanz, Jens Gerken ZuiScat An Overview of data quality problems and data cleaning solution approaches Data Cleaning Seminarvortrag: Digital.
The Integrated Surface Hourly (ISH) Global Database ESDIM/OGP funded, FCC effort 2 QC phases thus far Full POR online via FTP, partial POR via CDO ISH.
Library Staff Training
Lecture 6: Multiple Regression
A Procedure for Automated Quality Control and Homogenization of historical daily temperature and precipitation data (APACH). Part 1: Quality Control of.
Formula Auditing, Data Validation, and Complex Problem Solving
TNKids Duplicate Social Security Number. The following graphics are designed to help you to navigate through this Computer Based Training. The navigational.
ROSI Express Report Training: Scheduled Courses with Instructor/Coordinator Diagnostics.
India Emerging Markets Conference, May 2009 (1) Leigh Walton Animal Improvement Programs Laboratory Agricultural Research Service, USDA Beltsville,
Training Course 2 User Module Training Course 3 Data Administration Module Session 1 Orientation Session 2 User Interface Session 3 Database Administration.
MEDARE Workshop, Tarragona, Spain Rescue and Digitization of Climate Records of Cyprus 28 – 30 November 2007 Stelios Pashiardis Meteorological Service.
3.2 Data Checking.
The New & Improved. APW Worksheet & Database To obtain a blank excel and/or access file of the APW, you can download them from our website at:
United Nations Economic Commission for Europe Statistical Division Seasonal Adjustment Process with Demetra+ Anu Peltola Economic Statistics Section, UNECE.
Data entry: Validation
Climate Monitoring of Precipitation: The GPCC - Status and plans Global Precipitation Climatology Centre U. Schneider, A. Meyer-Christoffer, B. Rudolf.
Table of Contents (click on an error to jump to that slide)
Event Data History David Adams BNL Atlas Software Week December 2001.
Page 1© Crown copyright Report of the Global Collecting Centres Elanor Gowland, GCC UK SOT-IV, 16 th - 21 st April 2007.
The consistency Checker, or Overhauling a PGDB By Ron Caspi.
National Climate Monitoring Products Andrew Watkins and John Kennedy (updated 28/4/2014)
Quality control of daily data on example of Central European series of air temperature, relative humidity and precipitation P. Štěpánek (1), P. Zahradníček.
Cooperative Research Programs (CoRP) Satellite Climate Studies Branch (SCSB) 1 1 Reconstruction of Near-Global Precipitation Variations Thomas Smith 1.
Initiative overview 30 November 2011 Jay Lawrimore Chief, Ingest and Analysis Branch, NCDC.
WFM 6311: Climate Risk Management © Dr. Akm Saiful Islam WFM 6311: Climate Change Risk Management Akm Saiful Islam Lecture-7:Extereme Climate Indicators.
Statistical Expertise for Sound Decision Making Quality Assurance for Census Data Processing Jean-Michel Durr 28/1/20111Fourth meeting of the TCG - Lubjana.
6 th Annual Focus Users’ Conference 6 th Annual Focus Users’ Conference Import Testing Data Presented by: Adrian Ruiz Presented by: Adrian Ruiz.
U.S. Department of the Interior U.S. Geological Survey Processing ArcHydro Datasets with NHDPlus Version 2, Emphasizing StreamStats Data Development Webinar.
Genesys Shell development Input-side development progress.
Dictionary based interchanges for iSURF -An Interoperability Service Utility for Collaborative Supply Chain Planning across Multiple Domains David Webber.
NWS Calibration Workshop, LMRFC March, 2009 slide - 1 Analysis of Temperature Basic Calibration Workshop March 10-13, 2009 LMRFC.
Archived Data Management System Study Advisory Committee Meeting May 14, 2003.
XP New Perspectives on Microsoft Access 2002 Tutorial 31 Microsoft Access 2002 Tutorial 3 – Querying a Database.
IGCSE ICT Stock Control.
The world leader in serving science Overview of Thermo 21 CFR Part 11 tools Overview of software used by multiple business units within the Spectroscopy.
12 CVS Mauro Jaskelioff (originally by Gail Hopkins)
DTC Quantitative Methods Summary of some SPSS commands Weeks 1 & 2, January 2012.
1 00/XXXX © Crown copyright Hadley Centre MSLP Data Base Rob Allan Hadley Centre, Met Office, UK AOPC Surface Pressure Group Workshop, UEA, Norwich November.
10 th Argo data management 2009 Toulouse What is new at GDACs ?
Solutions Summit 2014 Comparison Codes, Their Families and Their Effects Terri Sullivan.
11 Chapter 111 Sequential File Merging, Matching, and Updating Programming Logic and Design, Second Edition, Comprehensive 11.
Public Libraries Survey Data File Overview. What We’ll Talk About PLS: Public Libraries Survey State level data Public library data (Administrative Entities)
GLOSS Training Workshop Course Japan Meteorological Agency May 15-26, 2006 Sea Level Data Processing with SLPR2 1. Introduction.
Automated Operational Validation of Meteorological Observations in the Netherlands Wiel Wauben, KNMI, The Netherlands Introduction QA/QC chain Measurement.
NOAA National Climatic Data Center Dr. Karsten Shein Climatologist NOAA/NESDIS/NCDC 151 Patton Ave. Asheville, NC
CTD Data Processing Current BIO Procedure. Current Processing Software Matlab Migrating to R & Python Code Version Control SVN Migrating to GitHub.
Quality Control of Soil Moisture and Temperature For US Climate Reference Network Basic Methodology February 2009 William Collins USCRN.
1 Handbook on Population and Housing Census Editing Department of Economic and Social Development United Nations Statistics Division Studies in Methods,
L1Calo Databases ● Overview ● Trigger Configuration DB ● L1Calo OKS Database ● L1Calo COOL Database ● ACE Murrough Landon 16 June 2008.
Jay Lawrimore, Matt Menne
U.S.-India Partnership for Climate Resilience
Downloading Weather Observations
Data and Information.
Tutorial 3 – Querying a Database
MOS Developed by and Run at the NWS Meteorological Development Lab (MDL) Full range of products available at:
Programming Logic and Design Fourth Edition, Comprehensive
Post Processing.
Guidance for managing international precipitation data

This presentation document has been prepared by Vault Intelligence Limited (“Vault") and is intended for off line demonstration, presentation and educational.
Final Design Authorization
NATS 101 Lecture 3 Climate and Weather
Precip, Tmax, Tmin scaled to match CRU monthly data.
NATS 101 Lecture 3 Climate and Weather
Presentation transcript:

N ational C limatic D ata C enter Development of the Global Historical Climatology Network Sea Level Pressure Data Set (Version 2) David Wuertz, Physical Scientist Climate Analysis Branch N ational C limatic D ata C enter

Why Version 2 and why now? 10 years since Version 1 was updated Version 1 was not subjected to rigorous quality control Wish to validate models and other data sets Desire to pursue other research questions

N ational C limatic D ata C enter Data Sources for GHCN SLP Electronically available sources only World Weather Records World Monthly Surface Station Climatology Australian Bureau of Meteorology Monthly Climatic Data for the World (includes CLIMAT messages via GTS)

N ational C limatic D ata C enter Process Overview Merge individual data sources Eliminate Duplicates Resolve remaining metadata issues Perform quality assurance checks Not homogeneity adjusted (yet)

N ational C limatic D ata C enter Merge individual data sources Compare station data and metadata Some stations combined (“mingled”) –Required exact match in period of overlap –Required excellent match in metadata Some stations added as new –Created new station when could not combine –Close matches considered in duplicate elimination process

N ational C limatic D ata C enter Eliminate Duplicate Stations Part automated, part manual Defining duplicate (“sameness”): –Floating tolerance – Values are “same” if 0.1 mb if both have 0.1 resolution 0.5 mb if A has 0.1 and B has 1.0 resolution 1.0 mb if both have 1.0 resolution –Compute difference statistics: Percent of overlap “same” Number of runs of same values, longest run Max diff, 90 th, 75 th, 50 th, 25 th, 10 th percentiles, Min diff

N ational C limatic D ata C enter Eliminate Duplicate Stations (Cont’d) Reorder according to sameness Examine statistics and metadata Decide if duplicates –Most get “mingled” –Some remain marked as duplicates (e.g., cases where only 70% same) Examine stations having similar names Examine stations having same location Check for transitivity violations –If A = B, and B = C, but A  C! –Manually inspect and resolve

N ational C limatic D ata C enter Resolve Remaining Metadata Issues Assign correct country codes –Match with stations in other databases (GHCN Precip, WMO Vol A) –Plot locations on high resolution map Assign unique station numbers –Use WMO numbers wherever possible –For others use nearest WMO + unique modifier

N ational C limatic D ata C enter Quality Assurance Checks Suspect values saved in separate file Manual inspection via plotting –Examine each time series –Examine difference series with neighbors –Look for mislocated or otherwise problematic stations (184 identified and removed) Reasonable range check –Values outside range mb (97 values involving 82 stations)

N ational C limatic D ata C enter Quality Assurance Checks (Cont’d) Gross errors using digital checks –Different years having largely the same data (5 stations involved) –Runs of identical consecutive values (71 runs involving 60 stations) –Runs of same value for a fixed month across all years (748 cases involving 459 stations)

N ational C limatic D ata C enter Quality Assurance Checks (Cont’d) Checks for statistically wild outliers –z scores based upon biweight mean and std dev –z scores > 5 flagged (298 points) –3.5 < z scores < 5 flagged … if neighbor’s z score < 3.5 (456 points) –Percent of data set flagged = 0.08%

N ational C limatic D ata C enter Data Set Summary Map of station locations Period of record information Comparison of GHCN and Hadley holdings List of files for GHCN SLP v2 How to obtain GHCN SLP v2 Future SLP work

GHCN Pressure Stations (Nstations = 2668)

N ational C limatic D ata C enter

GHCN SLP Files Filename Contents readme.slp.v2Format descriptions v2.slp.ZMain data file v2.slp.inv.Z"Inventory" file containing station metadata v2.slp.country.codesCountry code/name cross reference v2.slp.failed.qc.ZValues edited from main file by QC process

N ational C limatic D ata C enter Obtaining GHCN SLP Files ftp ftp.ncdc.noaa.gov ftp> cd /pub/data/ghcn/v2 ftp> prompt ftp> mget *slp* ftp> bye

N ational C limatic D ata C enter What next? Compare with HadSLP, NCEP Reanalysis Contribute to bigger and better AOPC Multinational SLP data set Suggestions are welcome