2/18/2016 Data Management Issues Related to Drought Monitoring at Environment Canada Robert Morris Data Analysis and Archive Division Meteorological Service of Canada Toronto, Canada Presentation at the North America Drought Workshop Mexico City, October 19-20, 2006

2/18/2016 Outline Introduction Observational Networks and Data Characteristics Quality Control Forward Looking – Data Management Framework Summary

2/18/2016 Introduction The Canadian climatological archives contain temperature and precipitation observations starting in A.D. 159 stations2000 A.D stations

2/18/2016 Trends and Challenges Automatic observing technology introduces changes often faster than data management keeps up resulting in –dissimilar data sets available for analysis –loss of long-term continuity (e.g. for comparison with long-term normals) Rationalization of networks (e.g. reductions and operational control reverting to other agencies such as observing sites at airports) Accessing and incorporating data from other agencies networks (e.g. provinces, conservation authorities, electrical utilities) Increased requirement for accessing data in real-time Use of remotely sensed data (e.g. weather RADAR) and numerical weather prediction model output to augment data sparse areas and assist the quality control of in-situ data

2/18/2016 Environment Canada Daily T&P Data Synoptic data (e.g. data transmitted in WMO standard synoptic formats on the GTS) –Human observations – mainly airports –Automatic stations Aviation Surface weather Reference Climate Stations (RCS) Cooperative / volunteer climate network –Processed from paper forms –Observer enters data on an Internet screen allowing for more timely access if the observer is so inclined

Climate VolunteersContractorsAutostations Surface weatherAviation AutostationsManned National Archive System (NAS) DLY02 DLY44 DLY04 via 2304 paper form keypunch via SM via 2322 paper form keypunch PRIM04 QC RQC via SM via COOLTAP …web or IVR Environment Canada Daily Climate Data Reporting and Processing ACCESS Available within hours or daysAvailable within weeks or months SMQC A datamart with access via web services is updated twice daily with options for each station based on prioity DLY04, then DLY02, then DLY44

2/18/2016 Currently for September 2006 From Synoptic Reports 449 stations From volunteer observers (COOLTAP) 187 stations Number stations with at least 28 observations

2/18/2016 Quality Control of Climate Data Based on “hard-wired” checks –Physical and probable limits –Physical consistency amongst elements –Trends –Spatial consistency Legacy system lacks flexibility to add new elements or data streams or to incorporate new related data into the system to assist checking e.g. RADAR, satellite, NWP model output The Data Management Framework (DMF) will provide a moderized approach for the future

Distribution Station Maintenance Interface Data Access Policy/Maintenance Interface Middleware: Common Data Access Routines (Real-time, General Purpose) Replicate In Replicate out Sites Gateways Servers Modules for non Real-time QA/QC (e.g. spatial, manual) Data Acquisition Decode and Transformations Automatic (Real-time) QA/QC Alerting Web Services Data Access Policy Station metadata Short-Term Storage Long-Term Storage Original concept of the DMF component-based architecture* User Applications (e.g. Value Added) Product Generators Sites Gateways Servers Data Management Framework

QC Flag Summary Determination QC Category 3 Physical Limits (Range) QC Category 5 Temporal Consistency QC Category 4 Inter-Variable Consistency QC Category 6 Spatial Consistency QC Functions Category 1 Flag Category 3 Summary Flag Category 4 Summary Flag Category 5 Summary Flag QC Summary Flag Outputs QC Flag Basic Test Default tests Test Test … Default tests Test Test … QC Flag QC Flag Default tests Test Test … QC Flag QC Flag Default tests Test Test … QC Flag QC Flag QC Category 1 Data Element Presence Category 6 Summary Flag Interact with other data elements and QC flags QC Category n QC Flag QC Flag QC Category 2 Data Element Integrity QC Flag Default tests Test Category 2 Flag QC Category Data Element QC Flag QC Flag QC Flag QC Flag QC Message QC Message QC Message QC Message QC Message QC Message QC Message QC Message QC Message QC Message QC Message QC Message Data/Metadata Out QC in the DMF

2/18/2016 Summary Climate data for drought monitoring in Environment Canada is steadily improving, especially in the area of availability of near-real time data. Improvements in data management are planned in the areas of –Near-real time quality control –Incorporating new data elements and other agencies’ data –Use of related data (e.g. RADAR) to assist quality control and data completeness –Metadata, access systems (e.g. web services), geomatics systems

2/18/2016 Gracias por su atención Thank you