1. IHOP-2002 Data Archive and Development of Composite Data Sets Steven F. Williams, Scot M. Loehrer, Linda E. Cully, Darren R. Gallant, Janine Goldstein, and Don Stott UCAR/Joint Office for Science Support IHOP-2002 Spring Science Workshop Boulder, CO March 2003

2. http://www.joss.ucar.edu/ihop/dm/

3. IHOP-2002 Data Policy and Protocol

5. http://www.joss.ucar.edu/ihop/dm/photo.html

6. IHOP-2002 DATA BASE STATUS (as of 24 March 2003) 196 of 254 Data sets (77%) have been submitted or linked. 183 are available on-line (13 in progress) Field Catalog (w/reports and preliminary products) is on-line Data submission instructions and guidelines are available on-line For details see: http://www.joss.ucar.edu/ihop/dm/

7. Composites: What and Why A composite dataset is a collection (over some time period and region) of similar data (e.g. surface met) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does UCAR/JOSS develop composites? - Provides data in a uniform format with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort.

8. Composite Data Sets To Be Developed by UCAR/JOSS for IHOP-2002 Surface Meteorological - 1 min, 5 min, and Hourly Precipitation - 15 min, Hourly, and Daily Soundings - High Resolution and 5 hPa

9. 1-min Surface Meteorological Data Composite (65 stations) ASOS (41) ARM SMOS (14) ABLE AWS (5) NCAR Supp. (5)

10. Oklahoma Mesonet (115) NCAR ISS/ISFF (10) West Texas Mesonet (34) 1-min sites (65) 5-min Surface Meteorological Data Composite (224 stations)

11. Hourly Surface Meteorological Data Composite (743 stations) CoAgMet (*) (23) AWOS (*) (110) RAWS (+) (10) SCAN (o) (6) GWMD (*) (18) Ameriflux (+) (3) HPCN (o) (72) Texas ET (o) (19) KVII-TV (+) (4) MADIS (o) (246) NMSU (*) (7) 5-min sites (o) (225)

12. 15-minute Precipitation Composite NCDC Coop (429) 5-min surface (224)

13. Hourly Precipitation Composite NCDC Coop (509) NCEP hourly (1155) Hourly Met (743)

14. Daily Precipitation Composite NCDC Coop (1581) Hourly Met (743) NCEP Daily (1611)

15. GENERAL DATASET PROCESSING LIFE CYCLE Contact SRC Agency to Acquire Data and Metadata Wait for SRC Agency Data/Metadata Preparation Examine Data & Metadata for Consistency and Completeness Acquire Data and Metadata OR Create “Best” Metadata Develop New Software or Modify Existing Software Acquire Computer System Space and Time Process Data and Metadata then Check Form Composite with Like Datasets Perform QC and Final Data Checks. Generate Statistics Create EBUFR format data & DOCS for CODIAC Load Data and Metadata onto CODIAC ACTION Prepare ASCii format data & docs for CD-Roms Prepare, manufacture, & distribute CD-Roms Add stn info to final list Incomplete? Processing Problems? 1 2 z:\cully\Esop96\esop3.ppt Incomplete? 5 6 7 8 9 3 4 Time Phone, email, paperwork, $, meetings, procurement arrangements. (Days to months) (Minutes to months) Auto retrieval, FTP, online requests, etc. (minutes to days) Verify AOI, TOI, Format; spot check values, visual inspection or limited processing (Minutes to days) Format change, new parameters manual “cleanup”, etc. (Days to months) Purchase hardware, configuration management, etc. (Minutes to days) Put into common format, run data checkers “check gross limits” (Days to months) HQC for SFC, Gross limits for precip, Composite level and value checkers, limited statistics (% G, B, D by network, stn, parameter for SFC). (days to months) days/months weeks/months months/year

16. QC of Surface Meteorological Data Parameters: Calculated Sea Level Pressure, Sea Level Pressure, Temperature, Dew Point, Wind Speed, Wind Direction Expected value derived from objective analysis method adapted from Cressman (1959) and Barnes (1964) using 300km Radius of Influence. Σθ i w i Σw i i i θ e = Expected Value θ i = Observation at site i w i = Weight factor for site i where w i = 1/d i d i = Distance between site i and QC site θ e =

17. QC of Surface Meteorological Data Deviation between actual and expected values subjected to dynamically determined limits. Values compared at like solar times. Limits are sensitive to diurnal and intra-seasonal variations and dependent upon spatial and temporal continuity. Additional consistency checks, visual quality assurance, and gross limit checks are typically also applied. Precipitation gross limit checked based on frequency. Data are never changed, only flagged. α = Normalization Factor (predetermined) σ = Standard Deviation (computed over previous 30 days for each time; 15 obs minimum) θ a - θ e  ασ

18. What Can Lead to Surface Data Being Flagged? Instrument failure, bias, drift, random errors Incorrect latitude, longitude, or elevation Differences in siting or height of instruments Misidentified units, dates, or stations Terrain impacts Data problems at surrounding stations Unique meteorological phenomena

19. Obvious Problems Detected

20. Subtle Problems Detected

21. Embedded Accumulative Precipitation

22. High Resolution and 5 hPa Radiosonde Composites Fixed Sites: Mobile Sites: NWS (14) ARM (5) ISS (1) Falcon Drops (86) Learjet (330) NCAR GLASS (146) NSSL CLASS (80)

23. UCAR/JOSS Atmospheric Sounding Processing Procedures Observing System Output e.g. NWS Micro-ART Convert to JOSS Quality-control Calculate Derived Format (ASCII) Parameters Vertical Consistency Checks QC Flags Gross Limit Checks Automated ChecksQC Flags e.g., u, v, dz/dt Visual Examination Error Files High-resolution and/or Interpolated Vertical Compositee.g., 5 hPa Inventory, Archive, and Develop Metadata Data and Metadata Available Online

24. Wind Oscillations Wind oscillations at low elevation angles (approaching limiting angles) due in part to questionable angles. Original Algorithm Adjusted Algorithm

25. Summary The High Resolution and 5 hPa Radiosonde Composites have been completed and are available: http://www.joss.ucar.edu/ihop/dm Surface Meteorological (1-min, 5-min, and hourly) and Precipitation (15- min, hourly, and daily) Composites currently being developed. Planned completion Fall 2003.