1. DATA MANAGEMENT UPDATE Steve Williams, Linda Cully, and Scot Loehrer NCAR/Earth Observing Laboratory (EOL) NAME SWG (7.5) Meeting 28 October 2005

2. NAME Data Management Timeline 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 20032005 Finalize Operations Plan Complete Operations Summary (WWW) Operational Data Staged to CODIAC On-line Catalog Operational (CODIAC) Final Specs from PIs Field Ops and Data Collection QC’d data from PIs Final Data Staged Finalize Data Management Plan OPS Planning Meeting Planning Meeting Preliminary data stagedIntegrated Data Staged Draft DM Plan 2004 Data Analysis Workshop

3. http://www.joss.ucar.edu/name/dm

4. NAME Data Flow Operational Data Ingest GTS Satellite Radar Model Special Research Data Ingest Aircraft Ships Profiler ISS NAME PI Data and Products NAME Online Field Catalog NAME Data Archive Center UCAR (JOSS) SMEX-04 Data Center (NSIDC) Distributed Data Centers CICESE IMTA NASA/DAACs NASA/GSFC NASA/JPL NASA/LaRC NOAA/ETL NOAA/NCDC NOAA/NODC NOAA/NSSL NSIDC SMN Univ. of AZ Delayed Operational Data “Final” PI Research Data Catalog Information “Preliminary” Data Sets “Preliminary” Products

5. EOL Data Archives metadata quality control special processing EOL Interactive Data System Data Sources Research Community Archive and Distribution

7. NAME DATA BASE STATUS (as of 26 October 2005) 239 of 266 Data sets (90%) have been submitted or linked. 11 Data sets are in progress Field Catalog (w/reports and “browse” products) is available on-line Precipitation (hourly and daily) and Upper Air “Composites” are under preparation Data Access: http://www.joss.ucar.edu/ihop/archive/

8. http://www.joss.ucar.edu/name/namap2/

10. www.joss.ucar.edu/name/catalog/.

11. NAME FIELD CATALOG SAMPLE PRODUCTS

13. NAME 2004 Field Catalog Statistics Reports/Summaries (Status, Mission, and Operations) 3,785 documents/imagery (0.27 GB) Research Platform Products (Aircraft, Surface, Upper Air, Ship) 17,606 Imagery files (1.25 GB) Operational Products (Satellite, Surface, Upper Air) 367,092 Imagery files (17.83 GB) Model Output Imagery (Analysis and Forecast Fields) 99,387 imagery files (6.76 GB) TOTALS: 487,870 Files (26.11 GB) http://www.joss.ucar.edu/name/catalog/

14. Composite Data Sets at NCAR/EOL A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does NCAR/EOL develop composites? - Provides data in a uniform format with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort.

15. “Composite” Data Sets Being Developed by NCAR/EOL for NAME Surface Meteorological – Datasets being converted Precipitation - Hourly, and Daily (June-Sept 2005) Soundings - Highest Res & 5 hPa (June – Sept 2005) Complete Station List for all Composites

16. NCAR/EOL Atmospheric Sounding Processing Procedures Observing System Output e.g. NWS Micro-ART Convert to JOSS Quality-control Calculate Derived Format (ASCII) Parameters Vertical Consistency Checks QC Flags Gross Limit Checks Automated ChecksQC Flags e.g., u, v, dz/dt Visual Examination Error Files High-resolution and/or Interpolated Vertical Compositee.g., 5 hPa Inventory, Archive, and Develop Metadata Data and Metadata Available Online

17. Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+) NAME TIER-1 EOL UPPER AIR “COMPOSITE” DATA

18. Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+) NAME TIER-2 EOL UPPER AIR “COMPOSITE” DATA

19. NAME TIER-3 EOL UPPER AIR “COMPOSITE” DATA Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+)

20. Belize - 140 soundings R/V Ulloa - 91 soundings R/V Altair - 154 soundings R/V El Puma - 32 tethersonde SMN - 2468 soundings ISS - 462 soundings Loreto GLASS - 176 soundings NWS - 16079 soundings Phoenix - 360 soundings San Jose - 324 soundings Yuma - 938 soundings pibal - 3235 pibals NAME ARCHIVED SOUNDINGS BY SOURCE Total of 24,459 soundings, pibals, & tethersondes to be included in the Final “composite” data set for Tier 3*. * Need NSSL Tezopaco and Pibal data to complete composite

21. Enhanced sites: Number Type Number Estimate of percent of sites of sondes number missing missing 10 USA-NWS enhanced1380171.2 6 ISS, Loreto,2 ships815101.2 7 SMN enhanced105730*2.8 1 Yuma (4 sites comb.)37792.3 1 San Jose173126.9 25 subtotal enhanced3802782.1 Operational sites: 33 USA-NWS operational3030411.3 8 SMN operational40257**14.2 1 Belize751722.7 42 subtotal35071153.3 67 Total73091932.6 * 14 missed due to unpaid electric bill at Empalme ** 11 missed due to broken generator at Isla Socorro CSU UPPER AIR COMPOSITE DATASET (V2.0) (1 July - 15 August 2004)

22. Embedded Accumulative Precipitation

23. “Preliminary” Tier-1 Hourly Precipitation (872 stations) ALERT (+ 405) Army Range (o 40) AZMET (o 27) ISS (* 3) MADIS (+ 84) METAR (o 18) NCDC Coop (* 58) NCEP (* 59) NERN/SMN/Sonora (+ 119) NMSU (+ 59)

24. “Preliminary” Tier-2 Hourly Precipitation (3913 stations) ALERT (+ 414) Army Range (o 40) AZMET (o 28) ISS (* 3) MADIS (+ 891) METAR (o 327) NCDC Coop (* 736) NCEP (* 1163) NERN/SMN/Sonora (+ 182) NMSU (+ 129)

25. “Preliminary” Tier-3 Hourly Precipitation (10328 stations) ALERT (+ 414) Army Range (o 43) AZMET (o 28) ISS (* 3) MADIS (+ 2855) METAR (o 1089) NCDC Coop (* 2328) NCEP (* 3249) NERN/SMN/Sonora (+ 190) NMSU (+ 129)

26. “Preliminary” Tier-1 Daily Precipitation (2050 stations) GASIR (* 195) Hourly Stations (+ 872) NCDC Daily Coop (+ 208) NCEP (* 557) Simple Raingage (o 218)

27. “Preliminary” Tier-2 Daily Precipitation (11955 stations) GASIR (* 1072) Hourly Stations (+ 3913) NCDC Daily Coop (+ 2544) NCEP (* 4208) Simple Raingage (o 218)

28. “Preliminary” Tier-3 Daily Precipitation (30232 stations) GASIR (* 1106) Hourly Stations (+ 10328) NCDC Daily Coop (+ 7326) NCEP (* 11254) Simple Raingage (o 218)

29. DATA ISSUES FOR DISCUSSION Mexican Navy SEMAR Data approval letter received; Request to collaborate with data processing. SMEX-04 Archives are being merged  Priorities? Are there any datasets not included in NAME Master List? Are there other “composites” that are needed? Should U.S. and Mexican Streamflow data be “composited”? Period of Record? Need re-organization (and cleanup) of NAME web pages. Need to establish “cutoff” date for datasets to be included in the “composites”. Others?