Presentation on theme: "AMRC Meteorological Data Distribution System Part I: Data Sources and Archive K. Elena Willmot, Matthew A. Lazzara, Linda M. Keller, Nicole M. Schroeder."— Presentation transcript:
AMRC Meteorological Data Distribution System Part I: Data Sources and Archive K. Elena Willmot, Matthew A. Lazzara, Linda M. Keller, Nicole M. Schroeder Antarctic Meteorological Research Center Space Science and Engineering Center University of Wisconsin-Madison
Introduction o AMRC since 1992... o Part I Outline: o Data Sources Internal & External o Data Formats o Data Archive Historical & Present o Distribution o Challenges No more FTP? Access and quality of data Lack of acknowledgements o Statistics (Photo by Michael Forster Rothbart, UW-Madison University Communications)
Data Sources: Internal o AWS (Automatic Weather Station) data o Significant portion of internal data comes from UW’s AWS network o Real-time and archive quality data sets o Ten minute intervals o Hourly intervals o Three-hourly intervals o 60+ weather stations across the continent
Data Sources: Internal o Antarctic Composites o Conceived by UW’s Professor Charles R. Stearns in October 1992. o Cover South Pole region to about 40° South. o Satellites used: GOES, GMS, MTSAT, METEOSAT, FY-2, KALPANA, NOAA, DMSP, Aqua & Terra, and MetOp o Created every 3 hours/1 hour every day o Currently, 4 sizes: o 512 x 512 o 683 x 683 o 1024 x 1024 (standard) o 2048 x 2048 o 6 types: o Infrared o Water Vapor o Experimental Visible o Long-wave (Coming soon!) o Short-wave (Coming soon!) o Pseudo Color
Data Sources: Internal o University of Wisconsin’s Space Science and Engineering Center (SSEC) Data Center o Surface synoptic observations o METARs (Meteorological Aviation Reports) o RAOB (Radiosonde Observations) o Ship/buoy observations o Numerical Models from GFS (Global Forecast System) and ECMWF (European Centre for Medium-Range Weather Forecasts). o Polar-orbiting LAC (Local Area Coverage) observations from NOAA satellites
Data Sources: External o Participation from various researchers, USAP (United States Antarctic Program) participants, and international groups within the Antarctic meteorological community o International AWS (real-time only) o USAP research vessel observations o Field experiment radiosonde observations o Surface observations at: o Palmer Station o McMurdo Station o South Pole Station o Various other field camps Lawrence M. Gould. Photo by: Al Hickey http://photolibrary.usap.gov/
Data Flow International AWS USAP Research Vessels McMurdo, Palmer & South Pole Stations Wisconsin AWS AMRC Office SSEC Data Center NOAA HRPT DMSP RTD Aqua/Terra DB (McMurdo only) NOAA GAC/LAC MTSAT Meteosat (2) GOES (2) FY-2C Kalpana-1 Aqua/Terra NASA/NOAA Products:
Data Formats Archive: o McIDAS o AREA o MD o GRID o ASCII Text o GIF, JPEG Distribution: o McIDAS o AREA o MD o GRID o ASCII Text o netCDF o GIF, JPEG
Data Archive: Historical o Since 1992... o Media used to archive data: 1.Magneto-optic disks 2.Compact disks (CD) 3.Digital video disks (DVD) 4.8 millimeter Exabyte tapes 5.DDS3 tapes (4 mm tapes) 6.3590 IBM tapes 7.LTO tapes 1 2 3 4 5 6 7
Data Archive: Present o RAID/5 disk arrays as well as DVDs are most prominent today o FTP site most widely used o Updated monthly with o AWS – SPAWAR obs o USAP research vessel observations o McMurdo Station obs o Palmer Station obs o South Pole Station obs o Summary / notification email sent out
Distribution o Data Services to be discussed in Part II: o FTP (File Transmission Protocol) o Rsync Service o Antarctic-IDD (Internet Data Distribution) LDM (Local Data Manager) o ADDE (Abstract Data Distribution Environment) o E-mail o Website o Future: o RAMADDA (R epository for Archiving, Managing and Accessing Diverse Data) o GIS (Geographic Information Systems) Software
Challenges:1 o USAP moving away from FTP site – advancing security rules o Don’t fret! FTP will still be available here at the AMRC: o Other options: Link with external system o Limited resources to maintain links o Manually providing data is expensive and not desired o We aim to provide data in a more semi-automated, self-service fashion.
Challenges:2 o Accessing data externally o Poor quality o Corrupt files o Missing o Gaps in the archive o Which data can be used / distributed and which cannot? o Third party redistribution limitations o Evolving toward nearly fully automated data flow system (from source to archive) o Maintenance will be a constant issue – especially at the source o Documentation
Challenges:3 - Metadata o Ever increasing need for Metadata o Significant effort to research/document and provide o Incomplete historical records, leave some metadata information missing… o E.g. South Pole...
We have found an issue with the South Pole data : From 1996 and earlier, the South Pole LCDs (Local Climate Data) used data that switched between using local time and GMT time when recording observation times. South Pole is working on re-running past LCDs and we do not know how long it will take / when they will be completed.
Acknowledgements The authors appreciate the support of the Automatic Weather Station Program and/or Antarctic Meteorological Research Center for the data set, data display, information, etc. ([ AMRC/AWS staff member, as appropriate ], NSF grants number ANT-0636873, and/or ANT-0838834) Sculptor Gabriel Warren used a transparency of the B-15A Iceberg in one of his pieces called “Interrogation: Dialogue” in his “Rhegmalogia” collection. http://www.art-farm.net/homeGabe.htm
AMRC/AWS Statistics o Show Gigabytes transferred / month on Website. o Average of 70-80 Gb year-round. o Gigabytes transferred / month on FTP site. o Generally very variable – maybe due to field season.
ANTIDD – Antarctic-IDD mailing list ANTDISCUSSION – Antarctic meteorology discussion group ANTAWS - Information and updates on AWS data ANTOBS - Information and updates on all other observational datasets ANTOPS - Notices on AMRC’s real-time data availability, distribution and status ANTICEBERG - Notices on latest Iceberg monitoring images posted to the web site ANTCOMMUNITY - Infrequent notices to the Antarctic community on events and significant developments Please join our mailing lists!: How to subscribe: Visit our new website and click on the “Subscribe” button next to the mailing list you wish to join:
THANK YOU! Acknowledgements: National Science Foundation Office of Polar Programs Grant #0838834 South Pole Meteorology Office/Raytheon Polar Services SPAWAR Office of Polar Programs McMurdo Station Research Associate Palmer Station Research Associate USAP Research Vessels – LM Gould & NB Palmer McMurdo Ground Station International partners & More Questions?