1. AMRC Meteorological Data Distribution System Part I: Data Sources and Archive K. Elena Willmot, Matthew A. Lazzara, Linda M. Keller, Nicole M. Schroeder Antarctic Meteorological Research Center Space Science and Engineering Center University of Wisconsin-Madison

2. Introduction o AMRC since 1992... o Part I Outline: o Data Sources Internal & External o Data Formats o Data Archive Historical & Present o Distribution o Challenges No more FTP? Access and quality of data Lack of acknowledgements o Statistics (Photo by Michael Forster Rothbart, UW-Madison University Communications)

3. Data Sources: Internal o AWS (Automatic Weather Station) data o Significant portion of internal data comes from UW’s AWS network o Real-time and archive quality data sets o Ten minute intervals o Hourly intervals o Three-hourly intervals o 60+ weather stations across the continent

4. Data Sources: Internal o Antarctic Composites o Conceived by UW’s Professor Charles R. Stearns in October 1992. o Cover South Pole region to about 40° South. o Satellites used: GOES, GMS, MTSAT, METEOSAT, FY-2, KALPANA, NOAA, DMSP, Aqua & Terra, and MetOp o Created every 3 hours/1 hour every day o Currently, 4 sizes: o 512 x 512 o 683 x 683 o 1024 x 1024 (standard) o 2048 x 2048 o 6 types: o Infrared o Water Vapor o Experimental Visible o Long-wave (Coming soon!) o Short-wave (Coming soon!) o Pseudo Color 

5. Making Antarctic Composite Satellite Imagery:

6. Data Sources: Internal o University of Wisconsin’s Space Science and Engineering Center (SSEC) Data Center o Surface synoptic observations o METARs (Meteorological Aviation Reports) o RAOB (Radiosonde Observations) o Ship/buoy observations  o Numerical Models from GFS (Global Forecast System) and ECMWF (European Centre for Medium-Range Weather Forecasts). o Polar-orbiting LAC (Local Area Coverage) observations from NOAA satellites

7. Data Sources: External o Participation from various researchers, USAP (United States Antarctic Program) participants, and international groups within the Antarctic meteorological community o International AWS (real-time only) o USAP research vessel observations o Field experiment radiosonde observations o Surface observations at: o Palmer Station o McMurdo Station o South Pole Station o Various other field camps Lawrence M. Gould. Photo by: Al Hickey http://photolibrary.usap.gov/

8. Data Flow International AWS USAP Research Vessels McMurdo, Palmer & South Pole Stations Wisconsin AWS AMRC Office SSEC Data Center NOAA HRPT DMSP RTD Aqua/Terra DB (McMurdo only) NOAA GAC/LAC MTSAT Meteosat (2) GOES (2) FY-2C Kalpana-1 Aqua/Terra NASA/NOAA Products:

9. Data Formats Archive: o McIDAS o AREA o MD o GRID o ASCII Text o GIF, JPEG Distribution: o McIDAS o AREA o MD o GRID o ASCII Text o netCDF o GIF, JPEG

10. Data Archive: Historical o Since 1992... o Media used to archive data: 1.Magneto-optic disks 2.Compact disks (CD) 3.Digital video disks (DVD) 4.8 millimeter Exabyte tapes 5.DDS3 tapes (4 mm tapes) 6.3590 IBM tapes 7.LTO tapes 1 2 3 4 5 6 7

11. Data Archive: Present o RAID/5 disk arrays as well as DVDs are most prominent today o FTP site most widely used o Updated monthly with o AWS – SPAWAR obs o USAP research vessel observations o McMurdo Station obs o Palmer Station obs o South Pole Station obs o Summary / notification email sent out

12. Distribution o Data Services to be discussed in Part II: o FTP (File Transmission Protocol) o Rsync Service o Antarctic-IDD (Internet Data Distribution) LDM (Local Data Manager) o ADDE (Abstract Data Distribution Environment) o E-mail o Website o Future: o RAMADDA (R epository for Archiving, Managing and Accessing Diverse Data) o GIS (Geographic Information Systems) Software

13. Challenges:1 o USAP moving away from FTP site – advancing security rules o Don’t fret! FTP will still be available here at the AMRC: o Other options: Link with external system o Limited resources to maintain links o Manually providing data is expensive and not desired o We aim to provide data in a more semi-automated, self-service fashion.

14. Challenges:2 o Accessing data externally o Poor quality o Corrupt files o Missing o Gaps in the archive o Which data can be used / distributed and which cannot? o Third party redistribution limitations o Evolving toward nearly fully automated data flow system (from source to archive) o Maintenance will be a constant issue – especially at the source o Documentation

15. Challenges:3 - Metadata o Ever increasing need for Metadata o Significant effort to research/document and provide o Incomplete historical records, leave some metadata information missing… o E.g. South Pole...

16.  We have found an issue with the South Pole data :  From 1996 and earlier, the South Pole LCDs (Local Climate Data) used data that switched between using local time and GMT time when recording observation times.  South Pole is working on re-running past LCDs and we do not know how long it will take / when they will be completed.

17. Acknowledgements  The authors appreciate the support of the Automatic Weather Station Program and/or Antarctic Meteorological Research Center for the data set, data display, information, etc. ([ AMRC/AWS staff member, as appropriate ], NSF grants number ANT-0636873, and/or ANT-0838834) Sculptor Gabriel Warren used a transparency of the B-15A Iceberg in one of his pieces called “Interrogation: Dialogue” in his “Rhegmalogia” collection. http://www.art-farm.net/homeGabe.htm

18. AMRC/AWS Statistics o Show Gigabytes transferred / month on Website. o Average of 70-80 Gb year-round. o Gigabytes transferred / month on FTP site. o Generally very variable – maybe due to field season.

19. ANTIDD – Antarctic-IDD mailing list ANTDISCUSSION – Antarctic meteorology discussion group ANTAWS - Information and updates on AWS data ANTOBS - Information and updates on all other observational datasets ANTOPS - Notices on AMRC’s real-time data availability, distribution and status ANTICEBERG - Notices on latest Iceberg monitoring images posted to the web site ANTCOMMUNITY - Infrequent notices to the Antarctic community on events and significant developments Please join our mailing lists!: How to subscribe: Visit our new website and click on the “Subscribe” button next to the mailing list you wish to join:

20. THANK YOU! Acknowledgements: National Science Foundation Office of Polar Programs Grant #0838834 South Pole Meteorology Office/Raytheon Polar Services SPAWAR Office of Polar Programs McMurdo Station Research Associate Palmer Station Research Associate USAP Research Vessels – LM Gould & NB Palmer McMurdo Ground Station International partners & More Questions?