Chart-o-Mat ICG-WIGOS TT on Metadata Management of Station Metadata at Deutscher Wetterdienst (DWD) Rainer März Section Administration of Observing Networks rainer.maerz@dwd.de

Principles for Metadata at DWD Metadata is stored together with observing data in one database Project ‚MIRAKEL‘ started around 1995 for CDC + GPCC All conditions are valid for a specific time-range (from – until) Integration of Metadata from other organisations Flexible Import/Export for future needs Semi-automatic update and quality checking procedures All applications should benefit from the data (and use it) Geneva, March 11, 2013

Amount of station Metadata Die Mess- und Beobachtungsnetze des DWD OF, November 2005 Amount of station Metadata Overview of the contents [count of largest categories in thousands]: WMO Main Stations (GSN, RBSN, RBCN ….) 13.000 Airports (with ICAO loc.ind.) reporting METAR/SPECI 5.000 German additional stations (voluntary, federal states) 12.000 Related partners (roads, renewable energy ….) 2.000 Voluntary observing ships and other marine vessels 9.000 Additional GPCC stations of all countries 75.000 Currently counting over all: 117.000 Geneva, March 11, 2013 Deutscher Wetterdienst - Abteilung TI 2 3

Storage of Metadata / groups of tables Die Mess- und Beobachtungsnetze des DWD OF, November 2005 Storage of Metadata / groups of tables Tables for metadata divided into : Basic Information meta tables; allowed values Joined Information intersection tables with validity dates (begin / end) Tables for observation data use the same unique station ID ! Staff Station Pub9VolA Equipment Measuring condidions Geneva, March 11, 2013 Deutscher Wetterdienst - Abteilung TI 2 4

Storage of Metadata / classes of information Die Mess- und Beobachtungsnetze des DWD OF, November 2005 Storage of Metadata / classes of information Main aspects of meta data storage: Main station table (name, start_date, end_date, status) IDs (WMO / ICAO / IATA / GPCC / Nat.Numbers …) Geographical (Lat./Lon., elevation, country, region, area etc.) Equipment Staff Operation and Production (programmes for encoding reports) Visits (Maintenance, Administrative) Representation (measuring conditions) Geneva, March 11, 2013 Deutscher Wetterdienst - Abteilung TI 2 5

Managing the Metadata Access and permits managed by database-roles Die Mess- und Beobachtungsnetze des DWD OF, November 2005 Managing the Metadata Access and permits managed by database-roles 4 accounts with full access - ‚Admin‘ 10+ accounts to maintain general station changes (add, move, close) Approx. 100 accounts allow a ‚restricted write-access‘ Approx. 1000 accounts have only read-permits (all users) Personalized data like bank accounts with restricted roles Geneva, March 11, 2013 Deutscher Wetterdienst - Abteilung TI 2 6

Updating and completing the Metadata Die Mess- und Beobachtungsnetze des DWD OF, November 2005 Updating and completing the Metadata Weekly import of Pub9VolA of WMO and semi-automatic changes Regional Network Groups report national changes to Offenbach and edit their local entries (persons, sensors, visits, history) Import of bilateral exchanged data with individual loading-procedures comparision with other sources when necessary (flagging of suspicious/unknown/inconsistent) GPCC maintains meta-data for non-WMO stations of other NMCs Geneva, March 11, 2013 Deutscher Wetterdienst - Abteilung TI 2 7

Typical trouble with metadata Die Mess- und Beobachtungsnetze des DWD OF, November 2005 Typical trouble with metadata No information about changes at a station Inconsistency between dependent sources (i.e. elevation…) Lack of syncronization with Offline-procedures Especially Lat./Lon. different various formats (dez.,°,min/sec) 4 sources / 5 different values ;-) …. Which is ‚the best‘ Representation of a complicated history for a time-series standard exchange format + automated process is necessary! Geneva, March 11, 2013 Deutscher Wetterdienst - Abteilung TI 2 8

Stations operated by Deutscher Wetterdienst Die Mess- und Beobachtungsnetze des DWD OF, November 2005 Stations operated by Deutscher Wetterdienst 4 Regional Oberserving Network Groups Hamburg, Potsdam, Offenbach, Munich 65 Stations with professional Observers, 35 occupied 00-24 UTC, 30 only daytime 48 measuring radioactivity (air+precipitation) 28 RBSN Stations 11 Climate Reference (traditional equipment) 4 GSN-Stations (GCOS) 1 GUAN Station 1 GAW Station 114 Automated Weather Stations with full equipment 17 Weather Radar Stations 9 Aerological Stations (4 autolaunchers, 1 ozone) 4 Wind Profilers +34 Stations of Bundeswehr Geoinfo Service Showing all stations that are published in WMO Pub.9 Vol.A Geneva, March 11, 2013 Deutscher Wetterdienst - Abteilung TI 2 9

Additional Stations operated by DWD and Partners Die Mess- und Beobachtungsnetze des DWD OF, November 2005 Additional Stations operated by DWD and Partners 1781 Voluntary Stations (climate; wind; precip), 1365 reporting online (24/1 reports per day) Ship-based: 740 Voluntary Observing Ships (VOS) 19 Ship (AWS) (24 reports/day) Last but not least …: 1292 Phenological Stations (observing plants) 1500 Partner stations (approx) (motorways, fed.states, wind-energy, (nuclear) power-plants, university, military) Historical (closed) stations are not included Geneva, March 11, 2013 Deutscher Wetterdienst - Abteilung TI 2 10

Example All 1018 GSN stations in WMO GCOS Die Mess- und Beobachtungsnetze des DWD OF, November 2005 Example All 1018 GSN stations in WMO GCOS Basic Meta Data and availability of FM12/FM94 observations received in Offenbach Tool is available for operations and can switch between global, regional and national view Retrieving live data as received and decoded from GTS, joined with meta-data from (names, locations, countries ….. ) Geneva, March 11, 2013 Deutscher Wetterdienst - Abteilung TI 2 11

There is still a lot of research necessary! Example Precip-Station and the surrounding presented in a tool of GPCC called ‚Mirakel-Live‘ There is still a lot of research necessary! Geneva, March 11, 2013

Die Mess- und Beobachtungsnetze des DWD OF, November 2005 Outlook WCDMP / Data rescue needs support (DARE @ CDC) national projects currently in progress: -> KliDaDigi for land stations -> HISTOR for ship logbooks Lots of undigitalized metadata from all German Stations +Electronical archives of BUFR since 1970s may be loaded Work for many years! Geneva, March 11, 2013 Deutscher Wetterdienst - Abteilung TI 2 13

QUESTIONS AND COMMENTS ? Thank you for your attention ! Geneva, March 11, 2013