COL: data acquisition, storage and preservation Roger Brugge Climatological Observers Link Department of Meteorology University of Reading Transmission, presentation and archiving of meteorological data, 5 Oct 2011

Outline What is COL? –What the data are used for Data gathering –Daily versus monthly data –Old and new methods Data quality – station gradings Data storage –What is stored? –How is it stored? Data preservation –For use by COL –For use by others

What is COL? June An advert in Weather in June 1950 by Tom Suttie –enquiring about setting up an organisation for the exchange of weather data by amateurs A chance sighting of this advert by Eric Bonsor –there followed an exchange of letters, and January first edition of the Directory of Amateur Observers published –Site details, instrumentation and observing routines June first issue of the Bulletin –48 stations reporting, describing the weather of May –Nowadays more than 330 sites each month Aim of COL - to publish the monthly bulletin at the earliest opportunity –Achieved for over 41 years now –The COL Bulletin is now a rare source of comprehensive UK monthly weather summaries. –Membership details added to the Directory of Amateur Observers –Almost 1300 stations are now included, together with photographs of stations

Data gathering Monthly data only –A few exceptions for a daily data table –COL cannot handle the volume of daily data that would otherwise accrue Paper –The original method (15% of returns) –Also used as a reminding mechanism to most members (15%) Web (70% of returns) –Underpinned by a database –PHP5 MyAdmin and SQL >archiving –Provides members with an early snapshot –The way of the future?

Paper form – main data features only shown

Data gathering Monthly data only –A few exceptions for a daily data table –The Bulletin Paper –The original method (15% of returns) –Also used as a reminding mechanism to most members (15%) Web (70% of returns) –Underpinned by a database –PHP5 MyAdmin and SQL > archiving –Provides members with an early snapshot –The way of the future?

reply form – the start of a message for August 2011

Data gathering Monthly data only –A few exceptions for a daily data table –The Bulletin Paper –The original method (15% of returns) –Also used as a reminding mechanism to most members (15%) Web (70% of returns) –Underpinned by a database –PHP5 MyAdmin and SQL > archiving –Provides members with an early snapshot –The way of the future?

Data quality COL welcomes observations from everyone –Multiple stations per town –Wide variety of instrumentation and exposure –Airports, universities, institutions and the hobbyist Instrumentation –Originally based around Stevenson screen, mercury-in-glass thermometers and Snowdon raingauges –Many good quality sites –Increasing use being made of AWS equipment Varying costs, and also performance Need a way to distinguish between stations that (even in the same locality) might report widely differing obs –> COL station grading system (Stephen Burt)

Buxton, Norfolk Stratfield Mortimer Elderslie

Tivington Red Wharf Bay, Anglesey Compton Martin

Data quality COL welcomes observations from everyone –Multiple stations per town –Wide variety of instrumentation and exposure –Airports, universities, institutions and the hobbyist Instrumentation –Originally based around Stevenson screen, mercury-in-glass thermometers and Snowdon raingauges –Increasing use being made of AWS equipment Varying costs, and also performance Need a way to distinguish between stations that (even in the same locality) might report widely differing obs –> COL station grading system (Stephen Burt)

Station grading system (as used in 2011) Grading system development over the years has been led by Stephen Burt

Incl. SYNOP and CLIMAT sites

Station grade + photograph + station details = a reasonable idea of data quality

Data storage What is stored? Monthly summaries –A subset of the data; coverage 1940s onwards Column-aligned text files Easily manipulated by Excel and Fortran, etc. Used to make quick calculations and climatological values –e.g. recently published averages All monthly data, numbers and text –Database; coverage 2006 onwards Generated by SQL/webpage Little-used All back issues of the bulletin available electronically –40 th anniversary CD

Members observations – where do/could they go? Lots of high-quality COL observations (especially grade A stations) Effectively unused or un-archived simply because they are not reporting to UKMO Data from these sites are likely to be lost unless special arrangements are made. Currently paper copies –traditionally, sometimes saved in local history libraries donated to an observer living close by sent to the local rubbish tip (the majority) –now: they can be deposited with the Chiltern Observatory Trust Electronic data –Monthly – some/all retained by COL –Daily – delete key –Either way – file formats soon become outdated A proper remote archiving/backup facility is preferable. Might BADC have a role here? Two suggested solutions…

Preserving members observations – Possible solution (1) 2011: Many amateur sites are fully/partial AWS Might consumer-AWS instrument manufacturers agree a common archive file export standard with BADC? –to simplify the archiving process –ensure amateur data is not lost when a disk crashes or the observer dies. If this was a file that could be exported then uploaded to a BADC site, say annually, it would take only minutes. A mandatory metadata section should be required. Might Davis… –who are the market leaders –who currently provide an export file format for NOAA …take a lead on this?

Preserving members observations – Possible solution (2) BADC to propose a standard Excel spreadsheet-type template for amateur observers to use. –This would greatly simplify the transmission and storage of amateur observations to BADC. –A mandatory metadata section should be required, perhaps including site plan/s and photographs from cardinal points. –Perhaps pilot using the existing COL archive template materials (hourly, daily, monthly templates have been created) and with a voluntary participation of a handful of A-grade COL stations in 2012? Issues: –There are as many observer-owned spreadsheet formats as observers –However, given a lead by BADC, COL members might be persuaded to modify or convert their formats

More information This presentation – –To be placed at Website – Weather article –Brugge, R., Forty years of the Climatological Observers Link. Weather, 65, the author Bulletin – some copies available today Processed data: an example –Burt, S.D. and R. Brugge, Climatological Averages for and for stations appearing in the monthly bulletin of the Climatological Observers Link. ISBN