1 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 Hourly Precipitation Data Processing System at NCDC Stuart Hinson Meteorologist National Climatic Data Center – Asheville, NC Climate Analysis Division – Climate Applications Branch

2 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 NCDC Mission Our mission is to provide access and stewardship to the Nation's resource of global climate and weather related data and information, and assess and monitor climate variation and change. This effort requires the acquisition, quality control, processing, summarization, dissemination, and preservation of a vast array of climatological data generated by the national and international meteorological services. NCDC's mission is global in nature and provides the U.S. climate representative to the World Meteorological Organization, the World Data Center System, and other international scientific programs. NCDC also operates the World Data Center for Meteorology, Asheville.

3 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 History The observations in the Hourly Precipitation Data File were taken by observers at principle (primary) stations, secondary stations, and cooperative observer stations operated by the National Weather Service (NWS), and the Federal Aviation Agency (FAA). Approximately 5,500 stations have recorded precipitation data through the period of this digital file. Initially from August 1948 to September 1951, data were keyed on punched cards by the regional Weather Records Processing Centers. Then the task was transferred to the National Climatic Data Center (NCDC) in Asheville, NC. The hourly precipitation data file was transferred from punched cards to magnetic tape (TD-9657) during the late 1960s. This data file was then converted to the element file structure during Several recording (weighing) rain gage instruments were used in measuring hourly precipitation, but by September 1963 many Fischer- Porter precipitation gage instruments with automated readout, recorded on paper tape, were phased in. By early 1965, about 200 of these were in operation and they became the primary recording instrument. In 1996, there were approximately 2,400 Fischer-Porter gages in operation. The Universal Rain gage is the other primary instrument used to create this data file. It has an automated readout recorded on paper charts. In 1948 there were over 3,000 universal rain gage stations in the HPD system. In 1996, there were approximately 100 Universal Rain gage stations in the HPD system. Station and dates of commissioning of weighing rain gages are in the Station History listings available at the NCDC in Asheville, NC. Fischer-Porter precipitation gages record data on punched paper tapes. A device known as the Mitron reader translates the data at NCDC. The Universal Rain gage records data on paper charts. The precipitation recorded on the charts is then digitized. The data from the Surface Climate Information Archive and Dissemination System (SCIADS) or primary stations is also entered into the TD3240. The data in this file are a combination of original observations of hourly and daily accumulated precipitation. Precipitation values are checked and edited as necessary by an automated and manual edit. Data before 1984 were converted from existing digital files (TD-9657) to the element 17 structure format. These (historical) data were processed through a gross value check only. Beginning January 1984, the hourly precipitation data were processed through a completely revised system which produces the element structure database file. This system was further enhanced beginning with the January 1996 data month. The new interactive quality control system introduced many features. Data are subjected to new computer editing procedures reducing the manual handling of the data. To make the pre-1996 data consistent with the January 1996 processing system, the historical data were re-processed in The rehabilitated data covered the 1900 through 1995 period and contained more than 53 million observations. Approximately 400 thousand inconsistencies were identified and corrected as a result of this effort. These inconsistencies were categorized into 22 error types. In addition to this effort, “last look” quality assurance software was implemented on HPD data (post 1996) operationally sent to the archive each month. The “last look” software uses similar checks as the rehabilitation software and should result in maintaining consistency between the historical data and operationally received data.

4 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 The Instrumentation Fisher & Porter rain gauge – A weighing-recording rain gauge that mechanically converts the depth of accumulated precipitation during a given period of time to a code disk position. The code disk position is recorded on punched paper tape at selected intervals. The time interval of recording is 15 minutes and the accumulated precipitation is read to the nearest 2.5 mm (0.10 in.). The gauge can be modified for remote transmission. NOAA/National Weather Service, 1972: National Weather Service Observing Handbook No. 2, Substation Observations, NOAA, US GPO.

5 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 Current HPD Processing

6 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 Current HPD Processing Data tapes (B-18) are mailed to contractor in KY. On remaining MITRON punched-paper tape readers, data are transcribed into ASCII files ASCII files are loaded into existing HPD QC system, automatically and manually QC’d Data are archived into Hourly (DSI-3240) and 15-Minute (DSI-3260) data sets Code written in early 1980’s and QC is performed on a PC (originally DOS, but surprisingly still works in an XP environment)

7 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 Current HPD Processing

8 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 NWS HPD Paper Tapes SourceCorp. Mt. Vernon, KY NCDC Ingest, QC (Joyce Shaffer’s PC) NCDC HPD Publication ( IMB AIX ) NCDC HPD Archive 3260 ( IMB AIX ) NCDC HPD Archive 3240 ( IMB AIX ) Inventory/QAR SIRIS (IMC Corp.) Current HPD Data Flow ascii files arrive on ftp0

9 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 Future HPD Processing Sutron, Coastal Environmental, Rainwise, Waterlog?

10 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008

11 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 Future HPD Process NWSFO HPD Data NCDC FTP Server NCDC HPD QC (Linux) Inventory Web Page NCDC Publication NCDC Archive 3240/3260 CSSA (Metadata) MMS (Metadata)

12 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 Future HPD Process NCDC provides the highest quality data to NOAA’s customers in a timely manner Ingest –Always retain original data in raw, unedited format –Make available for future reanalysis Quality Control / Validation –Use available technology for QC –Compute “edited value” for highly suspect or known error values for the publication –Make QC algorithms available to the user Publication –Continue to produce printed copy – v –Make data available via web query – Publications on the fly Archive –DSI-3240 Hourly Precipitation Data –DSI Minute Precipitation Data

13 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 Challenges Smooth transition from punch-block to data logger Data retrieval process – COOP observer training Quality – data continuity study to develop transfer functions for data bias QC/QA - new HPD processing system – end-to-end complete Future Instrumentation changes? (Sutron/Coastal/Rainwise/Waterlog, etc) Metadata – must make sure the we keep up with the equipment

14 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 Timeline NCDC Process Design Document – Identify Programmer – Ingest Code – QC Code – Publication and Archive Code – Final test – Feedback – Implement new Process Reprocess all data through new QC –

15 Fischer-Porter Retrofit Workshop Sterling, Virginia Nov , 2008 Conclusion With the replacement of the Fischer & Porter punch-blocks with various COTS data loggers, it if an opportune time to replace the antiquated HPD QC software as well. The technology available today will lend itself to the development of a much more robust quality control software system The last existing MITRON units will be shipped to the NCDC Museum In 30 or 40 years from now, people like us will be replacing these FPR units… and probably keeping the Belfort rain gage Communications is the key...