1. Norwegian Meteorological Institute met.no QC2 Status 20080220 paul.eglitis@met.no

2. Norwegian Meteorological Institute met.no QC2 QC2 Controls QC2 Functions QC1 Controlled Observations -Space Control Detect Outliers Construct Variogram … +Dip Test +Statistical Checks +Set Flags … -Interpolation Simple, Spline Kriging … +Distribute RR_24 +Dip Correction +Generate QC Products +Set Flags … QC2 Controlled Observations model values, corrected values, flags, products for HQC? prototype phase Objectives: To check viability of method(s) Construct an element of the whole QC2 system

3. Norwegian Meteorological Institute met.no RR_24 Step 1 Intepolated data calculated for all points (including missing rows as well as missing data -32767) using only RR_24 fd/c(12) = 1 as valid neighbours. Step 2 For data points where fd/c(12) = 2 and run of previous missing data or rows, redistributed values calculated based on interpolated data and original accumulated value. Step 3 Criteria for setting corrected value = redistributed value. Associated controlinfo flags to set. Responsibility for setting useinfo flags. Specification of user interfaces.

4. Norwegian Meteorological Institute met.no stationidobstimeoriginalpidtbtimetypeidslcorrectedcontrolinfouseinfocfailed 854402007-06-04 06:00:0001102007-06-0411:40:3330200000000000000020009999900000000000 854402007-06-11 06:00:003.41102007-06-1207:52:44302003.400000000000020009999900000000000 854402007-06-18 06:00:00531102007-06-1813:11:06302005300000000000020009999900000000000 854402007-06-25 06:00:0012.51102007-06-2512:41:263020012.500000000000020009999900000000000 854402007-07-02 06:00:000.21102007-07-0213:10:58302000.200000000000020009999900000000000 854402007-07-09 06:00:0001102007-07-0910:40:4230200000000000000020009999900000000000 854402007-07-23 06:00:0019.51102007-07-2310:44:443020019.500000000000020009999900000000000 854402007-08-06 06:00:00561102007-08-0612:26:37302005600000000000020009999900000000000 180302007-06-29 06:00:0016.51102007-06-2912:27:263020016.511100000000010007000000000000000 180302007-06-30 06:00:00-327671102007-09-0509:05:17302002.510000010000020063895900000000070 QC1-7- 110:1,hqc 180302007-07-01 06:00:00-327671102007-09-0509:05:1730200210000010000020063895900000000070 QC1-7- 110:1,hqc 180302007-07-02 06:00:00-327671102007-09-0509:05:173020010000010000020063895900000000070 QC1-7- 110:1,hqc 180302007-07-03 06:00:007.51102007-09-0509:05:1730200311400000000020063335900000000071 QC1-2- 72.b12:1,QC1- 7-110:1,hqchqc Missing data and Missing Rows

5. Norwegian Meteorological Institute met.no Equation Graphic: From Wikipedia, the free encyclopedia http://en.wikipedia.org/wiki/Inverse_distance_weightinghttp://en.wikipedia.org/wiki/Inverse_distance_weighting. Original map courtesy of Ole Einar Tveitto (karttegner) IDW interpolation d u k u k u k u k u k u k u k For now: p = 2 ; only neighbours d < 50 km included. Interpolation method applied

6. Norwegian Meteorological Institute met.no Example of interpolation for all Norwegian Stations November 2007

7. Norwegian Meteorological Institute met.no IDW prediction for outlier Detect and exclude outliers

8. Norwegian Meteorological Institute met.no Stationid ObstimeParamidLevel Modelid Original (GIS model value) FlagProb% 187002007-09-03 23:005000?-12.3?75 QC2 Table Notes: Initial idea to hold all model data, e.g. interpolated data, in a new table in the db Concerns large table added to kvalobs db, impacts operational system, table is derived from the original data and contains mainly redundant information, maybe subject to change when applying different algorithms, difficult to track history Alternatives: store GIS data / (QC2 derived data) in separate database archive in a scientific file format, e.g. netCDF, HDF 5, specific GIS format

9. Norwegian Meteorological Institute met.no Full Automatic Application Taken RR_24 data from operational kvalobs db 20070501 to 20071231. Method interpolates, fills out missing rows, detects runs and then redistributes (5-10 mins). stationidobstimeoriginalpidtbtimetypeidslcorrectedcontrolinfouseinfocfailed 854402007-06-11 06:00:003.41102007-06-1207:52:44302003.400000000000020009999900000000000 2007-06-12 06:00:00-32767 2007-06-13 06:00:00-32767 2007-06-14 06:00:00-32767 2007-06-15 06:00:00-32767 2007-06-16 06:00:00-32767 2007-06-17 06:00:00-32767 854402007-06-18 06:00:00531102007-06-1813:11:06302005300000000000020009999900000000000 180302007-06-29 06:00:0016.51102007-06-2912:27:263020016.511100000000010007000000000000000 180302007-06-30 06:00:00-327671102007-09-0509:05:17302002.510000010000020063895900000000070 QC1-7- 110:1,hqc 180302007-07-01 06:00:00-327671102007-09-0509:05:1730200210000010000020063895900000000070 QC1-7- 110:1,hqc 180302007-07-02 06:00:00-327671102007-09-0509:05:173020010000010000020063895900000000070 QC1-7- 110:1,hqc 180302007-07-03 06:00:007.51102007-09-0509:05:17302003 114000000000 2 006 3335900000000071 QC1-2- 72.b12:1,QC1- 7-110:1,hqchqc

10. Norwegian Meteorological Institute met.no interpolations original observations redistributed accumulated observation missing data ”data run”

11. Norwegian Meteorological Institute met.no precipcollected_flag.pl Some 2576 accumulated values (statistics in table below) fd aka Controlinfo(12) determines accumulation status Setting of c(12) relies on ”precipcollected_flag.pl” script … c(12)NumberFlag Interpretation 02651Not controlled (mainly missing data) 130937Normal RR_24 22576Accumulated 3683Times do not match 40 50 6129Corrected with model data 70 Statistics for typeid = 302

12. Norwegian Meteorological Institute met.no How much data is already corrected? Of 2576 accumulated values –1358 not corrected (++ all the missing rows) –1218 corrected Corrected data may be used to test automatic method. Propose: maintain original corrected data. Uncorrected data is candidate for replacement with redistributed accumulation.

13. Norwegian Meteorological Institute met.no Typical Example Cases STIDDATEORIGINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 1803006/29/0716.515.7216.5-10302[1110000000001000][7000000000000000] 1803006/30/07-327670.782.50.63302[1000001000002006][3895900000000070] 1803007/01/07-327672.5622.07302[1000001000002006][3895900000000070] 1803007/02/07-3276700302[1000001000002006][3895900000000070] 1803007/03/077.55.9234.79302[1140000000002006][3335900000000071]

14. Norwegian Meteorological Institute met.no Typical Example Cases STIDDATEORIGINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 1803006/29/0716.515.7216.5-10302[1110000000001000][7000000000000000] 1803006/30/07-327670.782.50.63302[1000001000002006][3895900000000070] 1803007/01/07-327672.5622.07302[1000001000002006][3895900000000070] 1803007/02/07-3276700302[1000001000002006][3895900000000070] 1803007/03/077.55.9234.79302[1140000000002006][3335900000000071] Keep corrected value as is, criteria to substitute result of automatic calculation unclear? Use such cases to test automatic method. To do: generate more exacting test data (i.e. generated from complete observations).

15. Norwegian Meteorological Institute met.no Comparison of automatic method with HQC corrections Human and machine are in concert!

16. Norwegian Meteorological Institute met.no Typical Example Cases STIDDATEORIGINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 1803006/29/0716.515.7216.5-10302[1110000000001000][7000000000000000] 1803006/30/07-327670.78-327670.63302[1000001000002006][3895900000000070] 1803007/01/07-327672.56-327672.07302[1000001000002006][3895900000000070] 1803007/02/07-327670 0302[1000001000002006][3895900000000070] 1803007/03/077.55.9234.79302[1140000000002006][3335900000000071] 18030| 2007-07-02 12:00:00 |-32767|34| 2007-09-05 09:12:31 |302 | 0 | 0 | 3 | 0000001000000005 | 4892900000000070 |hqc 18030| 2007-07-03 12:00:00 | 3|34| 2007-07-04 06:57:23 |302 | 0 | 0 | 3 | 1100000000000000 | 7000000000000000 | 18030| 2007-07-03 12:00:00 | 1|35| 2007-07-04 06:57:23 |302 | 0 | 0 | 1 | 1100000000000000 | 7000000000000000 | 18030| 2007-07-03 18:00:00 | 3|34| 2007-07-04 06:57:23 |302 | 0 | 0 | 3 | 1100000000000000 | 7000000000000000 | 18030| 2007-07-03 18:00:00 | 1|35| 2007-07-04 06:57:23 |302 | 0 | 0 | 1 | 1100000000000000 | 7000000000000000 | confidence level in original value → to be tracked when corrected set to intp or redis value (i.e. cases where corrected does not already exist). inconsistency

17. Norwegian Meteorological Institute met.no … as compared to STIDDATEDATAINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 1563006/21/070000302[1100000000002000][7300000000000000] 1563006/22/07-327671.5-327671.27302[1000003000002000][7890900000000000] 1563006/23/07-327672.11-327671.78302[1000003000002000][7890900000000000] 1563006/24/07-3276710.65-327678.99302[1000003000002000][7890900000000000] 1563006/25/07-327674.56-327673.85302[1000003000002000][7890900000000000] 1563006/26/07-327670.13-327670.11302[1000003000002000][7890900000000000] 1563006/27/07-327670 0302[1000003000002000][7890900000000000] 1563006/28/07160 0302[1100000000002000][7300000000000000] consistent

18. Norwegian Meteorological Institute met.no Measurement Accuracy STIDDATEORIGINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 1803006/29/0716.515.7216.5-10302[1110000000001000][7000000000000000] 1803006/30/07-327670.782.50.63302[1000001000002006][3895900000000070] 1803007/01/07-327672.5622.07302[1000001000002006][3895900000000070] 1803007/02/07-3276700302[1000001000002006][3895900000000070] 1803007/03/077.55.9234.79302[1140000000002006][3335900000000071] Redistributed data and previously corrected data correspond. Redistribution introduces data with unfeasible measurement accuracy Is this ok (homogenisation problems) ? Should 4.79 → 5.0 ; 0 → 0 ; 2.07 → 2 ; 0.63 → 0.5 ?

19. Norwegian Meteorological Institute met.no Another sanity check(1) STIDDATEORIGINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 1803006/29/0716.515.7216.5-10302[1110000000001000][7000000000000000] 1803006/30/07-327670.782.50.63302[1000001000002006][3895900000000070] 1803007/01/07-327672.5622.07302[1000001000002006][3895900000000070] 1803007/02/07-3276700302[1000001000002006][3895900000000070] 1803007/03/077.55.9234.79302[1140000000002006][3335900000000071]

20. Norwegian Meteorological Institute met.no Another sanity check(2)

21. Norwegian Meteorological Institute met.no … outlier … STIDDATEORIGINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 3757007/02/0726.81.8526.8-10302[1100000000001000][7000000000000000] 3757007/03/07-3276719.52-32767300.43999[0000000000000000][9999900000000000] 3757007/04/07-3276715.43-32767237.48999[0000000000000000][9999900000000000] 3757007/05/07-3276715.87-32767244.23999[0000000000000000][9999900000000000] 3757007/06/07-327677.48-32767115.17999[0000000000000000][9999900000000000] 3757007/07/07-327671.44-3276722.23999[0000000000000000][9999900000000000] 3757007/08/07-327671.22-3276718.79999[0000000000000000][9999900000000000] 3757007/09/07-327674.72-3276772.59999[0000000000000000][9999900000000000] 3757007/10/07 1075 4.16-3276664.09302[1600002000002000][7338100000000001] => Use interpolated data directly for the Corrected Value

22. Norwegian Meteorological Institute met.no Typical Example Cases, contd. STIDDATEDATAINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 1803006/29/0716.515.7216.5-10302[1110000000001000][7000000000000000] 1803006/30/07-327670.782.50.63302[1000001000002006][3895900000000070] 1803007/01/07-327672.5622.07302[1000001000002006][3895900000000070] 1803007/02/07-3276700302[1000001000002006][3895900000000070] 1803007/03/077.55.9234.79302[1140000000002006][3335900000000071]

23. Norwegian Meteorological Institute met.no Data not previously corrected STIDDATEDATAINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 1803011/23/0715.913.8615.9-10302[1110000000001000][7000000000000000] 1803011/24/07-327670.01-327670.02302[1000003000003000][7899900000000000] 1803011/25/07-327676.18-327679.62302[1000003000003000][7899900000000000] 1803011/26/07100.23100.36302[1140000000002000][7330900000000001]

24. Norwegian Meteorological Institute met.no precipcollected_flag exceeding itself since 25/9/2007 STIDDATEDATAINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 2136006/08/070 -10402[1110000000000000][7000000000000000] 2136006/09/07-327670 0402[0000003000002000][9999900000000000] 2136006/10/07-327670 0402[0000003000002000][9999900000000000] 2136006/11/07-327670 0402[0000003000002000][9999900000000000] 2136006/12/07-327670 0402[0000003000002000][9999900000000000] 2136006/13/07-327670 0402[0000003000002000][9999900000000000] 2136006/14/07-327671.06-327671.41402[0000003000002000][9999900000000000] 2136006/15/073.51.583.52.09402[1140000000002000][7320400000000001] ?

25. Norwegian Meteorological Institute met.no Missing rows Before: STIDDATEDATAINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 8544005/29/0772.40 -10302[0000000000002000][9999900000000000] 8544006/04/0700.0100302[0000000000002000][9999900000000000] 8544006/11/073.42.283.4 302[0000000000002000][9999900000000000] 8544006/18/07537.953 302[0000000000002000][9999900000000000] 8544006/25/0712.50.0712.5 302[0000000000002000][9999900000000000] 8544007/02/070.20.040.2 302[0000000000002000][9999900000000000] 8544007/09/0700.0100302[0000000000002000][9999900000000000] 8544007/23/0719.54.5119.5 302[0000000000002000][9999900000000000] 8544008/06/075611.7856 302[0000000000002000][9999900000000000] 8544008/20/073214.9732 302[0000000000002000][9999900000000000]

26. Norwegian Meteorological Institute met.no Missing rows After: STIDDATEDATAINTPCORRREDISTYPEIDCONTROLINFOUSEINFO 8544006/12/07-3276713.81-3276718.16999[0000000000000000][9999900000000000] 8544006/13/07-327670 0999[0000000000000000][9999900000000000] 8544006/14/07-327674.74-327676.24999[0000000000000000][9999900000000000] 8544006/15/07-327676.78-327678.91999[0000000000000000][9999900000000000] 8544006/16/07-327674.61-327676.06999[0000000000000000][9999900000000000] 8544006/17/07-327672.3-327673.02999[0000000000000000][9999900000000000] 8544006/18/07538.075310.61302[0000000000002000][9999900000000000] 8544006/19/07-327670.9-327673.06999[0000000000000000][9999900000000000] 8544006/20/07-327671.19-327674.04999[0000000000000000][9999900000000000] 8544006/21/07-327671.07-327673.64999[0000000000000000][9999900000000000] 8544006/22/07-327670.01-327670.03999[0000000000000000][9999900000000000] 8544006/23/07-327670.44-327671.48999[0000000000000000][9999900000000000] 8544006/24/07-327670 0.01999[0000000000000000][9999900000000000] 8544006/25/0712.50.0712.50.23302[0000000000002000][9999900000000000] Typeid, ControlInfo, etc., (all data row!) …, Useinfo to set for this case …

27. Norwegian Meteorological Institute met.no Discussions points …preliminary decisions/actions added (1) Specification of interpolation algorithm?Action: Paul, Matthias and Ole Einar to meet and discuss. (2) Criteria for setting: corrected value = redistributed value ( inter value) ? Associated controlinfo flags to set? Responsibility for setting useinfo flags? According to QC1 flags. Standard deviation of neighbours. If no flag set from point (6). Action: Lars will review examples included in this talk and advise on new flag settings. (3) Redistribution introduces data with unfeasible measurement accuracy. Any consequences? Round data to one decimal place, but keep sum equal to the accumulated value. (4) Storage of derived QC2 information?Store in external data files, i.e. netCDF as a first case. Include estimate of variability in the measurements / uniformity of data … (5) Scope of control, e.g. typeid 302, 402 …Run for both 302 and 402 … flagging will be the same in both cases. (6) Handle localised weather … comparison with satellite and radar data? To provide an indicator of how uniform the rainfall distribution is. First task to build in estimates of the variability from the space control, use of normal values, gradients of ratios etc. (7) Last six months we have precip_flag working well,and can use c(12)=2 criterion … what about older data? Priority is current data. Eventually process historic data too.

28. Norwegian Meteorological Institute met.no User Interface? Prototype code currently implemented in C++ with algorithms built in for performance. Either run process as per schedule and/or on demand. –Set of values that can be configured by a user/scheduler? Time Interval, Rules for Flags to set, i.e. utilise a scripting language or configuration file to set the controls. Run by operator who reviews results then clicks to submit change. Use of QC1 Perl Algorithm concept? Run algorithm on arbitrary set of 3D data? What are the boundaries? Priotity to develop an operational version with only essential user controls, a rich interface can be built on top later.