The break signal in climate records: Brownian motion or Random deviations? Ralf Lindau

Dipdoc Seminar – 15. October 2018 Break signal Climate records are affected by breaks resulting from relocations or changes in the measuring techniques. For the detection, differences of neighboring stations are considered to reduce the dominating natural variance. Homogenization algorithms identify breaks by searching for the maximum external variance (explained by the jumps). Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Benchmark datasets Benchmarking data sets are used to assess the skill of homogenization algorithms. These are artificial data sets with known breaks so that an evaluation of the algorithms is possible. However, benchmark datasets should reflect as much as possible the statistical properties of real data . An important question is how to model the breaks: As free random walk (Brownian motion) As random deviation from a fixed level (random noise) Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Conceptual model Same signal, two approaches: Which of the two DT is assumed to be an independent random variable? The deviations or the jumps? Depending on our choice different statistical properties of break signal will result. Random deviations Brownian motion Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Approach To distinguish BM and RD type breaks we use to following approach. We assume that the climate time series consists of four superimposed signals: Climate, noise, BM and RD type breaks 𝑥 𝑖 =𝛾 𝑖 +𝜀 𝑖 +𝛽 𝑖 +𝛿 𝑖 , with 𝛽 ~ 𝑁 0, 𝜎 𝛽 2 , 𝛿 ~ 𝑁 0, 𝜎 𝛿 2 , 𝜀 ~ 𝑁 0, 𝜎 𝜀 2 Breaks and noise are assumed to be normal distributed. The climate signal is expected to be more complicated, but will be cancelled out in the next step. Breaks occur randomly with an average probability (say 5%). Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Spatial difference The difference between two neighboring stations x1 and x2 is: 𝑑𝑖𝑓 𝑖 = 𝑥 1 𝑖 − 𝑥 2 𝑖 = 𝛽 1 𝑖 + 𝛿 1 𝑖 + 𝜀 1 𝑖 − 𝛽 2 𝑖 + 𝛿 2 𝑖 + 𝜀 2 𝑖 The climate signal is cancelled out, because it is the same at two neighboring stations. However noise due to the different weather at the two stations remains. Dipdoc Seminar – 15. October 2018

Spatiotemporal difference D Now we have the difference time series of station pairs. Within these time series the temporal difference between two time points i and i+L is built: 𝐷= 𝛽 1 𝑖 + 𝛿 1 𝑖 + 𝜀 1 𝑖 − 𝛽 2 𝑖 + 𝛿 2 𝑖 + 𝜀 2 𝑖 − 𝛽 1 𝑖+𝐿 + 𝛿 1 𝑖+𝐿 + 𝜀 1 𝑖+𝐿 + 𝛽 2 𝑖+𝐿 + 𝛿 2 𝑖+𝐿 + 𝜀 2 𝑖+𝐿 D is the sum (or difference) of 12 random numbers. Finally, we calculate the variance of D for classes of constant time lags L: Var(D(L)) Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Variance of D 𝐷= 𝛽 1 𝑖 + 𝛿 1 𝑖 + 𝜀 1 𝑖 − 𝛽 2 𝑖 + 𝛿 2 𝑖 + 𝜀 2 𝑖 − 𝛽 1 𝑖+𝐿 + 𝛿 1 𝑖+𝐿 + 𝜀 1 𝑖+𝐿 + 𝛽 2 𝑖+𝐿 + 𝛿 2 𝑖+𝐿 + 𝜀 2 𝑖+𝐿 A common rule is: Var 𝑎±𝑏 =Var 𝑎 +Var 𝑏 ±2 Cov 𝑎,𝑏 12 variance terms. Covariance only for breaks of the same station. These occur two times (for each station): Var 𝐷 =2 Var 𝛽 𝑖 +2 Var 𝛽 𝑖+𝐿 +4 Var 𝛿 +4 Var 𝜀 −4 Cov 𝛽 𝑖 ,𝛽 𝑖+𝐿 −4 Cov 𝛿 𝑖 ,𝛿 𝑖+𝐿 Dipdoc Seminar – 15. October 2018

Covariance of RD breaks For external pairs E(Cov) = 0 For internal pairs E(Cov) = Var(d) Cov 𝛿 𝑖 ,𝛿 𝑖+𝐿 = 𝑃 𝑖𝑛𝑡 Var 𝛿 The probability to find k breaks within a time span L: 𝑓 𝑘 = 𝜆 𝑘 𝑒 −𝜆 𝑘! , with 𝜆= 𝑝 𝛿 𝐿 𝑃 𝑖𝑛𝑡 = 𝑓 0 = 𝑒 −𝜆 = 𝑒 − 𝑝 𝛿 𝐿 Cov 𝛿 𝑖 ,𝛿 𝑖+𝐿 = Var 𝛿 𝑒 − 𝑝 𝛿 𝐿 Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Variance of BM breaks A classical BM is defined as: At time step i it consists of the sum of i random numbers: Breaks do not occur each year, but only with a probability pb: Analogously for i+L: 𝛽 𝑖 = 𝑗=1 𝑖 𝑎(𝑗) , 𝑎 ~ 𝑁 0, 𝜎 𝛽 2 Var 𝛽(𝑖) 𝑐𝑙𝑎𝑠 =𝑖 𝜎 𝛽 2 Var 𝛽 𝑖 =𝑖 𝑝 𝛽 𝜎 𝛽 2 Var 𝛽(𝑖+𝐿) = 𝑖+𝐿 𝑝 𝛽 𝜎 𝛽 2 Dipdoc Seminar – 15. October 2018

Covariance of BM breaks The covariance of two time steps within a Brownian motion is equal to the variance of the earlier one, because both values have all random numbers in common that constitutes the first: Cov 𝛽 𝑖 ,𝛽 𝑖+𝐿 =𝑖 𝑝 𝛽 𝜎 𝛽 2 Var 𝛽 𝑖 =𝑖 𝑝 𝛽 𝜎 𝛽 2 Var 𝛽(𝑖+𝐿) = 𝑖+𝐿 𝑝 𝛽 𝜎 𝛽 2 Our previous findings for the variance were: Together they give: Var 𝛽(𝑖) +Var 𝛽(𝑖+𝐿) −2Cov 𝛽 𝑖 ,𝛽 𝑖+𝐿 =𝐿 𝑝 𝛽 𝜎 𝛽 2 We obtain a linear function in L. Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Variance of D We return to the original formula : Var 𝐷 =2 Var 𝛽 𝑖 +2 Var 𝛽 𝑖+𝐿 +4 Var 𝛿 +4 Var 𝜀 −4 Cov 𝛽 𝑖 ,𝛽 𝑖+𝐿 −4 Cov 𝛿 𝑖 ,𝛿 𝑖+𝐿 and inserted our findings: Var 𝐷(𝐿) = 2 𝑝 𝛽 𝜎 𝛽 2 𝐿+ 4 𝜎 𝛿 2 1− 𝑒 − 𝑝 𝛿 𝐿 + 4 𝜎 𝜀 2 The variance of D(L) has three additive components: 1. Linear function for BM type breaks 2. Exponential function for RD type breaks 3. Constant offset for the noise Dipdoc Seminar – 15. October 2018

Test with simulated data RD breaks + noise BM breaks + noise RD + BM + noise sb = 0.0 pb = 0.00 sd = 0.1 pd = 0.05 sb = 0.1 sb = 0.1 pb = 0.05 sd = 0.0 pd = 0.00 sb = 0.1 pb = 0.05 sd = 0.1 pd = 0.05 The variance follows exactly the theory when the known parameters are inserted. But how good is a retrieval without a priori knowledge? Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Retrieval approach We had: Shortly written: Two tangents, one at the beginning, one at the end: 𝐿→0 𝐿→∞ Var 𝐷(𝐿) = 2 𝑝 𝛽 𝜎 𝛽 2 𝐿+ 4 𝜎 𝛿 2 1− 𝑒 − 𝑝 𝛿 𝐿 + 4 𝜎 𝜀 2 𝑉 𝐿 =𝑏𝐿+𝑑 1− 𝑒 −𝑐𝐿 +𝑒 𝑏=2 𝑝 𝛽 𝜎 𝛽 2 𝑐= 𝑝 𝛿 𝑑=4 𝜎 𝛿 2 𝑒=4 𝜎 𝜀 2 𝑉 1 𝐿 =𝑏𝐿+𝑐𝑑𝐿+𝑒= 𝑠𝑙𝑝 1 𝐿+ 𝑐𝑜𝑛 1 𝑉 2 𝐿 =𝑏𝐿+𝑑+𝑒= 𝑠𝑙𝑝 2 𝐿+ 𝑐𝑜𝑛 2 The two tangents have four parameters. From these we can calculate the unknowns b, c, d, and e. 𝐿→0 𝐿→∞ Dipdoc Seminar – 15. October 2018

Retrieval application Two-step retrieval: 1. Two tangents as first guess 2. Exhausting search around it. Nice geometrical interpretation Dipdoc Seminar – 15. October 2018

Retrieval test for sparse data 100 station pairs: Large scatter for high lags. But the retrieval works good, the data itself varies. Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Data ISTI data restricted to US and 1900 - 2000: At least 80 years of data. Distance less than 100 km. 1459 station pairs result. Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Result At short time lags the 1 – e-x increase caused by RD type breaks is visible. For long time lags the linear increase indicates BM type breaks. The offset determines the noise. BM: pb sb2 = 0.45 K2cty-1 RD: pd = 17.1 cty-1 sd2 = 0.12 K2 Noise: se2 = 0.15 K2 Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Conclusion Brownian motion and random deviation break types can be distinguished by calculating the variance of the spatiotemporal difference. The application shows that US data contain both break types. But we did not consider: Possible trend effects Stationarity of the variance Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Lag covariance for RD The covariance is an exponential function of the time lag. C(L) = a exp (-bL) break a = sb2 strength sb b = k/(n-k) number k As byproduct we have a nice method to retrieve also strength and number of breaks directly from the data. Input: sb = 1.000 k = 5.000 Output: k = 4.984 Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 US data, not normalized The covariance reflects mainly the mean difference between two stations. Therefore, the covariance (and variance) is strongly depended on the distance. Averaging over different distance classes would be dangerous. 10.0 350 km 250 km 150 km 50 km Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 US data, normalized Normalization with the time series mean helps. The expected function of the break covariance (e-function) becomes visible. But now the variance makes weird things. Minimum at L/4. Reaching the original value at L/2, increasing further for larger L. 0.5 350 km 250 km 150 km 50 km Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Simulated data not normalised normalised The normalization causes a deformation and a shift of both the covariance and the variance function. Dipdoc Seminar – 15. October 2018

Dipdoc Seminar – 15. October 2018 Rational 𝑥 𝑎 − 𝑥 𝑥 𝑏 − 𝑥 = 𝑥 𝑎 𝑥 𝑏 − 𝑥 𝑎 𝑥 − 𝑥 𝑏 𝑥 + 𝑥 𝑥 𝑥 𝑎 𝑥 = 𝑥 𝑎 ′ + 𝑥 𝑥 = 𝑥 𝑎 ′ 𝑥 + 𝑥 𝑥 𝑥 𝑎 ′ 𝑥 =0 𝑥 𝑎 − 𝑥 𝑥 𝑏 − 𝑥 = 𝑥 𝑎 𝑥 𝑏 − 𝑥 𝑥 𝑥 𝑎 ′ 𝑥 >0 The covariance of two time points a and b is: The mixed product is: Normally we say: Then we have just the shift: However, the mixed product is not zero, but depends on the lengths of the segment. For long segments: Segments at the beginning and at the end are shorter. For L/4 middle years dominate. Dipdoc Seminar – 15. October 2018