1. GNS Science Testing by hybridization – a practical approach to testing earthquake forecasting models David Rhoades, Annemarie Christophersen & Matt Gerstenberger 9 th International Workshop on Statistical Seismology Potsdam, Germany, 15-18 June 2015

2. GNS Science Outline Existing CSEP tests –Uses –Limitations Types of hybrid models –Uses –Examples Observations from experimenting with hybrid models Suggested new role for CSEP – testing by hybridization

3. GNS Science Existing CSEP Tests of forecasting models: two main classes Consistency tests, e.g. –N-test (Number of target earthquakes) –Conditional L-test (Likelihood of target earthquakes) –S-test (Distribution of target earthquake locations) –M-test (Distribution of target earthquake magnitudes) Ranking tests, –e.g.T-test (Likelihood ratio of two models) –W-test (Non-parametric T-test) Separate testing protocol for alarm functions

4. GNS Science Consistency tests Show if models are correctly implemented but not how good they are. Low information models pass many tests –SUP model passed the N-test and L-test in retrospective testing of 5-year models at M ≥ 5 in the NZ earthquake forecast testing centre from 1984- 2009. Poisson model assumption ignores clustering –Earthquake numbers not Poisson –Some models explicitly not Poisson

5. GNS Science Ranking tests Useful when models are of similar type –e.g. two different versions of ETAS model one is elaboration of another –e.g. involves an extra fitted parameter Not so useful when models have little in common, –e.g. ETAS vs EEPAS models based on different data – e.g. geodetic vs seismic vs geological data

6. GNS Science Systematic fitting and ranking of hybrid models a practical approach to testing brings diverse information into a “best” model assimilates likelihood models, alarm functions and gridded data into new hybrid models measures information gain of new hybrids over best previously available

7. GNS Science Hybridization a way to increase the information gain of forecasts. easily carried out in the CSEP system for simple types Hybrid types Additive – Protect against “blind-spots” of individual models Maximum –Compensate for diminishing time-varying information with increasing time horizon Multiplicative –Exploit independent information on earthquake occurrence held by different models or data sets to ramp up probabilities in some cells.

8. GNS Science Additive Hybrids - Protect against “blind-spots” of individual models Easy to construct for continuous or discrete models Many existing models are additive hybrids Examples: ETAS, EEPAS etc

9. GNS Science Maximum hybrids - Compensate for diminishing time-varying information with increasing time horizon Natural way to combine a static background model with a time- varying one. Examples: 1. STEP (Gerstenberger et al., 2005) 2. Canterbury operational forecasting model (EE) (Gerstenberger et al., 2014) Easy to manage in discrete space-time. Ca Canterbury EE model forecast for central Christchurch

10. GNS Science Tests of 50-year operational forecasting model for Canterbury Model nameAcronymTypeVariants Short-term Earthquake Probabilities STEPShort-termSTEP, STEP_TV *† Epidemic-type AftershockETASShort-termETAS, ETAS_TV * Every Earthquake a Precursor According to Scale EEPASMedium-term EEPAS_0F, EEPAS_0F_TV *†, EEPAS_1F, EEPAS_1F_TV * National Seismic Hazard Model Background NSHMBGLong-term NSHMBG_B_POLY *, NSHMBG_B_1 Proximity to Past Earthquakes PPELong-term PPE *†, PPE_1950, PPE_FROM_1840 *, PPE_PRE_DARFIELD, PPE_DECLUS * Stationary Uniform PoissonSUPReferenceSUP Average-MaximumAVMAXHybridAVMAX Expert ElicitationEEHybridEE * Component of EE hybrid; † Component of AVMAX hybrid

11. GNS Science Tests of operational forecasting model for Canterbury: Tested at time lags up to 25 years on whole NZ catalogue. EE hybrid model outperformed most individual models at all time lags up to 25 years in NZ CSEP testing region.

12. GNS Science Multiplicative hybrids Exploit independent information on earthquake occurrence held by different models by ramping up probabilities in cells where high rates coincide. Rooted in theory of multiple precursors in early papers by Aki, Utsu, et al. Easy to manage in discrete space-time, so suited to CSEP model environment.

13. GNS Science Practical approaches – fitting multiplicative hybrids Weighted geometric mean of cell rates Differential probability gain from Molchan diagram. (Shebalin et al., 2012, 2014) Fit multiplier as monotone increasing function of cell rate or alarm function value (Rhoades et al., 2014)

14. GNS Science Example: RELM five-year experiment Multiple models on same spatial grid

15. GNS Science T-test comparisons of Helmstetter et al. HKJ mainshock model with other models. The number of earthquakes in the comparison is given above, and Wilcoxon signed rank test significance below, the expected value of the information gain per earthquake. If error bar does not intersect vertical zero line, information gain is statistically significant with 95% confidence. RELM tests: first order analysis – HKJ model outperforms all others in ranking tests Shen

16. GNS Science Original modelsTransformation Cell rate -> multiplier Hybrid model Total cell rate A (summed over magnitudes) Monotone increasing multiplier function (Rhoades et al., 2014)

17. GNS Science Information gain per earthquake (penalized for fitted parameters) HKJ compared to hybrid models Models for whole of CaliforniaModels for southern California NB. Models to the left of the zero line are more informative than HKJ Target earthquakes: Mainshocks + Aftershocks Monotone increasing Multiplier function

18. GNS Science Fitting period 1987-2006 M > 4.95 158 events Testing period 2007-2014 M > 4.95 171 events Multiplicative hybrids using fault and earthquake data (Rhoades et al., submitted)

19. GNS Science AcronymModel or covariate name SUPStationary Uniform Poisson HBGNational Seismic Hazard Model Background PPEProximity to Past Earthquakes PMFProximity to Mapped Faults PPIProximity to Plate Interface FLTFault in cell (binary) Left: (a) PPE and (b) HBG models for fitting period. Above: Simple hybrids with SUP as baseline and covariate (a) FLT; (b) PMF; (c) PPI, optimised for fitting period. Multiplicative hybrids based on fault and earthquake data

20. GNS Science Fitting period Testing period Multiplicative hybrids with fault and & earthquake data: High order hybrids perform well

21. GNS Science Observations Hybrid models incorporating a variety of models, data, or ideas usually outperform individual models. High order hybrids often perform well relative to low-order hybrids.

22. GNS Science Testing by hybridization Various data streams and derived products relevant to earthquake occurrence –GPS network data –Stress change modelling –Precursory anomalies /alarm functions –Heritsge forecast algorithms (M8, etc) Use to improve forecasting models on various time scales, through hybridization with existing models No need to create individual models

23. GNS Science New role for CSEP: expert system for hybrid formation and testing –Facilitate building of models based on various geophysical data and physical ideas; –Roll alarm-function forecasts into likelihood models; –Find best combined model from existing models and data. –Perform ranking tests of hybrids against each other and against submitted models; –Downplay competition and maximise collaboration; –Advance the use of CSEP as a scientific tool to improve our knowledge of earthquake predictability.