1. Designing Ensembles for Climate Prediction
Peter Challenor
National Oceanography Centre

2. Why Ensembles for Climate Prediction?
Not just a point estimate
Uncertainty estimates as well
Calibration of models against data
Sensitivity analysis

4. Overview What is experimental design? Why should we be interested?
Perturbed physics ensembles
Space filling designs
Some recent results
Multimodel ensembles
Conclusions

5. What is experimental design?
Developed from agricultural experiments in the 1920’s
How should you apply treatments to experimental plots in a field?

6. R.A. Fisher Greatest Statistician of the 20th Century Randomisation
Block designs
Latin squares
Split plots …

7. Clinical Trials Randomisation Blind and Double Blind Trials
Sequential Designs

8. Why should we worry about designing our experiments?
Would you take medication that hadn’t been through a properly designed clinical trial?
Would you set climate policy without a properly designed climate model experiment?

9. Computer Experiments (Climate model ensembles)
Computer experiments are very different from either clinical trials or field experiments.
In general we are using them to explore the properties of some computer simulator (model). This is usually the numerical solution of a system of PDE’s or ODE’s

10. Computer Experiments
Mathematically we can write our computer simulator as
where y is the output, x is the input and η(.) is the unknown mathematical function represented by the simulator
y and x are often very high dimension
y=\eta(x)

11. Computer Experiments
Normally the purpose of our computer experiment is to make some inference about the model
Estimate what the model does at inputs we haven’t run it at
Optimise the model parameters w.r.to some data
Make predictions

12. Types of ensemble Perturbed physics ensembles
Change inputs (parameters, initial conditions, …) to a single model
Multimodel ensembles
Look at multiple models

13. What is a good experimental design?
Make our inferences to the highest accuracy with the minimum cost

14. Optimal Design Fisher Information ≃ inverse of variance
Maximise the information = minimising the variance
D-optimal designs
Minimise the determinant of the variance matrix
A-optimal designs
Minimise the trace of the variance matrix
There are others (I, V, G, E optimality)

15. Bayesian Design Set up a D-optimal design by maximising the utility
U(S)=\int_\theta log (det[\mathcal{I}(\theta|S)])p(\theta)d\theta
\mathcal{I}(\theta|S)
is the variance matrix of θ in design S

16. Fisher Information Matrix
The Fisher Information matrix is an approximation to the inverse of the variance matrix.
\mathcal{F}_{i,j}=\frac{\partial^2 L}{\partial x_i \partial x_j}

17. An Possible Design
‘Star’ design

18. No aphorism is more frequently repeated in connection with field trials experiments, than that we must ask Nature few questions or, ideally, one question, at a time. The writer is convinced that this view is wholly mistaken. Nature, he suggests, will best respond to a logical and carefully thought out questionnaire; indeed, if we ask her a single question, she will often refuse to answer until some other topic has been discussed.
R.A. Fisher, 1926

19. What we need from the design of a climate model ensemble
We want to
Span the whole input space
Observe interactions
Minimise the number of simulator evaluations

20. Space Filling Designs Factorial Designs Latin Hypercubes
Pseudo random sequences
Sobol sequence

21. The Full Factorial
We set each input (factor) at a set number of levels
All combinations are included in the design
n levels of m factors needs nm points
This gets large quickly

22. An Example
52 factorial

23. Fractional Factorial Full factorials are expensive
For large number of factors only 2 or 3 levels
Can use fractional factorials (2 levels)

24. The Latin Hypercube Decide how many simulator runs you can afford
Divide each input range into that number of intervals
Allocate a point to each interval
Randomly permute across each input

25. The Latin Hypercube

26. The Latin Hypercube
We don’t have an algorithm for the optimal Latin hypercube
What is a good Latin hypercube?
Maximin
Orthogonal designs
Pragmatic designs

27. A Latin Hypercube

28. A maximin LHC

29. Are Factorials better than Latin Hypercubes

30. Low Discrepancy Sequences
Alternative to Latin hypercubes
Designed for multi-dimensional integrals
Examples include Halton sequences, Niederreiter nets and Sobol sequences
\delta^*(D)=\sup_B \left | \frac{I_B(D)}{N}-|B| \right |
B= \left \{ \prod_{i=1}^{d} [0,u_i) : u_i \in (0,1] \right \}

31. Warning: Hard Maths

32. Discrepancy Discrepancy is defined as where
is the number of points in B and
\delta^*(D)=\sup_B \left | \frac{I_B(D)}{N}-|B| \right |
B= \left \{ \prod_{i=1}^{d} [0,u_i) : u_i \in (0,1] \right \}
I_B(D)
|B| = \int_B dx

33. Low Discrepancy Sequences
It is believed, but not proved, that minimum discrepancy sequences have the property
Look for low values of
Examples include Halton sequences, Niederreiter nets and Sobol sequences
\delta^*(D) \leqslant C_d(\log n)^d + O(\log n)^{d-1}
C_d

34. End of Hard Maths

35. Sobol Sequences A low discrepancy sequence
A 2n-1 Sobol sequence is a Latin hypercube
Some projections of multi-dimensional Sobol sequences are not ‘good’

36. Sobol Sequences

37. Sobol Sequences

38. Sequential Designs So far our designs have been one off
We make a design and that dictates how we run the simulator
We do not learn from the early runs
An idea from clinical trials is to learn as we carry out the experiment

39. Sequential Design for Computer Experiments
Perform an initial experiment (usually space filling)
Add additional points to satisfy some criteria
We might add additional points where our predictions of simulator output are most uncertain
We might add additional points for optimisation

40. A D-optimal design for smoothness
I’m fitting an emulator to a computer experiment
Can we design an experiment to estimate the ‘smoothness’ parameters of the emulator optimally?

41. Emulators δ is a zero mean Gaussian process
\eta(\theta) = \mu(\theta) + \delta(\theta)
\mu(\theta) = A(\theta) \beta’
δ is a zero mean Gaussian process
This is defined in terms of a variance (σ2 and a correlation function (C(x1,x2))

42. An Example of an Emulator

43. Zhu and Stein (2004)
In the geostatistical context Zhu and Stein show that the Fisher information is approximately given by
U(S)=\int_B log (det[\mathcal{I}(B|S)])p(B)dB
\mathcal{I}_{i,j}(B|S) = \frac{1}{2} tr\left( \Sigma^{-1} \Sigma_i \Sigma^{-1} \Sigma_j\right)
\Sigma_k(i,j) = \frac{\partial \Sigma(B)}{\partial B_k}-(x_{i,k} - x_{j,k})^2 \Sigma(x_i,x_j)

44. Bayesian Design
Approximate the inverse of the covariance matrix by the Fisher information matrix
Set up a D-optimal design by maximising the utility

45. 5-point Sobol

46. 10-point Sobol

48. Sobol 10 +5
One at time (5)
Five at a time

50. Designing for Multiple Climate Models
So far we have considered designs for single simulators
How might we design for multiple models?
The IPCC problem
‘Ensemble of opportunity’?

51. So What’s the Problem Common outputs between simulators
Not common inputs
An important area for research

52. Conclusions Designing model ensembles can make them more efficient
make the experimenter think about the problem
There are a variety of designs around
Consult a statistician before you design the experiment
Design of computer experiments is an active area of research (not only in climate/environmental sciences)