1. 1/26 LOÏC LE GRATIET Bayesian analysis of hierarchical codes with different levels of accuracy Masters thesis conducted from 07/04/2010 to 15/09/2010 in the French atomic energy authority AEA. PhD advisor: M. Josselin Garnier* Master advisor:M. Olivier Roustant** AEA advisor: Mlle. Claire Cannaméla * Professor at the Laboratory of Probability and Random Models & Jacques-Louis Lions, University of Paris VII ** Doctor at the Ecole Nationale Supérieure des Mines de Saint-Etienne

2. 2/26 AEA: the French atomic energy authority Defense and global security Technology for health and information Energy that does not emit greenhouse gases An organization for technological research Ensure the sustainability of nuclear deterrence Last thermonuclear experimentation 1996 Direction of military applications Simulation program

3. 3/26 The context of my master thesis is the design of a system modelled by a complex and time-consuming simulator Experimental system simulated Input x 1 output Input x N Accurate and complex code Cheap code Y i = f(X) X 2 X 1 X 1 X 2 Y i X 2 X 1 Y i X 2 X 1 Y i X 2 X 1 Simulations X 1 X 2 X 1 X 2 X 1 X 2 Space-filling design Metamodel or surrogate AEA AEA : Neural networks Kriging Bayesian calibration How can we use the cheap code to improve the approximation of the complex code ?

4. 4/26 How can we improve the approximation of a complex and accurate code using a faster but less accurate one? Literature review Aim of the master thesis Methods and results Conclusion Prospects Table of contents:

5. 5/26 The first multi-levels metamodel was suggested by Kennedy and O’Hagan and is built with Gaussian processes [1] T.J. Santner, B.J. Williams & W.I. Notz (2003) The design and Analysis of Computer Experiments, Springer Series in Statistics [2] C.E. Rasmussen & C.K.I. Williams (2006) Gaussian Processes for Machine Learning, The MIT Press. [3] M.C. Kennedy & A. O’Hagan (2000) Predicting the output from a complex computer code when fast approximations are available, Biometrika, 87, 1 [4] A.I.J. Forrester, A. Sobester & A.J. Keane (2007) Multi-fidelity optimization via surrogate modelling, Proc. R. Soc. A463 [5] Z. Qian & C.F.J. Wu (2008) Bayesian hierarchical modeling for integrating low-accuracy and high accuracy experiments, Technometrics, vol.50 no.2. Literature review Aim of the master thesis Methods and results Conclusion Prospects

6. 6/26 The first multi-levels metamodel — a particular case of co- kriging — is a stationary auto-regressive model [3] M.C. Kennedy & A. O’Hagan (2000) Predicting the output from a complex computer code when fast approximations are available, Biometrika, 87, 1. Modelling with Gaussian processes: Expensive code:Cheap code: Markov property between the two codes: AR(1) model: Distribution of Y 0, the prediction of the complex code’s output at x 0 : Literature review Aim of the master thesis Methods and results Conclusion Prospects The assumption of nested space-filling designs : D2 ⊆ D1

7. 7/26 The aim of the Master thesis — which is to study the method of multi-levels metamodelling for practical use — was declined into different stages deduced from the key issues found in the literature Highlighted key issues: Numerical estimation of   No pathologic cases The Experiments design strategies advantage the cheap code Underestimating the variance of prediction The stages: Model generalization Test on academic cases Improvement of the Experiments design strategies Bayesian modelling Literature review Aim of the master thesis Methods and results Conclusion Prospects

8. 8/26 The generalization of the model to the non-stationary case is difficult to use in practice without an analytical estimation of the coefficient  ρ. Let us consider: Maximum likelihood estimation: analytical expression for the estimation of β 1, β 2, σ 1 et σ 2 but the estimation of the parameter   is numerical ! First idea: estimate β ρ with Maximum likelihood estimation But this estimation depends on β 2 and the estimation of β 2 depends on β ρ Literature review Aim of the master thesis Methods and results  Generalization Academic cases Bayesian modelling Experiments design Real case Conclusion Prospects

9. 9/26 We cannot estimate β 2 and β ρ independently: we estimate them together with a Bayesian estimation method Bayesian estimation of parameters :  « Jeffreys prior »:  Bayes’ relation with: Considering and We obtain: with: and: Literature review Aim of the master thesis Methods and results  Generalization Academic cases Bayesian modelling Experiments design Real case Conclusion Prospects

10. 10/26 Toy example 1: the complex code is a linear combination of functions which includes the cheap code [4] A.I.J. Forrester, A. Sobester & A.J. Keane (2007) Multi-fidelity optimization via surrogate modelling, Proc. R. Soc. A463 Estimation of parameters: ρ = 2 β 1 = -3.49 β 2 = (20 ; -20) σ 1 2 = 32.75 σ 2 2 = 7.02e-30 θ 1 = 0.25 θ 2 = 0.80 Validation: RMSE = 5.68e-2 Q2 = 99.98% Literature review Aim of the master thesis Methods and results Generalization  Academic cases Bayesian modelling Experiments design Real case Conclusion Prospects

11. 11/26 Toy example 2: this example does not check the Markov property The expensive code is a dilatation of the cheap one: z 2 (x) = z 1 (3x/2) Literature review Aim of the master thesis Methods and results Generalization  Academic cases Bayesian modelling Experiments design Real case Conclusion Prospects

12. 12/26 The Bayesian modelling allows us to incorporate epistemic uncertainty due to the parameter estimation and to consider a possible expert advice. The aim :  From the distribution:  Deduce the distribution: Stages of Bayesian modelling: 1)Determine prior distributions of parameters 2)Deduce posterior distributions of parameters from prior distributions and codes responses 3)Deduce predictive distributions by integrating posterior distributions Literature review Aim of the master thesis Methods and results Generalization Academic cases  Bayesian modelling Experiments design Real case Conclusion Prospects

13. 13/26 1) Determine prior distributions of parameters: how can we consider a possible expert advice or describe our lack of knowledge? Informative prior distributions i.e. with expert advice : Non-informative prior distributions « Jeffreys prior » : We calculate the parameters posterior distributions with the Bayes’ relation: Literature review Aim of the master thesis Methods and results Generalization Academic cases  Bayesian modelling Experiments design Real case Conclusion Prospects

14. 14/26 2) Calculate the parameters posterior distributions: we use the data to improve the parameters estimations. Posterior distributions for regression coefficients: Posterior distributions for variance coefficients: with: Literature review Aim of the master thesis Methods and results Generalization Academic cases  Bayesian modelling Experiments design Real case Conclusion Prospects

15. 15/26 3) Calculate predictive distributions: integrate the parameters posterior distributions in the predictive distributions. When  1 2,  2 2,   et  2 are supposed to be known: Let’s consider: It can be shown that: Literature review Aim of the master thesis Methods and results Generalization Academic cases  Bayesian modelling Experiments design Real case Conclusion Prospects

16. 16/26 3) Calculate predictive distributions: we integrate all posterior distributions except hyper-parameters of correlation ones. We integrate the posterior distributions of   et  2 : where:. using a Monte-Carlo method We integrate the posterior distributions of  1 2 and  2 2 : using a quadrature method (trapezoidal numerical integration).  Note: we consider the hyper-parameters of correlation  1 et  2 as known and we estimate them by maximizing the concentrated restricted log-likelihood. Literature review Aim of the master thesis Methods and results Generalization Academic cases  Bayesian modelling Experiments design Real case Conclusion Prospects

17. 17/26 We want to simplify the space-filling design and advantage the complex-code experiments design. The assumption of nested space-filling designs : D 2 ⊆ D 1 First idea: build D 1 and extract from it the best D 2 possible. Problems: 1) We do not advantage the structure of D 2. 2) The extraction D 2 can be complex. Our idea: First stepSecond step Literature review Aim of the master thesis Methods and results Generalization Academic cases Bayesian modelling  Experiments design Real case Conclusion Prospects

18. 18/26 An example in 2D: hydrodynamic simulator« MELTEM » We use MELTEM to simulate a turbulence model for gaseous mixtures induced by Richtmyer-Meshkov instability. Studied response: ratio between transversal and longitudinal speed variation Parameters: Phenomenological coefficients used in the equation of the dissipation energy Estimated correlation: 98%  Number of runs for the cheap code: 200  Time-processing for one run: 20s  Number of runs for the complex code: 10  Time-processing for one run: 8min Literature review Aim of the master thesis Methods and results Generalization Academic cases Bayesian modelling Experiments design  Real case Conclusion Prospects

19. 19/26 First, we build a non-bayesian co-kriging Kernel: matern-5/2 Estimated coefficients for the cheap code:  β 1 = 2.87  Hyper-parameters: Difference between the expensive code and the adjusted cheap code:  β ρ = 0.99 ; β 2 = -0.38  Hyper-parameters: Literature review Aim of the master thesis Methods and results Generalization Academic cases Bayesian modelling Experiments design  Real case Conclusion Prospects

20. 20/26 Leave-one-out cross validation and external validation Literature review Aim of the master thesis Methods and results Generalization Academic cases Bayesian modelling Experiments design  Real case Conclusion Prospects

21. 21/26 Comparison between ordinary kriging and non bayesian co- kriging when the number of runs for the complex code varies. Literature review Aim of the master thesis Methods and results Generalization Academic cases Bayesian modelling Experiments design  Real case Conclusion Prospects

22. 22/26 Comparison between non bayesian co-kriging and bayesian co- kriging. Responses are sorted in ascending order Literature review Aim of the master thesis Methods and results Generalization Academic cases Bayesian modelling Experiments design  Real case Conclusion Prospects

23. 23/26 Comparison between non bayesian co-kriging and bayesian co- kriging. Literature review Aim of the master thesis Methods and results Generalization Academic cases Bayesian modelling Experiments design  Real case Conclusion Prospects

24. 24/26 Comparison between non bayesian co-kriging and bayesian co- kriging. Literature review Aim of the master thesis Methods and results Generalization Academic cases Bayesian modelling Experiments design  Real case Conclusion Prospects

25. 25/26 Predictive distributions at one point for bayesian and non- bayesian co-kriging. Literature review Aim of the master thesis Methods and results Generalization Academic cases Bayesian modelling Experiments design  Real case Conclusion Prospects

26. 26/26 Conclusion We manage to obtain an analytical expression for the estimation of β ρ with a Bayesian estimation.  The analytical estimation of β ρ allows us to generalize the model in practical use.  It also allows us to use the existing kriging library in our code, especially in order to estimate the hyper-parameters of correlation. We manage to obtain under some assumptions an analytical predictive distribution for the Bayesian modelling This predictive distribution allow us to avoid prohibitive implementation for the Bayesian modelling. Literature review Aim of the master thesis Methods and results Conclusion Prospects

27. 27/26 Prospects  Adapt the method for multi-levels codes (>2)  Use the co-kriging for Monte-Carlo codes.  Use the co-kriging to solve ODE with complex functions to assess  Improve the estimation of the predictive distributions variance  See how to use the co-kriging for sensitivity analysis and optimization  Study different experiments design strategies Literature review Aim of the master thesis Methods and results Conclusion Prospects

28. Thank you for your attention