1 Physical Experiments & Computer Experiments, Preliminaries Chapters 1&2 “Design and Analysis of Experiments“ by Thomas J. Santner, Brian J. Williams and William I. Notz EARG presentation Oct 3, 2005 by Frank Hutter

2 Preface What they mean by Computer Experiments Code that serves as a proxy for physical process Code that serves as a proxy for physical process Can modify inputs and observe how process output is affedcted Can modify inputs and observe how process output is affedctedMath Should be understandable with a Masters level training of Statistics Should be understandable with a Masters level training of Statistics

3 Overview of the book Chapter 1: intro, application domains Chapter 2: Research goals for various types of inputs (random, controlled, model parameters) Research goals for various types of inputs (random, controlled, model parameters) Lots of definitions, Gaussian processes Lots of definitions, Gaussian processes Chapters 3-4: Predicting Output from Computer Experiments Chapter 5: Basic Experimental Design (similar to last term) Chapter 6: Active Learning (sequential experimental design) Chapter 7: Sensitivity analysis Appendix C: code

4 Physical experiments: a few concepts we saw last term (2f) Randomization In order to prevent unrecognized nuisance variables from systematically affecting response In order to prevent unrecognized nuisance variables from systematically affecting responseBlocking Deals with recognized nuisance variables Deals with recognized nuisance variables Group experimental units into homogeneous groups Group experimental units into homogeneous groupsReplication Reduce unavoidable measurement variation Reduce unavoidable measurement variation Computer experiments Deterministic outputs Deterministic outputs “None of the traditional principles [...] are of use“ “None of the traditional principles [...] are of use“

5 Types of input variables (15f) Control variables x c Can be set by experimenter / engineer Can be set by experimenter / engineer Engineering variables, manufactoring variables Engineering variables, manufactoring variables Environmental variables X e Depend on the environment/user Depend on the environment/user Random variables with known or unknown distribution Random variables with known or unknown distribution When known for a particular problem: x e When known for a particular problem: x e Noise variables Noise variables Model variables x m Parameters of the computer model that need to be set to get the best approximation of the physical process Parameters of the computer model that need to be set to get the best approximation of the physical process Model parameters, tuning parameters Model parameters, tuning parameters

6 Examples of Computer Models (6ff) ASET (Available Safe Egress Time) 5 inputs, 2 outputs 5 inputs, 2 outputs Design of Prosthesis Devices 3 environment variables, 2 control variables 3 environment variables, 2 control variables 2 competing outputs 2 competing outputs Formation of Pockets in Sheet Metal 6 control variables, 1 output 6 control variables, 1 output Other examples Optimally shaping helicopter blade – 31 control variables Optimally shaping helicopter blade – 31 control variables Public policy making: greenhouse gases – 30 input variables, some of them modifiable (control variables) Public policy making: greenhouse gases – 30 input variables, some of them modifiable (control variables)

7 ASET (Available Safe Egress Time) (4f) Evolution of fires in enclosed areas Inputs Room ceiling height and room area Room ceiling height and room area Height of burning object Height of burning object Heat loss fraction for the room (depends on insulation) Heat loss fraction for the room (depends on insulation) Material-specific heat release rate Material-specific heat release rate Maximum time for simulation (!) Maximum time for simulation (!)Outputs Temperature of the hot smoke layer Temperature of the hot smoke layer Distance of hot smoke layer from fire source Distance of hot smoke layer from fire source

8 Design of Prosthesis Devices (6f) 2 control variables b, the length of the bullet tip d, the midstem parameter 3 environment variables , the joint angle E, the elastic modulus of the surrounding cancellous bone Implant-bone interface friction 2 conflicting outputs Femoral stress shielding Implant toggling (flexible prostheses minimize stress, but toggle more  loosen)

9 Formation of Pockets in Steel (8ff) 6 control variables Length l Width w Fillet radius f Clearance c Punch plan view radius p Lock bead distance dOutput Failure depth (depth at which the metal tears)

10 Research goals for homogeneous-input codes (17f) Homogeneous-input: only one of the three possible variable types present All control variables: x=x c Predict y(x) well for all x in some domain X Predict y(x) well for all x in some domain X Global perspective Global perspective Integrated squared error s X [y‘(x) - y(x)] 2 w(x) dx Can‘t be computed since y(x) unknown, but in Chapter 6 we‘ll replace [y‘(x)-y(x)] 2 by a computable posterior mean squared value Local perspective Local perspective Level set: Find x such that y(x) = t 0 t 0 = maximum value

11 All environmental variables: x=X e (18) How does the variability in X e transmit through the computer code ? Find the distribution of y(X e ) When the problem is to find the mean: Latin hypercube designs for choosing the training sites Latin hypercube designs for choosing the training sites

12 All model variables x=x m (18f) Mathematical modelling contains unknown parameters (unknown rates or physical constants) Calibration (parameter fitting) Choose the model variables x m so that the computer output best matches the output from the physical experiment Choose the model variables x m so that the computer output best matches the output from the physical experiment

13 Research goals for mixed inputs (19ff) Focus on case with control and environmental variables x=(x c,X e ) where X e has a known distribution Example: hip prosthesis y(x c,X e ) is a random variable whose distribution is induced by X e Mean  x c ) = E{y(x c,X e )} Mean  x c ) = E{y(x c,X e )} Upper alpha quantile   =   (x c ): Upper alpha quantile   =   (x c ): P{y(x c,X e ) >=   } = 

14 Research goals for mixed inputs: simple adaption of previous goals (20ff) Predict y well over its domain: Minimize s X [  ‘(x) -  (x)] 2 w(x) dx Minimize s X [  ‘(x) -  (x)] 2 w(x) dx (again, there is a Bayesian analog with computable mean) (again, there is a Bayesian analog with computable mean) Maximize the mean output: max x c  (x c )

15 When the distribution of X e is unknown Various flavours of robustness G-robust: minimax Want to minimize your maximal loss Want to minimize your maximal loss Pessimistic Pessimistic  (.)-robust  (.)-robust Minimize weighted loss (weighted by prior density on distribution over X e ) Minimize weighted loss (weighted by prior density on distribution over X e )M-robust Suppose for a given x c, y(x c,x e ) is fairly flat Suppose for a given x c, y(x c,x e ) is fairly flat Then value of X e doesn‘t matter so much for that x c Then value of X e doesn‘t matter so much for that x c Maximize  (x c ) subject to constraints on variance w.r.t. X e Maximize  (x c ) subject to constraints on variance w.r.t. X e