Experimental Design, Response Surface Analysis, and Optimization

Outline Motivation and Terminology Difficulties in Solving the Basic Problem Examples of Factors and Responses Types/Examples of Experimental Design Full Factorial Designs Randomness of Effects Example: Full Factorial Design Situations with Many Factors Response Surfaces and Metamodels Regression Analysis Response Surface Methodology

Motivation and Terminology Useful when there are many alternatives to consider (e.g., numerous capacity levels of various types, numerous parameters for a proposed inventory system) Two basic types of variables: factors and responses Factors: input parameters: -controllable or uncontrollable -quantitative or qualitative Responses: outputs from the simulation model: -uncertain in nature Basic problem: find the best levels (or values of the parameters) in terms of the responses Experimental Design can tell you which alternatives to simulate so that you obtain the desired information with the least amount of simulation

Difficulties in Solving the Basic Problem Multiple Responses Uncertain Responses

Examples of Factors 1. Mean interarrival time (uncontrollable, quantitative) 2. Mean service time (controllable or uncontrollable, quantitative) 3. Number of servers (controllable, quantitative) 4. Queuing discipline (controllable, qualitative) 5. Reorder point (controllable, quantitative) 6. Mean interdemand time (uncontrollable, quantitative) 7. Distribution of interdemand time (uncontrollable, qualitative)

Examples of Responses 1. Mean daily production rate 2. Mean time in the system for patients 3. Mean inventory level 4. Number of customers who wait for more than 5 minutes

Types/Examples of Experimental Designs Completely Randomized Designs Randomized Complete Block Designs Nested Factorial Designs Split Plot Type Designs Latin Square Type Designs Full Factorial Designs Fractional Factorial Designs

2k Factorial Designs Suppose that we have k (k > 2) factors. A 2k factorial design would require that two levels be chosen for each factor, and that n simulation runs (replications) be made at each of the 2k possible factor-level combinations (design points). For 3 factors, this yields a Design Matrix:

Design Matrix for 3 Factors 2k Factorial Designs Design Matrix for 3 Factors Factor 1 Factor 2 Factor 3 Design Point Level Level Level Response 1 - - - O1 2 + - - O2 3 - + - O3 4 + + - O4 5 - - + O5 6 + - + O6 7 - + + O7 8 + + + O8 “+” refers to one level of a factor and “-” refers to the other level. Normally, for quantitative factors, the smallest and largest levels for each factor are chosen.

2k Factorial Designs: Estimating Main Effects The main effect of factor 1 is the change in the response variable as a result of the change in the level of the factor, averaged over all levels of all of the other factors If the effect of some factor depends on the level of another factor, these factors are said to interact. The degree of interaction (two-factor interaction effect) between two factors i and j is defined as half the difference between the average effect of factor i when factor j is at its “+” level ( and all factors other than i and j are held constant) and the average effect of i when j is at its “-” level; for example,

Example: Full Factorial (2k) Design Consider a simulation model of reorder point, reorder quantity inventory system. The two decision variables, or factors, to consider are the reorder point (P) and the reorder quantity (Q) for the inventory system. The maximum and minimum allowable values for each are given below: Suppose that the response variable output by the model is the long-run average monthly cost (composed of three components: holding cost, shortage costs, and ordering costs) in thousands of dollars.

Example: Full Factorial (2k) Design A 22 factorial design matrix with simulation results (for 10 replications at each design point) might be given by: where a factor level of “-” indicates the minimum possible value for that factor, and a factor level of “+” indicates the maximum possible level; for example, design point 2 has P=40 and Q=15.

Example: Full Factorial (2k) Design The response given is the average cost over the 10 replications. Now, the main effects are given by: The interaction effect (ePQ) is given by: Therefore, the average effect of increasing P from 20 to 40 is to increase monthly cost by 1.2, and the average effect of increasing Q from 15 to 50 is to decrease monthly cost by 5.3. Hence, it would be advisable to set P as low as possible and set Q as high as possible.

Example: Full Factorial (2k) Design Also since the interaction effect is positive, it would seem advisable to set P and Q at opposite levels. (Of course, all of the above could be inferred from a cursory analysis of the responses for the various design points.) Note also that the literal interpretation of main effects assumes no interaction effects (pages 669 and 670 of Law and Kelton, 1991).

Randomness of Effects Note that the main and interaction effects computed in the previous examples are just random variables. To determine if the effects are “significant” or real, and not due to random fluctuations, one could compute values for the main and interaction effects 10 times (once for each replication) and form confidence intervals for each of the main effects, and the interactions effect. If the confidence interval contains 0, then the effect is not statistically significant. (Note that statistical significance does not necessarily imply practical significance).

Situations with Many Factors When there are many factors to consider, full factorial, or even fractional factorial designs may not be feasible from a computational standpoint. Other types of design (e.g., Plackett-Burman designs or “supersaturated designs”) may be appropriate (Mauro, 1986). Another approach is to reduce the number of factors to consider via “factor screening” techniques, involving, for example, group screening in which a whole group is treated as a factor.

Response Surfaces and Metamodels A response surface is a graph of a response variable as a function of the various factors. A metamodel (literally, model of the simulation model), is an algebraic representation of the simulation model, with the factors as independent variables and a response as the dependent variable. It represents an approximation of the response surface. The typical metamodel used in a simulation application is a regression model.

Response Surfaces and Metamodels A metamodel through the use of response surface methodology can be used to find optimal values for a set of factors. It can also be used to answer “what if” questions. (Experimentation with a metamodel is typically much less expensive than using a simulation model directly). An experimental design process assumes a particular metamodel, e.g.,

Basic Concepts of Regression Analysis Regression is used to determine the “best” functional relation among variables. Suppose that the functional relationship is represented as: E(Y) = f (X1, ..., Xp / B1, ..., BE) where E(Y) is the expected value of the response variable Y; the X1, ..., Xp are factors; and the B1, ..., BE are function parameters; e.g., E(Y) = B1 + B2 X1 + B3 X2 + B4 X1 X2

Basic Concepts of Regression Analysis The observed value for Y, for a given set of X ’s, is assumed to be a random variable, given by: Y = f (X1, ..., Xp/B1, ..., BE) + Where is a random variable with mean equal to 0 and variance . The values for B1,...,BE are obtained by minimizing the sum of squares of the deviations.

Response Surface Methodology Source: (Fu, 1994) Response surface methodology (RSM) involves a combination of metamodeling (i.e., regression) and sequential procedures (iterative optimization).

Response Surface Methodology RSM involves two phases: Fit a linear regression model to some initial data points in the search space (through replications of the simulation model). Estimate a steepest descent direction from the linear regressions model, and a step size to find a new (and better) solution in the search space. Repeat this process until the linear regression model becomes inadequate (indicated by when the slope of the linear response surface is approximately 0; i.e., when the interaction effects become larger than than the main effects). Fit a nonlinear quadratic regression equation to this new area of the search space. Then find the optimum of this equation.

Terminology in Experimental Design Source:(Ostle, 1963) Replication - the repetition of the basic experiment Treatment - a specific combination of several factor levels Experimental Unit - the unit to which a single treatment is applied to one replication of the basic experiment Experimental Error - the failure of two identically treated experimental units to yield identical results Confounding - the “mixing up” of two or more factors so that it’s impossible to separate the effects

Terminology in Experimental Design Randomization - randomly assigning treatments to experimental units (assures independent distribution of errors) Main Effect (of a factor) - a measure of the change in a response variable to changes in the level of the factor averaged over all levels of all the other factors Interaction is an additional effect (on the response) due to the combined influence of two or more factors