1. Lecture 9 Output Analysis for a Single Model

2. 2  Output analysis is the examination of data generated by a simulation.  Its purpose is to predict the performance of a system or to compare the performance of two or more alternative system designs.  This lecture deals with the analysis of a single system, while next lecture deals with the comparison of two or more systems.  Output analysis is the examination of data generated by a simulation.  Its purpose is to predict the performance of a system or to compare the performance of two or more alternative system designs.  This lecture deals with the analysis of a single system, while next lecture deals with the comparison of two or more systems.

3. 3 Type of Simulation with respect to Output Analysis  When analyzing simulation output data, a distinction is made between terminating or transient simulation and steady-state simulation.  A terminating simulation is one that runs for some duration of time T E, where E is a specified event (or set of events) which stops the simulation.  Example 11.1: Shady Grove Bank operates 8:30 – 16:30, then T E = 480min.  Example 11.3: A communication system consists of several components. Consider the system over a period of time, T E, until the system fails. E = {A fails, or D fails, or (B and C both fail)}  When analyzing simulation output data, a distinction is made between terminating or transient simulation and steady-state simulation.  A terminating simulation is one that runs for some duration of time T E, where E is a specified event (or set of events) which stops the simulation.  Example 11.1: Shady Grove Bank operates 8:30 – 16:30, then T E = 480min.  Example 11.3: A communication system consists of several components. Consider the system over a period of time, T E, until the system fails. E = {A fails, or D fails, or (B and C both fail)}

4. 4 Terminating Simulation  When simulating a terminating system, the initial conditions of the system at time 0 must be specified, and the stopping time T E, or alternatively, the stopping event E, must be well defined.  Whether a simulation is considered to be terminating or not depends on both the objectives of the simulation study and the nature of the system.  Examples 11.1 and 11.3 are considered the terminating systems because:  Ex. 11.1: the objective of interest is one day’s operation;  Ex. 11.3: short-run behavior, from time 0 until the first system failure.  When simulating a terminating system, the initial conditions of the system at time 0 must be specified, and the stopping time T E, or alternatively, the stopping event E, must be well defined.  Whether a simulation is considered to be terminating or not depends on both the objectives of the simulation study and the nature of the system.  Examples 11.1 and 11.3 are considered the terminating systems because:  Ex. 11.1: the objective of interest is one day’s operation;  Ex. 11.3: short-run behavior, from time 0 until the first system failure.

5. 5 Steady-state Simulation  A nonterminating system is a system that runs continuously, or at least over a very long period of time.  For example, assembly lines which shut down infrequently, continuous production systems of many different types, telephone systems and other communications systems such as the Internet, hospital emergency rooms, fire departments, etc.  A steady-state simulation is a simulation whose objective is to study long-run, or steady-state, behavior of a nonterminating system.  The stopping time, T E, is determined not by the nature of the problem but rather by the simulation analyst, either arbitrarily or with a certain statistical precision in mind.  A nonterminating system is a system that runs continuously, or at least over a very long period of time.  For example, assembly lines which shut down infrequently, continuous production systems of many different types, telephone systems and other communications systems such as the Internet, hospital emergency rooms, fire departments, etc.  A steady-state simulation is a simulation whose objective is to study long-run, or steady-state, behavior of a nonterminating system.  The stopping time, T E, is determined not by the nature of the problem but rather by the simulation analyst, either arbitrarily or with a certain statistical precision in mind.

6. 6 Stochastic Nature of Output Data  Consider one run of a simulation model over a period of time [ 0, T ]. Since the model is an input-output transformation, and since some of the model input variables are random variable, it follows that the model output variables are random variables.  The stochastic (or probability) nature of output variables will be observed.  Example 2.2 (Able-Baker carhop problem)  Input: randomness of arrival time and service time  Output: randomness of utilization and time spent in the system per customer.  Consider one run of a simulation model over a period of time [ 0, T ]. Since the model is an input-output transformation, and since some of the model input variables are random variable, it follows that the model output variables are random variables.  The stochastic (or probability) nature of output variables will be observed.  Example 2.2 (Able-Baker carhop problem)  Input: randomness of arrival time and service time  Output: randomness of utilization and time spent in the system per customer.

7. 7 Output Analysis for Terminating Simulations  Consider the estimation of a performance parameter,  (or  ), of a simulated system.  The simulation output data is of the form {Y 1, Y 2, …, Y n } (discrete-time data) for estimating .  E.g. the delay of customer i, total cost in week i.  The simulation output data is of the form {Y(t), 0  t  T E } (continuous-time data) for estimating .  E.g. the queue length at time t, the number of backlogged orders at time t.  Point Estimation:  Consider the estimation of a performance parameter,  (or  ), of a simulated system.  The simulation output data is of the form {Y 1, Y 2, …, Y n } (discrete-time data) for estimating .  E.g. the delay of customer i, total cost in week i.  The simulation output data is of the form {Y(t), 0  t  T E } (continuous-time data) for estimating .  E.g. the queue length at time t, the number of backlogged orders at time t.  Point Estimation:

8. 8 Output Analysis for Terminating Simulations (cont’)  By the Central Limited Theorem (CLT), for n  30, where  Interval Estimation:  An approximate 100(1 -  )% confidence interval for  is given by:  By the Central Limited Theorem (CLT), for n  30, where  Interval Estimation:  An approximate 100(1 -  )% confidence interval for  is given by:

9. 9 Output Analysis for Terminating Simulations (Example)  Example 11.10 (Able Baker Carhop Problem) Run, rUtilization,Average System Time, 10.8083.74 20.8754.53 30.7083.84 40.8423.98

10. 10 Number of Replications  PRECISION LEVEL  Suppose that an error criterion  is specified; in other words, it is desired to estimate  by to within with high probability, say at least 1 – .  PRECISION LEVEL  Suppose that an error criterion  is specified; in other words, it is desired to estimate  by to within with high probability, say at least 1 – .

11. 11 Number of Replications (Example)  Example 11.12 (Able Baker Carhop Problem)  Suppose that it is desired to estimate Able’s utilization in Example 11.7 to within with probability 0.95. An initial sample size R 0 =4 is taken.  Step 1:  Step 2:  Example 11.12 (Able Baker Carhop Problem)  Suppose that it is desired to estimate Able’s utilization in Example 11.7 to within with probability 0.95. An initial sample size R 0 =4 is taken.  Step 1:  Step 2: R131415 t 0.025, R-1 2.182.162.14 15.3915.1014.83 R = 15 Additional replications: R – R 0 = 15 – 4 = 11

12. 12 Output Analysis for Steady-State Simulations  Prior to beginning analysis of output data, the modeler must take every effort to ensure that the output represents an accurate estimate of the true system values.  One useful technique for improving the reliability of output results from steady-state simulation is to provide an initialization period for which statistics are not kept.  A steady-state condition implies that a simulation has reached a point in time where the state of the model is independent of the initial start-up conditions.  Prior to beginning analysis of output data, the modeler must take every effort to ensure that the output represents an accurate estimate of the true system values.  One useful technique for improving the reliability of output results from steady-state simulation is to provide an initialization period for which statistics are not kept.  A steady-state condition implies that a simulation has reached a point in time where the state of the model is independent of the initial start-up conditions.

13. 13 Output Analysis for Steady-State Simulations (cont’)  The amount of time required to achieve steady-state conditions is referred to as a warm-up period.  Data collection begins after a warm-up period is completed.  Determining the length of this period can be accomplished by utilizing moving averages calculated from the output produced by multiple model replications.  The amount of time required to achieve steady-state conditions is referred to as a warm-up period.  Data collection begins after a warm-up period is completed.  Determining the length of this period can be accomplished by utilizing moving averages calculated from the output produced by multiple model replications.

14. 14 Warm-up Period  Determine A Warm-up Period in a Steady-state Simulation PeriodAverage Costw = 5w = 10w = 19 1422.00 2468.16522.20 3676.45502.72 4572.88568.92 5374.10571.26 6842.90560.94 7625.92587.72563.86 8473.08585.46574.53 9685.88568.25569.68 10528.79588.95578.06 11500.22611.93565.67 12716.52575.28568.46558.81 13443.33546.57569.65561.05 14487.13593.43564.58563.82

15. 15 Moving Average d = 12

16. 16 Output Analysis for Steady-State Simulations (Example) Observed cost during i-th period and j-th replication PeriodRep 1Rep 2Rep 3Rep 4Rep 5 13376.81500.97192.96509.00636.92 14352.05329.30587.45336.11530.74 15518.96634.81716.81533.051899.13 16673.88853.97563.86179.72864.17 17376.991098.67290.92205.43276.93 18139.26339.08563.49319.10189.20 19199.544032.35355.94138.88215.99 20542.79908.48633.90349.55727.93 21383.47317.29165.11345.11106.20 22276.26387.05366.19789.89613.58 23336.39388.63605.87315.20818.90 24562.06323.831311.94339.76312.95 25931.46236.80706.43484.30658.11 26182.82352.79991.44271.731815.32 PeriodRep 1Rep 2Rep 3Rep 4Rep 5 27589.21649.52544.64296.36289.96 28103.42936.04393.21771.45151.18 29219.141338.29163.15169.59938.19 30169.36841.89651.41492.09232.72 31791.25137.11734.38807.81401.16 321360.99274.57457.19148.87231.46 33530.041259.50497.511300.90990.27 34198.98275.20177.60723.29414.02 35523.281012.45904.77212.75523.06 36633.30723.07431.72245.69158.22 37631.47455.121256.28287.57351.78 38807.241627.84994.93215.50603.36 39271.41138.54352.43441.73352.38 Avg.469.70752.74565.96427.05567.14

17. 17 Output Analysis for Steady-State Simulations (Example)  Confident Interval for a Steady-State Simulation