1. Building Valid, Credible & Appropriately Detailed Simulation Models

2. Validation and Verification
Validation is the process of determining whether a simulation model is an accurate representation of the system for the particular objectives of the study.
A valid model can be used to make decisions
Validation depends on the complexity of the model
There is no perfect or absolute validity, the simulation model is only an approximation
Models should be developed for a particular set of purposes
The performance measure used to validate a model should be actually used for evaluating system designs
Validation should be done during model development
Verification is the process of determining whether the conceptual simulation model is correctly translated into a computer program.

3. Credibility and Accreditation
Credibility of a simulation model and its results is obtained if the manager and other key personnel accept them as “correct”. To attain credibility:
The manager should understand and agree with model assumptions
Validation and verification should be demonstrated
The manager should own the project and be involved in it
Model developers should be reputable
Accreditation is the process of determining officially whether a simulation model is acceptable for a particular purpose. Issues to be considered are:
Validation and verification
Simulation model development and use history
Quality of data
Quality of documentation
Known problems and limitations

4. Timing and Relationships
Steps in a Sound Simulation Study

5. Validation vs Output Data Analysis
Suppose that we want to estimate the mean μS of some system. We construct a similar simulation model with corresponding mean M and make a simulation run to obtain an estimate of M. Then, the error is

6. Guidelines for Determining the Level of Model Detail
Carefully define the specific issues to be investigated and the measures of performance that will be used for evaluation
The entity moving through the simulation model does not have to be the same as the entity moving through the real system
Use subject-matter experts and sensitivity analyses to help determine the level of detail
A mistake often made by beginners is to include an excessive amount of model detail
Do not have more detail than necessary to address your issues
The level of model detail should be consistent with the type of data available
Time and money constraints are a major factor in determining the amount of model detail
If the number of factors for the study is large, then use a coarse simulation or analytical model to identify which factors have a significant impact on system performance

7. Verification of Simulation Computer Programs
Write and debug the computer program in modules and subprograms
Have more than one person to review the program
Run the simulation under a variety of settings of the input parameters, and check to see that the output is reasonable
Trace the program to compare the state of the simulated system by hand calculations. If available, an interactive debugger allows the analyst to stop the simulation at selected points in time
The model should be run, when possible, under simplifying assumptions for which its true characteristics are known or can be computed
It may be helpful to observe an animation of the simulation
Compare the sample mean and sample variance of input probability distributions with the desired mean and variance
Use a commercial simulation package to reduce the amount of programming

8. The M/M/1 queue
Inter-arrival times Expo(1) hours, Service times Expo(0.9) hours
Simulate for 1000 hours
Performance Measure
Simulation
Theoretical
Number in
1033
1000
WIP (number in system)
11.33 9
Utilization of server
0.93
0.9

9. Comparison with Theory

10. Techniques for Increasing Model Validity and Credibility - 1
Collect high-quality information and data on the system
Conversations with subject-matter experts
Observations of the system to obtain data
Use of existing theory on model assumptions
Relevant results from similar simulation studies
Experience and intuition of the modelers
Interact with the manager on a regular basis
The manager should formulate/reformulate the objectives
The manager’s involvement should be maintained
The manager’s knowledge contributes to actual validity
The model is more credible if management understands it

11. Techniques for Increasing Model Validity and Credibility - 2
Maintain an assumptions document and perform a structured walk-through
Record the assumptions in a report with information on
Goals, specific issues, performance measures
Detailed description of each subsystem
Simplifying assumptions
Summaries of data and input probability distributions
Sources of important and controversial information
Go over the assumptions by presenting a walk-through to all those involved
Use animation to find invalid model assumptions and to enhance credibility

12. Techniques for Increasing Model Validity and Credibility - 3
Validate components of the model using quantitative techniques
Use statistical tests: t-test, Chi-square goodness of fit test, Kolmogorov-Smirnov test, Kruskal-Wallis test of homogeneity, Other tests and procedures
Use sensitivity analysis to identify the important factors: The value of a parameter, The choice of a distribution, The entity moving through the system, The level of detail of a subsystem, Data crucial for simulation (using a course model)
Validate the output from the overall simulation model
Results validation establishes close resemblance of simulation output data to the expected output of the actual system
This often requires statistical procedures
Output data is often non-stationary and auto correlated!

13. Management’s Role in the Simulation Process
Formulating program objectives
Directing personnel to provide information and data
Interacting with the simulation modeler on a regular basis
Using the simulation results as an aid in decision-making

14. Statistical Procedures for Comparing Real-World Observations and Simulation Output Data
Inspection Approach
This is based on comparing the statistics (like mean, variance, correlation coefficient, histogram) without a formal testing procedure. The inherent problem is that each statistic is essentially from a sample of size 1 only!
Example 5.34:
Suppose that the real-world system is the M/M/1 queue with  = 0.6 and the corresponding simulation model is the M/M/1 queue with  = The arrival rate is 1 for both. Suppose that the output process is D1, D2, D3, … .

15. Inspection Approach
It is known from Heathcote and Winer (1969) that

16. Correlated Inspections
Game 1: Roll a single die. X is the outcome
Game 2: Roll a single die then toss a fair coin. Y is the number on the die + the outcome of toss (Heads=1, Tails=0).
In a simulation, we can compare E[X] and E[Y] in different ways:
Independent experiments: roll different dies to generate realizations of X and Y
Dependent experiments: roll the same die to generate realizations of X and Y

17. Correlated Inspection Approach
This is a more definitive approach to validate the assumptions of the simulation model other than the probability distributions. The system and the model both experience exactly the same observations from the input random variables using historical data.

18. Example
Suppose that the system is the five-teller bank with jockeying and the simulation model is the same bank without jockeying. Assume that the mean service time is 4 minutes.
X = average delay in queue for the system (mean X )
Y = average delay in queue for the simulation model (mean Y )
We want to estimate the difference X – Y by experimenting with the same data for 500 days and observing the differences Xj – Yj. Suppose also that independent random numbers are used to generate interarrival times and service times to obtain the average delays given by Y´j.

19. Simulation Results

20. Confidence-Interval Approach Based on Independent Data
{X1, X2, … , Xm} = system data (mean X )
{Y1, Y2, … , Yn} = simulation model data (mean Y )
We can construct a (1 - )% confidence interval for  = X - Y or test the null hypothesis H0: X = Y.
Paired t-interval If n = m, let W = X – Y, then  = W = X - Y

21. Example
We want to construct a 90% confidence interval for  = X - Y using the paired t-interval with n = 10 data.
So the difference is statistically significant at level  = 0.10.