Building Valid, Credible, and Appropriately Detailed Simulation Models Chapter 5 Building Valid, Credible, and Appropriately Detailed Simulation Models

Outline Introduction and definitions Guidelines for determining the level of model detail Verification of simulation computer programs Techniques for increasing model validity and credibility Management’s role in the simulation process Statistical procedures for comparing real-world observations and simulation output data 2019/4/28 Prof. Huei-Wen Ferng

Verification vs. Validation Verification is concerned with determining whether the conceptual simulation model has been correctly translated into a computer “program”. Validation is the process of determining whether a simulation model is an accurate representation of the system. 2019/4/28 Prof. Huei-Wen Ferng

Perspectives on Validation If a simulation model is “valid”, then it can be used to make decisions about the system. The ease or difficulty of the validation process depends on the complexity of the system being modeled and on whether a version of the system currently exists. A simulation model of a complex system can only be an approximation to the actual system. A model that is valid for one purpose may not be for another. 2019/4/28 Prof. Huei-Wen Ferng

Credibility and Accreditation Credibility is supported by the manager and other key project personnel. Note that credible is not necessarily valid. Accreditation is an official determination that a simulation model is acceptable for a particular purpose. 2019/4/28 Prof. Huei-Wen Ferng

Things Help Establish Credibility The manager’s understanding and agreement Demonstration that the model has been validated and verified The manager’s ownership of and involvement with the project Reputation of the model developers 2019/4/28 Prof. Huei-Wen Ferng

Issues of Accreditation Verification and validation that have been done Simulation model development and use history Quality of data that are available Quality of the documentation Known problems or limitations with the simulation model 2019/4/28 Prof. Huei-Wen Ferng

Timing and Relationship 2019/4/28 Prof. Huei-Wen Ferng

Validation and Output Analysis Validation should be contrasted with output analysis. Output analysis is a statistical issue concerned with estimating a simulation model’s true measures of performance. 2019/4/28 Prof. Huei-Wen Ferng

Validation and Output Analysis (Cont’d) Validation is concerned with the second term. Output analysis is concerned with the first term. 2019/4/28 Prof. Huei-Wen Ferng

Outline Introduction and definitions Guidelines for determining the level of model detail Verification of simulation computer programs Techniques for increasing model validity and credibility Management’s role in the simulation process Statistical procedures for comparing real-world observations and simulation output data 2019/4/28 Prof. Huei-Wen Ferng

Before the Simulation A simulation practitioner must determine what aspects actually needed to be incorporated; what level of detail; what aspects can be safely ignored. It is rarely necessary to have a 1-1 correspondence between each element of the system and that of the model. 2019/4/28 Prof. Huei-Wen Ferng

Some General Guidelines Carefully define the specific issues to be investigated. The entity in the simulation does not have to be the same as the entity of the corresponding system. Use subject-matter experts (SMEs) and sensitivity analyses to help determine the level of model detail. A mistake for novice is to include an excessive amount of model detail. Do not have more detail in the model than is necessary to address the issue of interest. The level of model detail should be consistent with the type of data available Time and money constraints are a major factor. Use a “coarse” simulation model or an analytic model to identify what factors have a significant impact on system performance. A “detailed” simulation model is then built. 2019/4/28 Prof. Huei-Wen Ferng

Outline Introduction and definitions Guidelines for determining the level of model detail Verification of simulation computer programs Techniques for increasing model validity and credibility Management’s role in the simulation process Statistical procedures for comparing real-world observations and simulation output data 2019/4/28 Prof. Huei-Wen Ferng

Eight Techniques Write and debug the computer program in modules or subprograms. More than one person to review the computer program Run the simulation under a variety of settings. Trace a program. 2019/4/28 Prof. Huei-Wen Ferng

Eight Techniques (Cont’d) The model should be run under simplifying assumptions for which its true characteristics are know or can easily be computed. Observe an animation of the simulation output. Check the input part. Use a commercial simulation package to reduce the amount of programming required. 2019/4/28 Prof. Huei-Wen Ferng

Outline Introduction and definitions Guidelines for determining the level of model detail Verification of simulation computer programs Techniques for increasing model validity and credibility Management’s role in the simulation process Statistical procedures for comparing real-world observations and simulation output data 2019/4/28 Prof. Huei-Wen Ferng

Six Classes to Increase the Validity and Credibility Collect high-quality information and data on the system Interact with the manager on a regular basis Maintain an assumptions document and perform a structured walk-through Validate components of the model by using quantitative techniques Validate the output from the overall simulation model Animation 2019/4/28 Prof. Huei-Wen Ferng

Collect High-Quality Information and Data on the System In developing a simulation model, the analyst should make use Conversations with SMEs A simulation model is not an abstraction developed by an analyst working in isolation Observations of the system Data are not representative of what one really wants to model Data are not of the appropriate type or format Data may contain measurement, recording, or rounding errors Data may be “biased” because of self-interest Data may have inconsistent units 2019/4/28 Prof. Huei-Wen Ferng

Collect High-Quality Information and Data on the System (Cont’d) Existing theory Relevant results from similar simulation studies Experience and intuition of the modelers 2019/4/28 Prof. Huei-Wen Ferng

Interact with the Manager on a Regular Basis -- Benefits As the simulation study proceeds and the nature of the problem becomes clearer, this information should be conveyed to the manager. The manager’s interest and involvement in the study are maintained. The manager’s knowledge of the system contributes to the actual validity of the model. The model is more credible since the manager understands and accepts the model’s assumptions. 2019/4/28 Prof. Huei-Wen Ferng

Assumptions Document An overview section Detailed descriptions of each subsystem, in bullet format, and how these subsystems interact. What simplifying assumptions were made and why. Summaries of data Sources of important or controversial information 2019/4/28 Prof. Huei-Wen Ferng

Structured Walk-Through A structured walk-through will increase both the validity and credibility of the simulation model. The structured walk-through of the assumptions document might be called conceptual model validation. 2019/4/28 Prof. Huei-Wen Ferng

Validate Components of the Model by Using Quantitative Techniques Sensitivity analysis is used to determine which model factors have a important impact on the desired measures of performance. Examples of factors including The value of a parameter The choice of a distribution The entity moving through the simulated system The level of detail for a subsystem What data are most crucial to collect 2019/4/28 Prof. Huei-Wen Ferng

Validate the Output from the Overall Simulation Model The idea of comparing the model and subsystem output data for the existing system might be called results validation. 2019/4/28 Prof. Huei-Wen Ferng

Animation An animation can be an effective way to find invalid model assumptions and to enhance the credibility of a simulation model. 2019/4/28 Prof. Huei-Wen Ferng

Outline Introduction and definitions Guidelines for determining the level of model detail Verification of simulation computer programs Techniques for increasing model validity and credibility Management’s role in the simulation process Statistical procedures for comparing real-world observations and simulation output data 2019/4/28 Prof. Huei-Wen Ferng

Some of Responsibilities of the Manager Formulating problem objectives Directing personnel to provide information and data to the simulation modeler and to attend the structured walk-through Interact with the simulation modeler on a regular basis Using the simulation results as an aid in the decision-making process 2019/4/28 Prof. Huei-Wen Ferng

Outline Introduction and definitions Guidelines for determining the level of model detail Verification of simulation computer programs Techniques for increasing model validity and credibility Management’s role in the simulation process Statistical procedures for comparing real-world observations and simulation output data 2019/4/28 Prof. Huei-Wen Ferng

Three Approaches Inspection approach Confidence-interval approach based on independent data Time-series approaches 2019/4/28 Prof. Huei-Wen Ferng

Inspection Approach Suppose that are observations from a real-world and that are output data from a corresponding simulation model. The difficulty with this (basic) inspection approach is that each statistic is essentially a sample of size 1. Example 5.34 illustrate the danger of using inspection. 2019/4/28 Prof. Huei-Wen Ferng

Inspection Approach (Cont’d) 2019/4/28 Prof. Huei-Wen Ferng

Inspection Approach (Cont’d) Correlated inspection approach 2019/4/28 Prof. Huei-Wen Ferng

Inspection Approach (Cont’d) 2019/4/28 Prof. Huei-Wen Ferng

Confidence-Interval Approach Suppose we collect m independent sets of data from the system and n independent sets of data from the model. Let Xj be the average of the observations in the jth set of system data Let Yj be the average of the observations in the jth set of model data Two approaches: paired-t (see Ch 10.2) and Welch. (see Example 5.38) Two difficulties: it may require a large amount of data; it provides no information about the autocorrelation structures. 2019/4/28 Prof. Huei-Wen Ferng

Time-Series Approaches A time series is a finite realization of a stochastic process Requires only one set of each type of output data May yield information on the autocorrelation structures of two output processes Examples see Ch 9. 2019/4/28 Prof. Huei-Wen Ferng