1 Validation & Verification Chapter 10

2 VALIDATION & VERIFICATION Very Difficult Very Important Conceptually distinct, but performed simultaneously You must be sure your model is correct Your client must be confident that your model is correct Should be an integral part of model building

3 VALIDATION Goals –Produce a model that represents system behavior closely enough to be a substitute for the system for experimentation Analyzing & predicting performance –Increase credibility of model to managers & decision makers

4 DEFINITIONS Verification –Ensures that the model developed is correctly implemented in the software Validation –Ensures that the model accurately represents the real- world system

5 Validation & Verification Process An integral part of model design & implementation process – not separate Most methods are informal or heuristic in nature Experience in model development, simulation programming, & the system are essential

6 MODEL BUILDING 1.Observation of real system 1.Collect data 2.Talk to members of system 2.Construct conceptual model 1.Assumptions about components & structure of system 2.Hypothesis – values of input parameters 3.Implement operational model 1.Usually using simulation software **Not linear process, iterative!!

7 SUGGESTIONS FOR VERIFICATION 1.Operational model checked by simulation software expert – not developer 2.Flow diagram for each possible action 3.Examine output for reasonableness under various inputs – use wide variety of output statistics 4.Print input at end of run to ensure stability 5.DOCUMENTATION!!! 6.Ensure animation of model is correct

8 SUGGESTIONS FOR VERIFICATION 7.Use debugger of interactive run controller (IRC) – advantages 1.Can monitor simulation progress & display results 2.Can focus on single line or process 3.Can observe model components & variables 4.Can pause; reassign values 8.GUI – always recommended ** Basic Software Engineering Principles

9 Suggestion 3 Examine output for reasonableness –Calculate expected results –Vary input –Ask users to review Examples Suppose multiple servers & only look at throughput. Maybe too many went to one server & too few to the other. If priority queue, are they actually processed in correct order.

10 Suggestion 3 (cont’d.) Utilize standard output from simulation environments Current Count & Total Count are important variables Consider predictions –Mathematical (e.g. Utilization) –Experts Perform a Trace –Snapshots –By hand

11 CALIBRATION & VALIDATION Validation – comparing model to system Calibration – iterative process of comparing model to real system & adjusting the model – repeat! Comparisons –Subjective – experts review –Objective – use of data & results

12 VALIDATION Never truly completely validated –Model never totally represents the real system Be sure model is not “fit” to one set of data

13 3-Step Approach to Validation Naylor & Finger [1967] 1.Build a model with high face validity 2.Validate model assumptions 3.Compare model I/O transformations to corresponding I/O transformations for the real system

14 Step 1. FACE VALIDITY Construct a model that seems valid to the users/experts knowledgeable with system Include users in calibration – builds perceived credibility Sensitivity Analysis – change one or more input value & examine change in results – Are results consistent with real system? –Choose most critical variables to reduce cost of experimentation

15 Step 2. Validation of Assumptions 2 categories of assumptions –Structural assumptions –Data Assumptions Structural Assumptions –Involves how system operates –Includes simplifications & abstractions of reality Data Assumptions –Based on data collection & statistical analysis

16 Step 2. Validation of Assumptions (contd) Review – Analysis of Data –Identify probability distribution –Estimate parameters of distribution –Perform goodness-of-fit test Chi Square, Kolmogorov-Smirnov tests Test homogeneity of data –Are two independently collected sets of data come from the same parent population?

17 Step3. Validating I/O Transformations Ultimate Test of a Model –Ability to predict the future behavior of the real system Model viewed as an I/O Transformation Can also us historical data to test prediction ability Note: If main purpose of simulation changes, model should be revalidated

18 Step 3. Validating (cont’d) Models are often used for comparing alternate system designs –Minor changes in parameters IA rate, # servers –Minor change in statistical distribution –Major change in logical structure of subsystem Queue discipline –Major design change Manual to automated system

19 Using Historical Input Data An alternative to randomly generated data – don’t mix different data sets File, Spreadsheet, or Database –{A1, A2,…,An} & {S1, S2,…Sn} –Feed data into the FEL Compare output to what happened in the real system May be able to use technology to collect historical data for use

20 I/O Validation – Turing Test What is the Turing Test? Generate 5 “fake” reports from simulation & mix with 5 real reports; ask experts if they can distinguish fake from real If cannot, then pass Turing Test!

21 Validation Techniques In order of cost-to-value ratio – Van Horn (1969, 1971) 1. Develop model with high face validity by including experts, previous research, studies, observation, experience 2. Test input data for homogeneity, randomness, goodness-of-fit 3. Turing test – use knowledgeable people

22 Validation Techniques (cont’d) 4. Compare model & system output using statistical tests 5. After model development, collect new data & repeat steps 2 to 4 6. Build new system or redesign old one, collect data on new system & use to validate model (not recommended) 7. Do little or no validation. Implement. (not recommended)

23 Conclusion Do not become obsessed with validation & verification to the detriment of progress; causing excessive cost Modeler should select techniques most valuable and appropriate to the particular system being modeled and the goals of the project Validate & Verify to assure Accuracy & Credibility