1. Teaching an Advanced Simulation Topic Verification and Validation of Simulation Models Stewart Robinson School of Business and Economics WSC 12, Berlin

2. Develop an understanding of the concepts of verification, validation and confidence in a model Understanding some of the methods that can be used in V&V Session Aim Aimed at:  Specialists: undergraduate and graduate students on a simulation course; industrial training in simulation  Management students: e.g. MBA

3. Session Outline  Define V&V  V&V in the modelling life-cycle  Difficulties in performing V&V  Impossibility of validating a model!  (Techniques of V&V)  Role-play illustrating V&V

4. Verification: The model design (conceptual model) has been satisfactorily converted into a computer model Validation:The model is sufficiently accurate for the purpose at hand Verification and Validation

5. V&V in the Modelling Process Real world (problem) Solutions/ understanding Conceptual model Computer model Conceptual modelling Model coding Experimentation Implementation Solution validation Experimental validation Conceptual model validation Verification Black-box White-box validation Data validation

6. Conceptual Model Validation: determining that the content, assumptions and simplifications of the proposed model are sufficiently accurate for the purpose at hand. Data Validation: determining that the contextual data and the data required for model realisation and validation are sufficiently accurate for the purpose at hand. White-Box Validation: determining that the constituent parts of the computer model represent the corresponding real world elements with sufficient accuracy for the purpose at hand. Black-Box Validation: determining that the overall model represents the real world with sufficient accuracy for the purpose at hand. Experimentation Validation: determining that the experimental procedures adopted are providing results that are sufficiently accurate for the purpose at hand. Solution Validation: determining that the results obtained from the model of the proposed solution are sufficiently accurate for the purpose at hand.

7. Implications for V&V Verification and Validation needs to be performed continuously throughout the modelling process. Key point Since the modelling process is iterative in nature, so too verification and validation need to be iterated and reiterated from the point of model conception to the implementation of the results.

8. Difficulties in Performing V&V 1. There is no such thing as general validity 2. There may be no real world to compare against 3. Which real world? 4. Often the real world data are inaccurate 5. There is not enough time

9. Implications for V&V It is impossible to validate a model! Model validation is a process of increasing confidence in a model – to the point where there is a willingness to use it for decision-making. When validating a model the aim is to demonstrate that the model is in fact invalid. The more tests that can be performed in which it cannot be proved that a model is invalid, the greater the confidence that can be placed in that model. Key points

10. Natland Bank Natland Bank: Planning a New Bank Branch Question: How many ATMs are required (95% of customers queue for less than 3 minutes)? ATM 1 ATM 2 Queue Customers (Arrival rate) Service time Simplifications: 1. No breakdowns of ATMs 2. No customers balk or leave Proposed model

11. Natland Bank: Confidence Check Conceptual Model Validation High Medium Low

12. Natland Bank: Data Time of dayAverage number of arrivals 9:00-10:00110 10:00-11:0095 11:00-12:00140 12:00-13:00165 13:00-14:00205 14:00-15:00145 15:00-16:00160 16:00-17:00190 Customer Arrivals

13. Natland Bank: Data Service type Service time (seconds)% of customers C3040 B2010 T308 C, B6025 C, T4510 B, T402 C, B, T755 Service Time

14. Natland Bank: Confidence Check Data Validation High Medium Low

15. Natland Bank White-Box Validation (also performed in verification)  Watch the model animation: face validation  Inspect the model code: correct entry of data  Extreme value testing: very high service time

16. Natland Bank: Confidence Check White-Box Validation High Medium Low

17. Black-Box Validation Comparison with the real system Real system I R O R Simulation model I S O S H 0 : If I S =I R then O S  O R

18. Black-Box Validation Comparison with other models Alternative model I A O A Simulation model I S O S H 0 : If I S =I A then O S  O A

19. Black-Box Validation Comparison with other models Accuracy derived from complexity Simulation Alternative model  Extreme approach is to make the simulation deterministic

20. Natland Bank Black-Box Validation: Comparison with Another (Simpler) Model Deterministic model comparison: Arrival rate = 100/hour 2 tellers: service time = 1 minute Customers served/hour = 60 x 2 = 120 Expected teller utilisation = 100/120 = 83.3%

21. Natland Bank Black-Box Validation: Comparison with Another (Simpler) Model Full model comparison: Mean arrival rate = 157.14/hour 2 tellers: mean service time = 40.45 seconds Mean customers served/hour = 89.00 x 2 = 178.00 Expected teller utilisation = 157.14/178.00 = 88.28%

22. Natland Bank: Confidence Check Black-Box Validation High Medium Low

23. Will you use my model to determine the number of ATMs in the bank?