1. David J. Murray-Smith, School of Engineering, Rankine Building, University of Glasgow, Scotland, U.K. Keynote Tutorial: Methods for Testing and Validation of Simulation Models for Engineering Applications E-mail: david.murray-smith@glasgow.ac.uk Keynote Tutorial: Model quality, testing and validation 0

2. The presentation 1.Introduction 2.Model testing and validation issues Uncertainties, modelling errors, testing, verification and validation. Model quality measures and model improvements. 3. Model management Libraries of sub-models and development of generic models. Model version control and model documentation. 4. Educational implications 5. Examples External validation and upgrading of linear and nonlinear helicopter models for handling qualities and flight control studies. 6.Discussion and Conclusions Keynote Tutorial: Model quality, testing and validation 1

3. To provide a basis for design. To assist in human decision making. To explain complex system behaviour. For use within fault detection systems etc.. For simulator development (e.g. for operator training or for engineering development applications). The purpose of models in engineering Keynote Tutorial: Model quality, testing and validation 2

4. 1.Conceptual models allow investigation of performance limitations at an early stage of the design process for normal and abnormal operating conditions. 2.Fully developed and proven models can provide information about key parameter sensitivities and inter-dependencies – useful for design decisions and optimisation. 3.Full models allow virtual prototypes to be created before any hardware prototype is available so identifying necessary design alterations at an early stage, avoiding expensive changes later on. Design benefits with fit-for-purpose models Keynote Tutorial: Model quality, testing and validation 3

5. Models in design: some pointers  “Improved modelling of physical and manufacturing processes will improve our ability to predict the behaviour, costs and risks of future systems, and dramatically reduce the development timescale”. UK Office of Science and Technology, Technology Foresight Panel Report (1995).  “Verification, validation and accreditation” (VV&A).  “Smart procurement” methods.  Concept of “the model as a specification” (as promoted by T.S. Ericsen, US Office of Naval Research). Keynote Tutorial: Model quality, testing and validation 4

6.  Level of model quality necessary is of critical importance for any given application........ an inappropriate model is less than useless as it may delay the project and lead to cost escalation  Balance needs to be found between model accuracy and the cost of developing the model.  Rigorous consideration of model quality is most common in applications involving safety critical issues (e.g. aeronautical engineering, automotive engineering etc.) Levels of model quality Keynote Tutorial: Model quality, testing and validation 5

7. Current Problems with ModelsCurrent Problems with Models  Models are often used with very little systematic testing.  Model documentation is often minimal and is not recognised as a vital part of the model development process.  In many organisations models are passed from project to project and end up being used in ways that were never intended by the original developer of the model. Current problems with models Keynote Tutorial: Model quality, testing and validation 6

8. Example: Hydro-turbine system modelling. Figure from Bryce, G.W., Foord, T.R., Murray-Smith, D.J. and Agnew, P., ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35-44, 1976 Keynote Tutorial: Model quality, testing and validation 7

9. Photographs: © D. J. Murray-Smith Keynote Tutorial: Model quality, testing and validation 8

10. Pipeline geometry Figure from Bryce, G.W., Foord, T.R., Murray-Smith, D.J. and Agnew, P., ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35-44, 1976 Keynote Tutorial: Model quality, testing and validation 9

11. First simplification Figure from Bryce, G.W., Foord, T.R., Murray-Smith, D.J. and Agnew, P., ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35-44, 1976 Keynote Tutorial: Model quality, testing and validation 10

12. Model comparisons (((Real-time approximation: continuous line Non-real-time model: dashed line) Figure from Bryce, Fo Murray-Smith and Agnew, Simulation Councils Proceedings, Vol.6, No. 1, 35-44. Figure from Bryce, G.W., Foord, T.R., Murray-Smith, D.J. and Agnew, P., ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35-44, 1976 Keynote Tutorial: Model quality, testing and validation 11

13. Second approximation F Figure from Bryce, G.W., Foord, T.R., Murray-Smith, D.J. and Agnew, P., ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35-44, 1976 Keynote Tutorial: Model quality, testing and validation 12

14. Model comparisons for second approximation ( Real-time approximation: continuous line Non-real-time model: dashed line) Figure from Bryce, G.W., Foord, T.R., Murray-Smith, D.J. and Agnew, P., ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35-44, 1976 Keynote Tutorial: Model quality, testing and validation 13

15. Site test and simulation comparisons Figure from Bryce, G.W., Foord, T.R., Murray-Smith, D.J. and Agnew, P., ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35-44, 1976 Keynote Tutorial: Model quality, testing and validation 14

16. Model predictions Keynote Tutorial: Model quality, testing and validation 15

17. a) Model quality, uncertainties and modelling errors. b) Testing, verification and validation of models. c) External validation methods. d) Model quality measures in external validation. Part 2: Issues of model testing and validation Keynote Tutorial: Model quality, testing and validation 16

18.  Tests only deal with a small number of cases.  General statements about validity are impossible.  All testing must be carried out in the context of the application and especially the precise range of operating conditions for that application.  Should start from a well-understood case, even if much simplified; then move incrementally to testing for less certain situations for that application.  The more complex the model the harder the problem of quality assessment becomes: measures of model performance become harder to define and visualisation becomes more difficult. Testing of models: fitness-for-purpose Keynote Tutorial: Model quality, testing and validation 17

19.  Establishing the useful range of the dynamic model for a specific application.  Estimating the limits of accuracy of the model (usually both for steady-state and transient conditions) in terms the magnitude of expected errors in model predictions. Aspects of model quality Keynote Tutorial: Model quality, testing and validation 18

20. Sources of errors and uncertainties Incorrect modelling assumptions Errors in a priori information (e,g, model parameter values) Errors in numerical solutions of model equations Errors in experimental procedures and in the measurements used for model testing. Keynote Tutorial: Model quality, testing and validation 19

21. Internal verification and external validation  Internal Verification – establishing that a computer–based model is consistent with the underlying mathematical model. i.e. “Is the simulation model right?”  External Validation – demonstrating that a final (nonlinear)) model is adequate for the intended application. i.e. “Is it the right simulation model?” Note that checks of identified linear models are sometimes referred to as “external verification”. Keynote Tutorial: Model quality, testing and validation 20

22. External validation Need to distinguish between:  Functional validation: where model is assessed in terms of how well it mimics input-output behaviour of the real system.  Physical/Theoretical validation where the model is based on theory and intermediate variables in model and system are compared. Approximations and assumptions are investigated within this process. Keynote Tutorial: Model quality, testing and validation 21

23. Approaches to external validation  Holistic approaches (e.g. subjective opinion of an expert on the real system such as an operator).  Model component approaches (e.g. each sub- system tested independently and compared with corresponding components of the real system). Keynote Tutorial: Model quality, testing and validation 22

24. Methods for external validation  Methods involving direct comparisons of response data from model and real system.  Methods based on system identification and parameter estimation.  Methods involving parameter sensitivity analysis.  Methods based on inverse models and inverse simulations. Whatever method is used, data for external validation must be appropriate for the intended application. Careful experimental design is essential. Keynote Tutorial: Model quality, testing and validation 23

25. Methods for system and model comparisons  Graphical comparisons  Quantitative measures  System identification methods  Expert opinion Different approaches can provide different kinds of insight. Keynote Tutorial: Model quality, testing and validation 24

26.  Plots of simulated and measured responses against an independent variable (often time).  Plots of simulated values against the corresponding measured values (should be 45 degree line).  Different graphical methods may emphasise different aspects of the simulation model performance so there are possible benefits from combining different approaches. Methods of system/model comparison: some examples Keynote Tutorial: Model quality, testing and validation 25

27. Typical time history comparisons Keynote Tutorial: Model quality, testing and validation 26

28. Another time history comparison: a multi-input multi-output case The original version of this figure was published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991 BO-105 helicopter flight test data, DLR SIMH simulation model Keynote Tutorial: Model quality, testing and validation 27

29. Examples of simple cost functions Mean absolute error Mean absolute percent error Weighted error Keynote Tutorial: Model quality, testing and validation 28

30. Theil’s Inequality Coefficient (TIC) Keynote Tutorial: Model quality, testing and validation 29

31. Use of quantitative measures: an example Keynote Tutorial: Model quality, testing and validation 30

32. Polar diagram form of display From Smith, M.I., Murray-Smith, D.J. and Hickman, D., ‘Verification and validation issues in a generic model of electro-optic sensor systems ‘ J. Defense Modeling and Simulation, 4(1), 17-27, 2007 Keynote Tutorial: Model quality, testing and validation 31

33. Issues of identifiability in external validation Test input design is important since inputs must excite the system and model over an appropriate range of frequencies and amplitudes. The concept of identifiability is central to issues of test input design external validation and is thus very important for external validation. Structural identifiability relates to situations where a model may have an excess of parameters so that some specific parameters cannot be estimated uniquely for any possible experimental design (e.g. Bellman, R. and Åström, K.J., Mathematical Biosciences, 7, 329-339, 1970). Structural identifiability is also important for external validation. Keynote Tutorial: Model quality, testing and validation 32

34. Numerical unidentifiability arises when a structurally identifiable model is being used with data that is inappropriate for the application. Numerical identifiability investigated from parameter information matrix M, the related dispersion matrix D and the parameter correlation matrix P. All depend on the sensitivity matrix X where: Inputs may maximise the overall accuracy of all parameter estimates or may be chosen to maximise accuracy of specific parameter estimates. Issues of numerical identifiability and test input design in model validation Keynote Tutorial: Model quality, testing and validation 33

35. Upgrading of simulation models  Following comparison of model and system behaviour usually need to analyse discrepancies and propose upgrades for model.  Changes must be evaluated systematically on a physical basis – with further iterations in the development cycle.  Parametric changes usually considered first, before structure.  Sometimes possible to associate model deficiencies with specific state variables model (e.g. correlation of output error with a state variable may help identify problem source).  Correlation of model errors with derivatives of state variables may suggest that a higher-order description would be more useful.  Optimisation tools (including evolutionary computing methods such as GA and GP) may be useful but should be used along with physical knowledge and understanding of the model. Keynote Tutorial: Model quality, testing and validation 34

36. Part 3: Model management a) Libraries of sub-models and generic models. b) Model version control and model documentation. Keynote Tutorial: Model quality, testing and validation 35

37. Generic and re-usable sub-models  Generally accepted that system design should be based on use of generic descriptions and re-usable sub-models.  Examples of the generic approach may be found in automotive engineering, gas turbines etc. Issues inevitably arise in the external validation of generic models – one approach is discussed in Smith, M.I., Murray-Smith, D.J. and Hickman, D., ‘Verification and validation issues in a generic model of electro-optic sensor systems’ J. Defense Modeling and Simulation, 4(1), 17-27, 2007 Keynote Tutorial: Model quality, testing and validation 36

38. Model documentation and version control  Extra costs of creating good documentation should be more than balanced by the resulting re-usability of models. Version control processes should ensure that changes are fully documented.  Documentation and version control well developed in software engineering field. Same principles should be applied to the model development process. Keynote Tutorial: Model quality, testing and validation 37

39. Items for documentation  Purpose of model and the intended application.  Full description of model and corresponding computer code.  List of all assumptions and approximations used.  Details of all tests carried out on the real system to provide information for model development.  Details of the internal verification process.  Details of the external validation process, with statements about why model was accepted or rejected and information about usable range for model. Keynote Tutorial: Model quality, testing and validation 38

40. Part 4: Educational implications Keynote Tutorial: Model quality, testing and validation 39

41. Modelling and Simulation in Engineering EducationModelling and Simulation in Engineering Education  Engineering students encounter mathematical and computer- based modelling repeatedly in their university education.  Emphasis is most often on development of models from physical principles and on using models/simulations in place of experiments on real systems. Modelling and simulation in engineering education Keynote Tutorial: Model quality, testing and validation 40

42. Modelling and Simulation in Engineering EducationModelling and Simulation in Engineering Education  Issues of model quality and fitness-for-purpose are seldom emphasised in the teaching of modelling and simulation.  Model validation is neglected in education. The teaching of system modelling and simulation should include much more on model validation methods.  Model testing should become second nature for students.  Documentation, model re-use and libraries of models must be given much more emphasis (especially in more advanced teaching). Model quality and testing issues in engineering education Keynote Tutorial: Model quality, testing and validation 41

43. Part 5: Examples Drawn from external validation and upgrading of helicopter models for flight control system design. Keynote Tutorial: Model quality, testing and validation 42

44.  Plant model limitations often impose serious performance limitations within control system, especially in systems with high-performance requirements.  Particularly important to have highly accurate plant models for the part of the frequency range close to the “cross-over” region in the frequency domain. Model limitations in control Keynote Tutorial: Model quality, testing and validation 43

45. Examples from aircraft and helicopter flight control system design There are many well documented aircraft flight control examples illustrating problems of model quality and model limitations in integrated system design. Problems are often identified during initial testing of hardware. These lead to development of improved models and corresponding control design changes. The later in the design cycle these changes have to be made the more costly they are and the greater the delays to the project. Keynote Tutorial: Model quality, testing and validation 44

46. Helicopter model requirements  Must perform well over a defined range of frequencies.  Must perform well over a defined range of manoeuvre amplitudes.  For flight control design model must perform especially well close to open-loop gain and phase cross-over frequencies (as with control applications involving other types of system). Photograph: © D. Murray-Smith Keynote Tutorial: Model quality, testing and validation 45

47. Nominal model: simulation and flight data compared for same test inputs. (Westland Lynx helicopter in 300 ft. quick-hop manoeuvre) From Bradley, R., Padfield, G.D., Murray-Smith, D.J. and Thomson, D.G., ‘Validation of helicopter mathematical models’, Transactions of the Institute of Measurement and Control’, 12(4), 186-196, 1990.

48. SS Common test inputs used for system identification and “external verification” of the identified model (BO-105 flight data) The original versions of these figures were published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991

49. ...........and flight testing Keynote Tutorial: Model quality, testing and validation 49

50. Simulation, identification and “external verification” results (BO-105 flight test data, DLR SIMH model) The original version of these figures were published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991

51. YvYv LvLv NvNv LpLp NpNp LrLr NrNr L δ lat N δ lat L δ ped N δ ped 2ζω 0 Normalised HELISTAB values Values estimated from flight data using system identification methods. Estimated and theoretical parameter values of the identified model for lateral/directional characteristics (SA-330 Puma helicopter flight test data : 80 knots straight and level flight)

52. Assessment of a theoretical nonlinear model for a Puma helicopter Parameter values for two different flight conditions, showing parametric trends from a physically-based nonlinear simulation model (HELISTAB) and the trends in estimates from flight tests involving system identification of separate linearised models for each flight condition. From Bradley, R., Padfield, G.D., Murray-Smith, D.J. and Thomson, D.G., ‘Validation of helicopter mathematical models’, Transactions of the Institute of Measurement and Control’, 12(4), 186-196, 1990. Keynote Tutorial: Model quality, testing and validation 53

53.  Good vehicle models are essential for design of high-bandwidth full-authority active flight control systems.  Published examples show that the achievable performance of flight control systems have, in some cases, been overestimated in initial design studies, usually because of limitations of the flight mechanics model of the vehicle (see e.g. Tischler, M. B. Advances in Aircraft Flight Control Systems, Taylor & Francis, London 1996).  Although control systems can be made robust to compensate for poor accuracy this is usually at the expense of performance. Improved modelling procedures and improved models can offer significant benefits. Otherwise, problems may not be apparent until the flight testing stage, leading to costly redesign, extended flight test programmes and delays in certification. Summary of model quality issues for helicopter flight control system design Keynote Tutorial: Model quality, testing and validation 55

54. Part 5: Discussion and conclusions Keynote Tutorial: Model quality, testing and validation 56

55. Discussion: Model quality in design  The more demanding the design specification the more important is the fitness-for-purpose of models used in design.  More attention needs to be given to the external validation of models for the specific application in question. a) Establishing the useful range of the model. b) Estimating the accuracy of the model within that range.  Model validation is part of the model building process and external validation techniques need to be applied repeatedly.  Models should be retained, maintained and updated throughout the whole life-cycle of the system that they represent. Keynote Tutorial: Model quality, testing and validation 57

56. More attention should be given to the fitness-for-purpose of models used in design, especially for demanding applications. Current methods for external validation are time-consuming and difficult to apply in many situations. More effort should be devoted to improving validation methods. Techniques of version control and rigorous documentation should borrowed from software engineering and applied to the model development process. Re-use of proven models should be made easier and more comprehensive model documentation should be available within model libraries. Issues of model quality should be given far more attention within engineering education. Recommendations Keynote Tutorial: Model quality, testing and validation 58

57. Conclusions  With suitable structure and parameter values and rigorous external validation models that are fit-for-the-purpose of a given application can be developed (iteratively of course).  Good model management can reduce the cost of design and development.  There are no quick answers: a systematic approach is essential, moving incrementally from well-understood cases to less well known situations.  Educational and cultural changes are needed as well as improved management of the modelling, simulation and design processes within most organisations. Keynote Tutorial: Model quality, testing and validation 59