1. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Progress to date on NMSG 031 The Cost Effectiveness of Modelling and Simulation A Presentation by Gary Summons (QinetiQ) and Lt Cdr Alastair Macdonald (MOD)

2. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Introduction/Background: The use of modelling and simulation (M&S) has grown rapidly Continued growth will depend on M&S being seen as cost effective and adding value. Programmes generally make a specific business case for any M&S outlay and there is a need for generic business case guidance. By identifying non cost effective application areas, the barriers preventing the proliferation of M&S will be identified providing an indication of future research priorities.

3. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Aims Provide a framework to collect evidence associated with the cost effective application of M&S based upon the NATO procurement process: –Look at how the use of M&S can be categorised and mapped to the NATO process and what measures of effectiveness should be documented. –Produce a report discussing the cost effectiveness associated with the exploitation of M&S and providing evidence to support these arguments where it exists. This presentation will provide an insight in the work to date that the member countries have undertaken.

4. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Members of the NMSG –Canada –France –Germany –Netherlands –UK –US

5. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 NATO Procurement Process In order to place information gathered into a useful context it was decided to use the procurement process as a framework. –Allows end users to see both the applicability of M&S as well as its Cost Effectiveness and relate it to their particular issues Each Nation has it’s own “flavour” of the procurement process it was therefore agreed to adopt the NATO instantiation and map each individual country’s to it –Each member country produced it’s own mapping following consultation with appropriate home nation procurement specialist –While it did not provide a “one to one” mapping for all countries there was a broad alignment –Key to this activity was an understanding of decision points and outputs required at each stage

6. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 NATO Procurement Process Mission need evaluation Pre-Feasibility Feasibility Project definition Design Production In-Service Mission need evaluation Pre- feasibility Feasibility Project definition Design Production In service

7. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 NATO Procurement Process Example showing the NATO process with UK’s mapped Mission need evaluation Pre- feasibility Feasibility Project definition Design Production In service Pre - concept Manufacture Demonstration Assessment Concept UK In service Disposal

8. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Modelling and Simulation (M&S) Classification A way to provide common classification of M&S use is required 3 separate views of this were investigated –Application –Category –Generic use These classifications were then combined in a matrix with the NATO procurement process to provide an indication of M&S applicability

9. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Defence Applications for M&S Concept Development –Requirements gathering –Doctrine development (concepts) Procurement (Capability) Development –Supporting procurement process –Test and evaluation Warfighting Development –Mission rehearsal –Training and education Research

10. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 M&S Categories Component level modelling Individual/team virtual environment Team/collective enviroments Constructive platform level interactive Constructive aggregate level interactive Constructive closed analytical model LVC (Virtual Simulation with constructive, platform) Pure visualisation Infrastructure and services The group adopted this series of M&S categories, in doing so it recognises that there are many other ways this could have been “sliced”

11. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 NL Mission need evaluation Pre feasibilityFeasibilityProject definitionDesign and development ProductionIn-service Component level modelling Individual/team virtual environment Team/collective enviroments Constructive platform level interactive Costructive aggregate level interactive Constructive closed analytical model Virtual Simulation with constructive, platform Pure visualisation Infrastructure and services Example of Procurement Process vs M&S Categories Key Used oftenUsed occasionallyUsed infrequently

12. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Generic Uses The group also elected to look at generic uses of M&S This approach allowed non M&S experts to evaluate the benefits of M&S to the procurement process It is intended that both this and the M&S categories will point at the same underpinning evidence but will present it in different ways

13. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Generic Uses Analysis (of alternative) cost performance trades (Systems) Engineering Experimentation and Concept Development Training Test Exercise Operation

14. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Mission need evaluation Pre feasibilityFeasibilityProject definitionDesign and development ProductionIn-service Analysis (of alternative) cost performance trades (systems) engineering Experimentation and Concept Development Training Test Exercise Operations Key Good cost saving Some costing saving Little or no cost saving Procurement Process vs Generic Uses

15. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Establishing the case for M&S We can determine cost effectiveness by weighing the costs associated with M&S against the benefit accrued However determining the contributing factors to cost is not quite as simple Cost needs to be considered with the following in mind: –Sunk costs, those which have already been invested –Costs incurred whether M&S is employed or whether a more traditional approach is used –Costs not incurred because M&S is used –Costs only incurred because M&S is employed In some cases the use of M&S will need consideration under one or more elements i.e. the re-use of a model will have an element of sunk but most likely incur a cost associated with it’s new use

16. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Establishing the case for M&S Benefits are more easily identified and can be classified as: –Cost avoidance –Risk management –Quality of end product –Reduction in manpower costs –Reduction in Time –Reduction of Environmental impact –Increased capability –Reduction in High value asset dependency

17. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Establishing the case for M&S Once the evidence for cost and benefit has been collected it will be associated with the particular category of M&S, Generic use and Procurement phase We will inform the factors associated with the cost effective application of M&S We will make available the evidence gathered in a coherent fashion so that decision makers can make evidence based judgements on the applicability of M&S to their particular situation Based on this evidence it will be possible to identify areas where the application of M&S is cost effective and those areas it is not

18. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Evidence tool To be useful all the evidence gathered needs to be readily accessible to end users and decision makers A simple tool is required which will both present the existing evidence and aid the collection of new evidence The tool must be able to present the information from the different views previously described: –Stages of procurement process –Different categories of M&S –Different Generic Uses

19. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Example of Query in Evidence tool

20. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Anticipated Results Framework for cost effectiveness evaluation and comparison across projects/nations More qualitative than quantitative evidence Populated evidence schema No general rule (one size does not fit all) difficult to generalise from specific examples

21. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Early Conclusions Difficulty in collecting and analyzing data Quantifiable evidence hard to source No magic bullet solution for Business Case This work provides a framework for cost effect analysis Template for the continued gathering of evidence Allows comparison of cost effectiveness across different projects/national boundaries Problems identifying ‘sunk’ costs Cost of non-M&S approach vs M&S approach i.e. costs are still incurred if M&S not used

22. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 Anticipated Recommendations Develop the Evidence Tool schema as a web based solution on the NATO web Nations to continue populating schema as evidence becomes available Further work needed on “sunk” vs specific costs Further work required on Metadata

23. Symposium on "The Effectiveness of Modelling and Simulation” MSG-031 Briefing on Progress so farPoland, 13 October 2005 NMSG 031 The Cost Effectiveness of Modelling and Simulation Gary Summons gnsummons@qinetiq.comgnsummons@qinetiq.com Lt Cdr Alastair Macdonald alistair.macdonald478@mod.ukalistair.macdonald478@mod.uk