Presentation is loading. Please wait.

Presentation is loading. Please wait.

Dr Paul Hubbard DRDC Ottawa Presented by: Nacer Abdellaoui October 13, 2011 For NATO MSG-087: Enhance or Replace – Finding the Right Live vs. Synthetic.

Similar presentations


Presentation on theme: "Dr Paul Hubbard DRDC Ottawa Presented by: Nacer Abdellaoui October 13, 2011 For NATO MSG-087: Enhance or Replace – Finding the Right Live vs. Synthetic."— Presentation transcript:

1 Dr Paul Hubbard DRDC Ottawa Presented by: Nacer Abdellaoui October 13, 2011 For NATO MSG-087: Enhance or Replace – Finding the Right Live vs. Synthetic Balance BERN - SWITZERLAND Measuring the Appropriateness of Live and Simulation

2 1 1.Why is this hard? Complicating Factors 2.Emergence of LVC 3.Indicators for when to use Live vs Simulation 4.Implications and Conclusions Outline

3 2 Why is it hard to decide between L, V and C? More options than before on how to do experimentation and training Lack of knowledge about capability of virtual and constructive Evolving simulation technology (sophistication) Changing cost estimates, initial investments and maintenance costs Cultural reliance on live exercises V L C

4 3 Emergence of LVC LVC offers a taxonomy of M&S (see the excellent wikipedia page) Traditionally, simulation supports live, e.g. mission rehearsals, Hardware-in-the-loop Appears to be a good marriage for an experimental campaign: Live has high validity Simulation has high reproducibility Offers interesting possibilities such as augmented reality V or C?

5 4 Philosophy Along the lines of Donald Rumsfeld*: simulation is good at turning known unknowns into known knowns, simulation is not good at turning unknown unknowns into known unknowns, *Rumsfeld, D. Press conference by US Secretary of Defence at NATO HQ, 6 June 2002, http://www.nato.int/docu/speech/2002/s020606g.htm.http://www.nato.int/docu/speech/2002/s020606g.htm L or V?

6 5 Indicators to support selection of LVC A workshop at DRDC Ottawa sought to answer: How should live and SE experimentation be coordinated? Developed 32 indicators with name, relevant question, and suitability Indicator of Utility ID #Questions/Issues with respect to the Measure Comments upon Dominant Suitability Very suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL) Iteration Ease1Are multiple iterations required for the experiment? SE experiments are ideally suited, subject to computational complexity, to multiple iterations. For Live experiments this is challenging due to uncontrollable events. – VSE

7 6 How to use the following table of indicators If indicator and question are of high value to your experiment, then the assessment indicates a recommended use, i.e. VSE implies simulation more appropriate. Caveat: assessments were made in context of a large trial, rather than a tightly controlled experiment. The table itself needs to be validated.

8 7 Indicators to support selection of LVC Indicator of UtilityID #Questions/Issues with respect to the Measure Comments upon Dominant Suitability Very suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL) Iteration Ease1Are multiple iterations required for the experiment? SE experiments are ideally suited, subject to computational complexity, to multiple iterations. For Live experiments this is challenging due to uncontrollable events. – VSE Controllability2To what extent do external variables or events need to be controlled? SE trials can be completely scripted, but when there are humans in the loop, i.e. virtual rather than constructive simulation, branching in the scenario execution still occurs. Live trials attempt to follow a script, but are much more susceptible to uncontrollable events such as weather can affect scripts. – B Credibility3Is it important for the results and conclusions to appear credible by decision-makers? If both experiments are successful, results of Live will be seen as more credible. If both experiments fail then failure for the SE experiment will generally be viewed as a problem in design/implementation or fidelity; whereas failure in a Live experiment is generally attributed to issues outside of experimental control (e.g., weather) and results will not be viewed as pessimistically. Staff may choose live experiments for this reason. – L

9 8 Indicators to support selection of LVC Indicator of UtilityID #Questions/Issues with respect to the Measure Comments upon Dominant Suitability Very suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL) Ground Truth5Is ground truth data required to support evaluations or calculation of metrics in the experiment? (note this is not a validity question, but simply the availability of the data deemed ground truth.) Ground truth is generally considered available for SE, when relative measures are needed. All data is considered available in principle either through data recording or playback. Ground truth data may not be available as readily for complicated Live trials that include multiple platforms. – SE Fidelity6Does the experiment need to match reality? Considered variable for SE – though not always necessary, if, for instance, the experiment is based on a fictitious future world with conceptual systems. Increasing the fidelity requires more modelling, cost and effort. Live experiments are assumed to be real and therefore maximum fidelity. – VL Cost7What is the cost comparison of an SE vs. Live trial? For total costs of a single experiment, these are assessed as equal for SE and Live, due to the potentially high development cost for SE experiments. – B. However, for subsequent repetitions and when re-use in another trial is considered, there is an advantage to SE. – VSE

10 9 Indicators to support selection of LVC Indicator of UtilityID #Questions/Issues with respect to the Measure Comments upon Dominant Suitability Very suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL) Repeatability8Is it important that a repeated version experiment give identical results? Or slight variations (as in Monte Carlo simulation)? Computer components can be repeated deterministically if identical results are needed, which is almost impossible in Live experiments – VSE, However, when controlling variables, live trials can result in only slight variations trial-to-trial. – SE Safety10Is human safety and risk a key requirement for the experiment? Assessed as distinct advantage to SE, both in safety to participants and the ability to test risky operational scenarios. – VSE Ethics11Which of live experiments or simulation enables assessment of a broader ethical spectrum? The live trial spectrum is limited, e.g. nuclear effects or explosive echo ranging cannot be tested easily. Ethical dilemmas can be posed to participants as easily in an SE experiment. – VSE Environmental Impact 12Which form of experiment has lower environmental impact? For SE, impact comes from power consumption for manufacturing and at run-time, as well as obsolescent equipment waste. This is considered minimal in comparison to Live trials, that may damage ecosystems and generate industrial-scale debris. – VSE

11 10 Indicators to support selection of LVC Indicator of UtilityID #Questions/Issues with respect to the Measure Comments upon Dominant Suitability Very suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL) Collateral Training 15Is there ancillary training that was not designed in experiment? Assumed unlikely in a SE, apart from machine learning, whereas highly likely mission-relevant training occurs within Live trial. – L Time Compress- ibility 16Is it important to compress time in this experiment? i.e. skip long transits or operational delays. Not possible in Live trial – SE is well suited. – VSE Validation22To what extent is the experiment valid, or valid to a given specification? For SE, validation can only be done based on an input specification. Matching to reality requires SME assessment, or computational comparison to live experimental data. Live trials are considered real so deemed valid for that moment in time and experimental circumstances. – VL Obsolescence of Result 26How long will the results be of utility? More dependent on the content of the experiment. – B

12 11 Indicators to support selection of LVC Indicator of UtilityID #Questions/Issues with respect to the Measure Comments upon Dominant Suitability Very suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL) Observability29How hard is it to observe (key) events? Due to availability of ground truth data, observation of key events may be done with simple code additions. In live experiments, key events may be obscured unless explicitly accounted for. VSE Data Quality30How good is the data in terms of format, persistence, and coverage May be easier to plan for and conceptualize in an SE trial. Also, there is the restart potential to restart SE experiments with additional data recording. In live trials, you get what you get, but if well planned, can lead to the ideal data set. – B Sensitivity to Fraud 32To what extent can the experiment be faked or misinterpreted? Assumed easier to manufacture results in SE trial. Also inadvertent manipulation of SE and results possible. – VL

13 12 Discussion and Implications 1.Cultural preference for live experiments / training still exists - perceived credibility / validity / acceptability all assess in favour of LIVE 2.Live experiments / training already perceived as more expensive. 3.Validity and reproducibility are polarities 4.Collateral training in a live military exercise not to be undervalued.

14 13 Conclusion A score-card approach to support the decision of simulation vs live was presented This will remain a tricky question for some time because the decision space is evolving: Sims are more sophisticated, Fuel and manufacturing costs increasing Parting Thought: Should LVC really be LVCA? A=Autonomous = M&S involving simulated people working real equipment.


Download ppt "Dr Paul Hubbard DRDC Ottawa Presented by: Nacer Abdellaoui October 13, 2011 For NATO MSG-087: Enhance or Replace – Finding the Right Live vs. Synthetic."

Similar presentations


Ads by Google