Presentation on theme: "Why is this hard? Complicating Factors Emergence of LVC"— Presentation transcript:
0 Measuring the Appropriateness of Live and Simulation Dr Paul HubbardDRDC OttawaPresented by:Nacer AbdellaouiOctober 13, 2011For NATO MSG-087: Enhance or Replace – Finding the Right Live vs. Synthetic BalanceBERN - SWITZERLANDThankyou for opportunity to present the work.Introduce author and presenter.
1 Why is this hard? Complicating Factors Emergence of LVC OutlineWhy is this hard? Complicating FactorsEmergence of LVCIndicators for when to use Live vs SimulationImplications and Conclusions-Nations are facing difficult decisions about how and when to use simulation.-Spectrum of applications: R&D, concept development, acquisition support, test and evaluation, training, mission rehearsal, operations.-we are interested in all, but the contents of this paper were in the context of the use of simulation for experimentation in support of concept development.- Main part of papers is # 3. A list of 30 ‘indicators’1
2 Why is it hard to decide between L, V and C? More options than before on how to do experimentation and trainingLack of knowledge about capability of virtual and constructiveEvolving simulation technology (sophistication)Changing cost estimates, initial investments and maintenance costsCultural reliance on live exercisesV-Speaks for itself.-5 issues. There are others.C
3 Emergence of LVCLVC offers a taxonomy of M&S (see the excellent wikipedia page)Traditionally, simulation supports live, e.g. mission rehearsals, Hardware-in-the-loopAppears to be a good marriage for an experimental campaign:Live has high validitySimulation has high reproducibilityOffers interesting possibilities such as augmented realityV or C?LVC, not a new termEmphasize wikipedia page – it really is goodEvolution of the use of simulaiton (Bullet 2)Why it looks good in an experimental campaign …. We should spiral and alternate as needed …. Also flexibility, adaptability on which tool to use.
4 L or V? Along the lines of Donald Rumsfeld*: PhilosophyAlong the lines of Donald Rumsfeld*:simulation is good at turning known unknowns into known knowns,simulation is not good at turning unknown unknowns into known unknowns,simulation is a good tool for optimization questions, to find the answer to that question, what are ideal parameters for a piece of equipment or within a defined tactic or even more broad questions regarding how much a new potential acquisition will improve a common operating picture.simulation is not good at turning unknown unknowns into known unknowns, that is: simulation doesn’t help us discover what we don’t know. This is because the design and build of the simulation itself embodies much of the domain knowledge surrounding the experimental question, and if we don’t have that knowledge originally, it is hard to build a simulation that will magically provide it. It is difficult to provide the flexibility in a simulation at run-time, for human planners, or machine computation, to discover novelty.SO: know when to use sim and when not to: for concrete experiments, yes, for brainstorming and free-thinking, maybe notHOWEVER: Things are changing and simulation is becoming better at the brainstorming as well!L or V?*Rumsfeld, D. Press conference by US Secretary of Defence at NATO HQ, 6 June 2002,
5 Indicators to support selection of LVC A workshop at DRDC Ottawa sought to answer:How should live and SE experimentation be coordinated?Developed 32 indicators with name, relevant question, and suitabilityIndicator of UtilityID #Questions/Issues with respect to the MeasureComments upon Dominant SuitabilityVery suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL)Iteration Ease1Are multiple iterations required for the experiment?SE experiments are ideally suited, subject to computational complexity, to multiple iterations. For Live experiments this is challenging due to uncontrollable events. – VSEHow to use the table:-You have an experiment.-Read the list, and for an indicator decide if it is important consideration.-If it is, then look at the suitability for guidance on what to use (e.g. VSE means guidance is to use sim)CAVEAT:-The suitability has not been validated: only the opinion of 12 scientists.-Also has the context of an experimental trial with multiple platforms (ALIX)
6 How to use the following table of indicators If indicator and question are of high value to your experiment, then the assessment indicates a recommended use, i.e. VSE implies simulation more appropriate.Caveat: assessments were made in context of a large trial, rather than a tightly controlled experiment. The table itself needs to be validated.MORE CAVEATS:Before drawing conclusions from Table 1, it is important to note the context in which it was developed. First, the table is clearly only a preliminary discussion of these indicators. Further analysis is needed to increase the confidence in the assessed suitability. Furthermore, the table represents the consensus of a dozen practitioners with a specific concept of what constitutes an ‘experiment’. The workshop followed a live trial to assess the value of a new sensor platform, and required the coordination of a large number of other platforms. This is distinctly different from a tightly controlled experimental facility. Table I provides a qualitative discussion of topic in the context of a large military trial, with all the biases that implies. The discussion should be of value at the conceptual stage of experimental design, and may indicate components of a live trial that are better suited to simulation.
7 Indicators to support selection of LVC Indicator of UtilityID #Questions/Issues with respect to the MeasureComments upon Dominant SuitabilityVery suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL)Iteration Ease1Are multiple iterations required for the experiment?SE experiments are ideally suited, subject to computational complexity, to multiple iterations. For Live experiments this is challenging due to uncontrollable events. – VSEControllability2To what extent do external variables or events need to be controlled?SE trials can be completely scripted, but when there are humans in the loop, i.e. virtual rather than constructive simulation, branching in the scenario execution still occurs. Live trials attempt to follow a script, but are much more susceptible to uncontrollable events such as weather can affect scripts. – BCredibility3Is it important for the results and conclusions to appear credible by decision-makers?If both experiments are successful, results of Live will be seen as more credible. If both experiments fail then failure for the SE experiment will generally be viewed as a problem in design/implementation or fidelity; whereas failure in a Live experiment is generally attributed to issues outside of experimental control (e.g., weather) and results will not be viewed as pessimistically. Staff may choose live experiments for this reason. – L-NOT THE FULL LIST: Only excerpts.
8 Indicators to support selection of LVC Indicator of UtilityID #Questions/Issues with respect to the MeasureComments upon Dominant SuitabilityVery suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL)Ground Truth5Is ground truth data required to support evaluations or calculation of metrics in the experiment? (note this is not a validity question, but simply the availability of the data deemed “ground truth”.)Ground truth is generally considered available for SE, when relative measures are needed. All data is considered available in principle either through data recording or playback. Ground truth data may not be available as readily for complicated Live trials that include multiple platforms. – SEFidelity6Does the experiment need to match reality?Considered variable for SE – though not always necessary, if, for instance, the experiment is based on a fictitious future world with conceptual systems. Increasing the fidelity requires more modelling, cost and effort. Live experiments are assumed to be real and therefore maximum fidelity. – VLCost7What is the cost comparison of an SE vs. Live trial?For total costs of a single experiment, these are assessed as equal for SE and Live, due to the potentially high development cost for SE experiments. – B.However, for subsequent repetitions and when re-use in another trial is considered, there is an advantage to SE. – VSESpend time on these. Some are controversial.Value is in the list – not necessarily the rating, which each experimenter should confirm for thmesleves.Like a methodology for the decision making.
9 Indicators to support selection of LVC Indicator of UtilityID #Questions/Issues with respect to the MeasureComments upon Dominant SuitabilityVery suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL)Repeatability8Is it important that a repeated version experiment give identical results? Or slight variations (as in Monte Carlo simulation)?Computer components can be repeated deterministically if identical results are needed, which is almost impossible in Live experiments – VSE, However, when controlling variables, live trials can result in only slight variations trial-to-trial. – SESafety10Is human safety and risk a key requirement for the experiment?Assessed as distinct advantage to SE, both in safety to participants and the ability to test risky operational scenarios. – VSEEthics11Which of live experiments or simulation enables assessment of a broader ethical spectrum?The live trial spectrum is limited, e.g. nuclear effects or explosive echo ranging cannot be tested easily. Ethical dilemmas can be posed to participants as easily in an SE experiment. – VSEEnvironmental Impact12Which form of experiment has lower environmental impact?For SE, impact comes from power consumption for manufacturing and at run-time, as well as obsolescent equipment waste. This is considered minimal in comparison to Live trials, that may damage ecosystems and generate industrial-scale debris. – VSE
10 Indicators to support selection of LVC Indicator of UtilityID #Questions/Issues with respect to the MeasureComments upon Dominant SuitabilityVery suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL)Collateral Training15Is there ancillary training that was not designed in experiment?Assumed unlikely in a SE, apart from machine learning, whereas highly likely mission-relevant training occurs within Live trial. – LTime Compress-ibility16Is it important to compress time in this experiment? i.e. skip long transits or operational delays.Not possible in Live trial – SE is well suited. – VSEValidation22To what extent is the experiment valid, or valid to a given specification?For SE, validation can only be done based on an input specification. Matching to reality requires SME assessment, or computational comparison to live experimental data. Live trials are considered “real” so deemed valid for that moment in time and experimental circumstances. – VLObsolescence of Result26How long will the results be of utility?More dependent on the content of the experiment. – B
11 Indicators to support selection of LVC Indicator of UtilityID #Questions/Issues with respect to the MeasureComments upon Dominant SuitabilityVery suitable for SE (VSE), Suitable to SE (SE), Equal suitability between SE and Live (B), Suitable to Live (L), Very suitable to Live (VL)Observability29How hard is it to observe (key) events?Due to availability of ground truth data, observation of key events may be done with simple code additions. In live experiments, key events may be obscured unless explicitly accounted for. VSEData Quality30How good is the data in terms of format, persistence, and coverageMay be easier to plan for and conceptualize in an SE trial. Also, there is the restart potential to restart SE experiments with additional data recording. In live trials, you “get what you get”, but if well planned, can lead to the ideal data set. – BSensitivity to Fraud32To what extent can the experiment be faked or misinterpreted?Assumed easier to manufacture results in SE trial. Also inadvertent manipulation of SE and results possible. – VL
12 Discussion and Implications Cultural preference for live experiments / training still exists - perceived credibility / validity / acceptability all assess in favour of LIVELive experiments / training already perceived as more expensive.Validity and reproducibility are polaritiesCollateral training in a live military exercise not to be undervalued.-1 I think everybody accepts this and it won’t go away, however look at commercial aviation and high levels of training in simulators.-2 It used to be the start-up cost of hardware and software made sim more expensive, but that is already not the case. Mainly because the expertise now exists. Capacity is still and issue.-3 Polarity is an issue that won’t go away and needs to be MANAGED. (as opposed to a ‘problem’ that can be solved once and done with). Of course there is another topic: importance (or relevance) In physics, when you have all three, then you get a nobel prize.-4 Yes, you want to train sleeping in a tent in the rain at the same time as figuring out if the new unmanned aircraft works right. Big picture of employment is being tested.
13 Sims are more sophisticated, Fuel and manufacturing costs increasing ConclusionA score-card approach to support the decision of simulation vs live was presentedThis will remain a tricky question for some time because the decision space is evolving:Sims are more sophisticated,Fuel and manufacturing costs increasingParting Thought:Should LVC really be LVCA? A=Autonomous = “M&S involving simulated people working real equipment.”Score-card appoach: a list with values to be filled in, then weighted sum the list to make a decision.This should be just one tool to support the decision. Availability of existing sims, and planned exercises, for instance would be other factors.The overall trend will be to more sim in the future (guess).Parting thought is from the Wikipage. Think of quandrants:Synthetic person, synthetic platform: ConstructiveLive person, synthetic platform: VirtualLive person, live platform: LiveSynthetic person, Live platform: ? Autonomous system!!Thankyou and questions can be answered or forwarded to author.