1. Towards Appropriate Selection of Analysis Tools and Methods

2. Overview Research to determine the appropriate tools and methods for effective application within a broad range of analysis categories. Longer term goal to complete whole activity –Summarised later This paper focuses on –categorisation of study and problem types Initial activities description –Discussion on derivation of metrics for assessing the value of study methods and tools vs problem and study types.

3. Aim Overall Aim and Introduction The aim of the whole research is to determine the appropriate tools and methods for effective application within a broad range of analysis categories. The work will –investigate and analyse a broad range of differing analysis study types and –research optimal methods and tools for utilisation within these types. The analysis study types –will be example operational and system analysis studies –investigated to determine the approaches used, the constraints and issues associated with them and the overall success and outcomes.

4. Activities to date Initial Literature Review –OR Society –College Library –MORS Material Methods Handbook Model Taxonomy –SME Contact No evidence of complete approach But –Problem Structuring Methods –Paradigms

5. Problem Types Obeng’s Problem types –Fog – Open –Movie – Semi Open –Quest – Semi Closed –Paint by Numbers – Closed Mike Pidd –Problems –Puzzles –Messes Obeng preferred –More refined

6. –Organisational Level Corps Div Bn –Combat Type High Intensity PK Hybrid –Timescale of Battle Hour Day Days Proposed Factors affecting the problem

7. The Question –Cost Effectiveness Comparison –Force Mix –Battlefield Effectiveness –Capability Gap –Tactics comparison –Numbers The Customer Soft Issues Proposed Factors affecting the problem II

8. Factors affecting the solution Skill set Analysis Timescales Analysis Budget Model Types –Hard –Soft –etc

9. Factors affecting the solution Skill set Defence Environment Analysis within Acquisition and Systems Process Uses Capability Analysis Acquisition Processes Business Case BOI COEIA Operational Support Wargaming SeBA Training Systems Approach Validation

10. Factors affecting the solution – Skill set Tools and Methods OR Techniques Soft Methods Spreadsheets Mathematical Modelling Engineering Modelling Modelling and Simulation Virtual Live Constructive Stochastic Deterministic etc Games Simulations Analytical methods – lanchester etc SEs Model Development Process

11. Factors affecting the solution – Skill set Design of Experiments Analysis and interpretation of results Design of Experimental Studies Confidence Verification Communication and Understanding Model Components Networking/ Distribution of Simulation HCI Protocols Behavioural Representation/Intelligent Systems Data Data Analysis & Databases Combat Sys, Terrain Scenarios Software Development

12. Skill set Level of Skill required for differing roles –Senior Decision Maker –Problem Owner –Study Leader –Analyst –Developer –Player –Project Manager –Technical Manager

13. Measuring how well the method(s) fits the problem Success Criteria - Measuring how well the method(s) fits the problem A. Answer obtained –Within time/budget/other constraints B Answer accepted –Validated C Answer Used D Repeat work

14. Measuring how well the method(s) fits the problem Success Criteria Measuring how well the method(s) fits the problem A. Answer obtained –Within time/budget/other constraints –Scale On budget/time Ahead of budget/time 20% slip 50% slip 100% slip

15. Measuring how well the method(s) fits the problem Success Criteria Measuring how well the method(s) fits the problem B Answer accepted –Validated –By peer/QA review –By acceptance –Accepted after changes

16. Measuring how well the method(s) fits the problem Success Criteria Measuring how well the method(s) fits the problem C Answer Used –By one user –By several –Citations

17. Measuring how well the method(s) fits the problem Success Criteria Measuring how well the method(s) fits the problem D Repeat work –Once –Extension –Frequent

18. Next Steps Study Examples - Drawn from industry and Dstl where possible. For each study –What was done, –what techniques used –resources used –Output measured Study Analysis Studies categorised according to the range of criteria or dimensions, – Problem type –Effort levels available –Timescales –Tools available –Skills available –Audience/Customer –Other Soft issues – political Comparative Analysis of UK MoD Guidelines for conduct of Analysis

19. Outline Evaluation of Study Approaches –Identification of a full range of approaches including tools and methods for potential use within these types of analyses. –Identification of the approaches and the associated tools and methods used within each example study. –Derivation of metrics for assessing the value of each of approaches and the tools and methods in achieving successful outcomes in the studies. Derivation of Optimising Approaches –Derivation of theoretical/initial rule sets for identifying the optimal approaches and tools and methods for use within study categories.

20. Outline Analytical Review of Example Study –Testing of the initial theoretical rule sets against the example studies to determine where non-optimal methods were applied and identification of the improved approaches which might have been employed and the impact these would have had on the study outcome. Test phase using New Studies –Experimental phase using real study examples to develop approaches and test the hypothesised rule sets. Conclusions –Identified optimised approaches including the risk areas and constraints and the ‘region of validity’ of the conclusions.

21. Towards Appropriate Selection of Analysis Tools and Methods