Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Rigorous Subjectivity How will OA survive the “soft” revolution?

Similar presentations


Presentation on theme: "1 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Rigorous Subjectivity How will OA survive the “soft” revolution?"— Presentation transcript:

1 1 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Rigorous Subjectivity How will OA survive the “soft” revolution? Graham L Mathieson Centre for Defence Analysis Defence Evaluation and Research Agency Portsdown West, Fareham, Hants., UK e-mail: glmathieson@dera.gov.uk 15th International Symposium on Military Operational Research RMCS Shrivenham, September 1998

2 2 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Agenda zWhat’s the problem? zProblem space and analysis methods zObjectivity and quantification zAdding rigour to multi-criteria analysis

3 3 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA What’s the problem? Complexity of problem space Breadth of problem space Hard systems modelling

4 4 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Complexity and Scope zJointness and Combinedness zIntegration and digitization zSpectrum of conflict zDuration of operations zMultiplicity of dimensions

5 5 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Methods and Problem Spaces Breadth of problem space Complexity of problem space Hard systems modelling Soft systems analysis Multi-criteria analysis Exploratory modelling

6 6 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA So, what’s the real problem?

7 7 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Facilitative decision support Partial model reflecting DM’s mind

8 8 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA “Off-line” decision analysis Complete model incorporating DM understanding

9 9 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA “Off-line” decision analysis Complete model incorporating DM understanding Verification & Validation

10 10 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Operational Analysis = Rigour R epeatability I ndependence G rounding in reality O bjectivity of process U ncertainty in data R obustness of results

11 11 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Objectivity  Quantification

12 12 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Objectivity  Quantification Subjective Qualification Objective Quantification

13 13 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Objectivity  Quantification Subjective Qualification Objective Quantification Subjective Quantification Objective Qualification

14 14 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Other inequalities Numeric  Rigorous Valid  Realistic Verbal  Subjective Simulation  Expensive Soft Analysis  Cheap Analysis Consultancy  Analysis

15 15 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Multi-criteria analysis (MCA)

16 16 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Common problems with MCA 8Measures of value which are not “real” 8Bias in subjective preference scores 8Unrepeatable, “one-shot” studies 8Inability to represent dynamic interdepency Methodological improvements to rigour of MCA 4Causal benefits modelling 4Development of “real” scales 4Explicit uncertainty modelling 4Compensating for interdependencies

17 17 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Benefits modelling Investments Value metrics How do we benefit from what we invest in?

18 18 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Benefits modelling Investments Value metrics How do we benefit from what we invest in? How much is enough? What is the right balance?

19 19 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Generic Benefits Model Structure Investments Value metrics System functions and attributes Organisation activities and attributes Military capabilities and attributes “...and attributes” is important! Causal modeling is essential!

20 20 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Causality and attributes zThe benefits chain must be causal to obtain: –auditability –objectivity –rigour zBUT... zCausal chains in complex systems are difficult. zMany benefits, e.g. of IT, are indirect and diffuse. zInvestment in architecture, infrastructure, integration. zTHEREFORE... zNeed to focus on more stable attributes of systems, organisations, and military capabilities. zAlso need less contingent measures.

21 21 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA faster & better informed decision efficient allocation of forces efficient allocation of assets responsive logistics situational awareness mission rehearsal aids ability to send/receive information accurate, timely & appropriate recognised picture asset tracking system interoperability minimise data interchange requirements complete, timely & accurate information Causal mapping (a fragment from a C3I study)

22 22 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Valuing a digitised system of systems ORGANISATION ACTIVITIES/ATTRIBUTES zInformation capture zInformation exchange zInformation processing zDecision making zProcess interoperability zProcess flexibility MILITARY CAPABILITIES zSimultaneity zConcentration of force - situation awareness zConcentration of force - synchronisation zFlexibility of disposition zFlexibility of capability zSpeed of decision zInteroperability zSustainability zEfficiency / economy of effort zReach VALUE SYSTEM zManoeuvre zFirepower / attrition zCoercion zSurvivability zPresence

23 23 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA FOR ALL EQUIPMENTS zImprove –Stretch potential –Reliability –Robustness / –Survivability –Time to service –Value for money –Interopability zReduce –Cost –Slippage –Maintenance –Mass –Vulnerability –Need for support in the field From a presentation by Lt General Burton, DCDS(S), July 1997

24 24 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Consequences, not preferences zTypically, MCA proceeds by asking: “Is option (a) better than option (b) in respect of criterion X?” “Is criterion X more important than criterion Y?” Aggregation of expressed preferences zBenefits modelling proceeds by asking: “How much does output criterion Y depend upon input criterion X?” Not an aggregation of preferences, but an estimation of consequences

25 25 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Using “real” scales z“The process of determining values for criteria weights calls for a lot of hard thinking on the part of the decision-maker” (R.L.Pratt, V.Belton). zOff-line analysis do not have this luxury. zTherefore, scales at each stage in the model must be meaningful so that scores are open to independent verification.

26 26 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Using “real” scales z“The process of determining values for criteria weights calls for a lot of hard thinking on the part of the decision-maker” (R.L.Pratt, V.Belton). zOff-line analysis do not have this luxury. zTherefore, scales at each stage in the model must be meaningful so that scores are open to independent verification. 0.01.0

27 27 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Uncertainty and sensitivity zStudy related to a particular HQ C2 system zBenefits modelling used to establish the overall requirement and to compare options in a COEIA zMulti-stage benefits model...

28 28 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Scenario MOEs Dependency and value measures Dependency and value measures C2I System Non-CIS non-C2I MoPs (text based) C2I system MoPs Mission MOEs Mission Weightings C2I fns Other CIS investments Stage 1: System functions and attributes Stage 2: Organisational functions and attributes Stage 3: Military Capabilities Stage 4: Value System Benefits model used for a C2 system

29 29 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Maths from a typical Benefits Model

30 30 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Maths from a typical Benefits Model

31 31 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Example sensitivity conclusions zThe confidence limits were normally ~ +/- 10% with some weightings varied by up to +/- 30%. zThe benefits model was fully connected with most system criteria contributing to most organisational ones. zEffectiveness was relatively insensitive to variations any one MOP or weight, but did show the accrued benefit of improvement in many MOPs zThe conclusions from statistical sensitivity analysis were confirmed by sampled manual variations in scores and weights.

32 32 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Interdependence of criteria zBenefits modelling, like other MCA methods, is basically a set of linear filters. zNon-linearities in individual causal chains can be included. zNon-linearities due to criterion interdependence are more problematic. zA partial solution is the additional of first order corrections via interdependency matrices.

33 33 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Generalised BM transition (based on Quality Function Deployment) Input categories Output categories Dependency matrix Interdependency matrix

34 34 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Linking to dynamic modelling zCausal mapping leads naturally on to dynamic network modelling techniques such as System Dynamics or Petri-Nets. zBenefits modelling can be used to identify reduce the scope of the problem by identifying driving parameters. zIn the C2 system study, it was concluded from the benefits analysis that timeliness was not a driver and, thus, dynamic modelling would not be cost-effective.

35 35 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Summary z“Soft” analysis methods are firmly established in OA because they can cope with the breadth and complexity of the problem space. zMany methods have arisen from facilitative decision support and have limited rigour. zWithout rigour, OA is nothing but consultancy. zThe rigour of “soft” methods can be improved. èOA can survive the “soft” revolution......but only if it adopts rigorous subjectivity.

36 36 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Questions?


Download ppt "1 15 ISMOR - 01/09/2014 Centre for Defence Analysis  British Crown Copyright 1998/DERA Rigorous Subjectivity How will OA survive the “soft” revolution?"

Similar presentations


Ads by Google