Presentation is loading. Please wait.

Presentation is loading. Please wait.

© Copyright 2010 Hewlett-Packard Development Company, L.P. 1 Adrian Baldwin, Yolanta Beres, Marco Casassa Mont, Simon Shiu (all HP Labs) Geoff Duggan,

Similar presentations


Presentation on theme: "© Copyright 2010 Hewlett-Packard Development Company, L.P. 1 Adrian Baldwin, Yolanta Beres, Marco Casassa Mont, Simon Shiu (all HP Labs) Geoff Duggan,"— Presentation transcript:

1 © Copyright 2010 Hewlett-Packard Development Company, L.P. 1 Adrian Baldwin, Yolanta Beres, Marco Casassa Mont, Simon Shiu (all HP Labs) Geoff Duggan, Hilary Johnson (University of Bath) Chris Middup (Open University) AN EXPERIMENT IN SECURITY DECISION MAKING

2 © Copyright 2010 Hewlett-Packard Development Company, L.P. 2 CONTEXT –TSB funded trust economics project: We developed an approach (using economic and mathematical modelling) to help enterprises make “better” security decisions A series of case studies providing good feedback and anecdotal evidence that were on a good path –Challenge – can we do better than that? –This paper: An in depth study of a small group of security professionals (one stakeholder type), on how our approach to security decision making affects them

3 © Copyright 2010 Hewlett-Packard Development Company, L.P. 3 A RIGOROUS APPROACH TO SECURITY DECISION MAKING System Model Problem Architectur e consequences of preferences problem refinement things to measure components of utility Problem Preferences Utility

4 © Copyright 2010 Hewlett-Packard Development Company, L.P. 4 SDM HYPOTHESES Our methods will positively influence: –the conclusions or decisions made, –the thought process followed, –the justifications given, and –the confidence the stakeholder has in the final conclusions or decisions made.

5 © Copyright 2010 Hewlett-Packard Development Company, L.P. 5 SDM EXPERIMENT SCOPE –Measure effect on security professionals/experts (i.e. not our effect on other stakeholders nor groups/organisations) –Qualitative in depth study of decision making process (of twelve professionals) –Bundled economic framing and system modelling as a “single” intervention –Controlled experiment, i.e. two groups one intervened using our methods, one left as a control

6 © Copyright 2010 Hewlett-Packard Development Company, L.P. 6 THE SDM PROBLEM –Chose a problem on the security of client infrastructure –Why – we had several similar case studies that meant we knew: it was a representative current and challenging business security problem we had decent/realistic empirical data relating to the problem there are interesting “trade-offs” that meant the answer is subjective and contextual and likely to be different for different stakeholders –We had 4 decision options that represented different trade-offs –We had to iterate a number of times before we had sufficient supporting material and a problem we could control, and that was rich enough!

7 © Copyright 2010 Hewlett-Packard Development Company, L.P. 7 EXPERIMENT DESIGN 5a. Preference/ Economic Framing 5b. Modelling & Results 2. Problem Description 4. Decision Options 6. Choice & Justification 7. Introspection 1. Session Introduction 3. Question & Answers 5. Question & Answers

8 © Copyright 2010 Hewlett-Packard Development Company, L.P. 8 PHASES 1.Session introduction 2.Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES –Options Invest in patching Invest in Host based intrusion prevention (HIPS) technology Change policy to lock down (remove admin privileges) from users Do nothing

9 © Copyright 2010 Hewlett-Packard Development Company, L.P. 9 PHASES 1.Session introduction 2.Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES (INTERVENE PHASE ONLY) –Identify major outcomes (components of utility) –Identify appropriate proxy metrics for each outcome –Prioritise outcomes

10 © Copyright 2010 Hewlett-Packard Development Company, L.P. 10 PHASES 1.Session introduction 2.Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES –Describe model of concurrent processes, and how options are explored –Show (chosen proxy measure) results in 3*3 results tool

11 © Copyright 2010 Hewlett-Packard Development Company, L.P. 11 DATA ANALYSIS –173 questions before intervention (from all twelve participants) –152 justifications (from all twelve participants) –6 ordered prioritised outcomes –12 decision options –48 Likert scores on confidence (four from each participant)

12 © Copyright 2010 Hewlett-Packard Development Company, L.P. 12 THE CHOICES –In the control group: 3 selected Lockdown, 2 selected HIPS and 1 selected Patching –In the intervention group: 3 selected Lockdown and 3 selected HIPS –A very security oriented set of options!

13 © Copyright 2010 Hewlett-Packard Development Company, L.P. 13 CATEGORIZATION OF QUESTIONS Similar balance between groups

14 © Copyright 2010 Hewlett-Packard Development Company, L.P. 14 CATEGORIZATION OF JUSTIFICATIONS More balanced business justification for the intervened group

15 © Copyright 2010 Hewlett-Packard Development Company, L.P. 15 SDM HYPOTHESES Our methods will positively influence: the conclusions or decisions made, the thought process followed, the justifications given, and the confidence the stakeholder has in the final conclusions or decisions made. SDM RESULTS WHAT DO THE DATA RESULTS SAY IN RELATION TO OUR ORIGINAL HYPOTHESIS –Not sufficient evidence that we influenced conclusions or decisions made –There is evidence we influenced the justifications given Which in turn suggests we affected their thought processes –There was a slight (but not significant) increase in confidence in decisions made

16 © Copyright 2010 Hewlett-Packard Development Company, L.P. 16 SOME FURTHER ANALYSIS potential theoretical explanations NB on study style: smaller qualitative studies often fertile for early theoretical development –Security priority in questions (and control group’s justifications) suggest presence of confirmation bias –The intervened group’s broader justifications suggest our methods managed to counter some of this bias –The intervened group did not value the economic framing “i’d made those trade offs already” is at odds with this result - suggests cognitive dissonance

17 © Copyright 2010 Hewlett-Packard Development Company, L.P. 17 CONCLUSIONS & NEXT STEPS –Encouragement that economic framing improves analysis Assume that a study of group decision support would make this results stronger –Encouragement to use tools to support simultaneous comparison of multiple outcomes and choices –More cognitive science should be done to complement security economics –Future analysis Study ‘question’ data to see methods/structure followed by security profession (compared with ISO27k, hunting for low hanging fruit,...) –Future studies To test the suggested theories To explore the effect on multi-stakeholder decision making

18 © Copyright 2010 Hewlett-Packard Development Company, L.P. 18 © Copyright 2010 Hewlett-Packard Development Company, L.P. QUESTIONS

19 © Copyright 2010 Hewlett-Packard Development Company, L.P. 19 PHASES 1.Session introduction 2.Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES (INTERVENE PHASE ONLY) –Identify major outcomes (components of utility) –Identify appropriate proxy metrics for each outcome –Prioritise outcomes

20 © Copyright 2010 Hewlett-Packard Development Company, L.P. 20 PHASES 1.Session introduction 2.Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES –Prioritise outcomes

21 © Copyright 2010 Hewlett-Packard Development Company, L.P. 21 PHASES 1.Session introduction 2.Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES –Describe model of concurrent processes, empirical studies, and how options are explored

22 © Copyright 2010 Hewlett-Packard Development Company, L.P. 22 PHASES 1.Session introduction 2.Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES –Show results in 3*3 (option to proxy measure) results tool

23 © Copyright 2010 Hewlett-Packard Development Company, L.P. 23 PHASES 1.Session introduction 2.Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES –10 minutes to ask any questions they deem relevant –Scripted answers (e.g. on history, culture, processes, architecture, business, regulations etc…) –Answers to “new” questions were added to the script for future sessions –After 10 minutes we provided “essential” information that had not been asked about –This allowed us to collect data on what questions were asked and in what order

24 © Copyright 2010 Hewlett-Packard Development Company, L.P. 24 PHASES 1.Session introduction 2. Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES –Choose preferred option –For each option: Pro’s – reasons why option would be good Con’s – reasons why option would be bad Likert scale 1-7 confidence in the option

25 © Copyright 2010 Hewlett-Packard Development Company, L.P. 25 PHASES 1.Session introduction 2. Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES –For intervened group What difference the interventions and tools made –What information they used to reach their conclusion –Any strategies they used when asking questions

26 © Copyright 2010 Hewlett-Packard Development Company, L.P. 26 PHASES 1.Session introduction 2.Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES –3 Roles: interviewer, expert and observer –Interviewer explained and gathered: Structure of session Incentives for trying hard Experience of participant

27 © Copyright 2010 Hewlett-Packard Development Company, L.P. 27 PHASES 1.Session introduction 2.Problem description 3.Q&A 4.Decision options 5.(a) Preference Elicitation (b) Model analysis 6.Choice & Justification 7.Introspection EXPERIMENT PHASES –Verbally scripted, web based and written material introducing them to the security role they are being asked to play and the client infrastructure security problem the CISO has. –Whether/how to deal with rising risk from malware on client infrastructure

28 © Copyright 2010 Hewlett-Packard Development Company, L.P. 28 DATA ANALYSIS –All questions and justifications were transcribed and put in ‘random’ order –3 experts categorised these – differences resolved through discussion Relation to ISO Relation to main business outcomes (compliance, productivity, cost, security risk)


Download ppt "© Copyright 2010 Hewlett-Packard Development Company, L.P. 1 Adrian Baldwin, Yolanta Beres, Marco Casassa Mont, Simon Shiu (all HP Labs) Geoff Duggan,"

Similar presentations


Ads by Google