Comparing two techniques for intrusion visualization Vikash Katta 1,3, Peter Karpati 1, Andreas L. Opdahl 2, Christian Raspotnig 2,3 & Guttorm Sindre 1 1) Norwegian University of Science and Technology, Trondheim 2) University of Bergen, Norway 3) Institute for Energy Technology, Halden, Norway

The ReqSec Project Method and tool support for security requirements engineering: involve non-experts lightweight integrated, add-on industrially evaluated Funded by the Norwegian Research Council (NFR), Many techniques proposed, e.g., anti-behaviours...

Perspective System security models: black-box models of monolothic systems single systems security analysis and specification Security architecture models: high-level organisational views enterprise architecture for security Need for intermediate solutions: security modelling for SOA white-box models of service collaborations bordering organisation and technology

Misuse Case Maps (MUCM) Inspired by Use Case Maps (R.J.A. Buhr, D. Aymot...)

Misuse Case Maps (MUCM) Use case maps: components, scenario paths, responsibilities Misuse case maps: vulnerabilities, exploit paths, vulnerable responsibility Preliminary evaluations: good for architectural overviews need better visualisation of attack step sequences

Misuse Sequence Diagrams (MUSD)

Sequence diagrams: actor, object/component, action, event/message Misuse sequence diagrams: attacker, vulnerability, exploit action and event/message Initial evaluation: can MUSD complement MUCM? how do the two techniques compare wrt. understanding performance perception

Comparison Controlled experiment with 42 subjects Latin squares organisation, random assignment Treatment (independent variables): technique: MUCM, MUSD task: bank intrusion (BAN), penetration test (PEN) Measures (dependent variables): understanding (UND) performance (VULN, MITIG, VUMI) perception (PER) Control (control variables): background (KNOW, STUDY, JOB)

Hypotheses H1 1 : MUCM better on architectural questions H2 1 : MUSD better for temporal sequence questions H3 1 : Either technique better on the neutral questions H4 1 : Either technique better overall H5 1 : Different numbers of vulnerabilities identified H6 1 : Different numbers of mitigations identified H7 1 : Different total numbers of vulnerabilities and mitigations identified H8 1 : Usefulness perceived differently H9 1 : Ease of use perceived differently H10 1 : Intentions to use perceived differently H11 1 : MUCM and MUSD perceived differently

Procedure 4 groups of nd year computer science students 10 steps: Filling in the pre-experiment questionnaire (2 min) Reading a short introduction to the experiment (1 min) First technique on first task:  introduction to the technique (9 min)  read about task, looking at diagrams (12 min)  20 true/false questions about the case (8 min)  finding vulnerabilities and mitigations (11 min)  post-experiment questionnaire (4 min) Easy physical exercise (2 min) Repeat for second technique and task (44 min)

Results Backgrounds: No sig. differences between groups:  Kruskal-Wallis H test – 2-4 semesters of ICT studies – 2.07 months of job experience (three outliers) Sig. knowledge differences across groups: – Wilcoxon signed-rank tests – KNOW_MOD > KNOW_SEC, p =.000 – KNOW_SD > KNOW_UCM, p =.003 – KNOW_MUSD ≈ KNOW_MUCM

Understanding Wilcoxon signed-rank tests H1 & H2 accepted, H3 & H4 rejected Medium effect size (Cohen) No impact of technique or task order

Performance Two blank outliers removed (from 11-student groups) H5, H6 & H7 rejected No impact of technique order More identifications for bank task

Perception H8, H9, H10 & H11 accepted Medium to large effect sizes (Cohen) Only one insig. statement (“would be useless”) More positive perception of first technique used

Conclusion The techniques are complementary They facilitate understanding better for their “intended use”: – MUCM best for architectural issues – MUSD best for temporal sequences They are equal in performance – the bank task was more productive MUSDs were perceived more positively – the first technique was perceived more positively Further work: simpler MUCMs, qualitative analysis, more techniques, industrial subjects, notation and method integration, industrial case studies and action research...

Central concepts RFC 2828: vulnerability: a weakness in a system... that can be exploited to violate its security policy threat: a potential for violation of security... that could cause harm countermeasure: something that reduces a threat or attack by eliminating... preventing... minimizing the harm... or by reporting it to enable corrective action