1. Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet Security Research Lab Computer Science Department Brigham Young University

2. Confused Johnny o E-mail encryption for the masses o We developed a system maximizing usability ◦Made everything transparent o Johnny became confused o Designed another system with manual encryption ◦This helped Johnny gain clarity

3. Encrypted E-mail o Exists, but largely goes unused o S/MIME, PGP ◦Tools available o “Why Johnny can't encrypt: A usability evaluation of PGP 5.0” ◦Whitten and Tygar, 8th USENIX Security Symposium (1999) ◦Later research confirmed findings o What can be done?

4. Usability Issues o Users resist change ◦Users are using webmail ◦If security is difficult users will forgo it o Key management is confusing ◦Hierarchical, web-of-trust ◦Recipient must already have key ◦Chicken and egg problem o Cryptography is complicated ◦Unclear which properties are provided ◦Unclear which properties are needed

5. Private Webmail (Pwm) o Pronounced “Poem” o Adds end-to-end encryption to existing webmail systems ◦Gmail, Hotmail, Yahoo! Mail ◦Runs on all modern browsers o Designed to maximize usability o Provide good-enough security ◦Improvement for those already sending sensitive e-mail

6. Users Resist Change o Security overlays ◦Integrates tightly with existing webmail systems ◦Users do not need to learn yet-another-system o Tightly integrates with existing systems ◦Replaces small portions of the interface ◦Displayed using iFrames o Functionally transparent ◦Low barrier to adoption o Visually distinctive ◦Easy to identify

7. Usability Fixes o Users resist change ◦Focus on bootstrapping first-time users ◦Helpful instructions in e-mail ◦Bookmarklet-based installation o Key management is confusing ◦Key escrow based on IBE ◦Simple Authentication for the Web (EBIA) ◦No user interaction required o Cryptography is complicated ◦Encryption is automatically handled by Pwm ◦Users never interact with ciphertext

8. Pwm: Walkthrough

13. Pwm User Studies o Two studies o First study measured usability of Pwm ◦Also evaluated bookmarklets for use during installation o Second study compared Pwm to Voltage Secure Mail Cloud ◦Voltage Secure Mail Cloud is an existing depot-based secure email system ◦Pwm was run using a browser extension o Evaluation ◦Pre- and post-survey questionnaire ◦Monitored participants actions for unrecognized mistakes ◦Post-survey interviews

14. System Usability Scale o Brook (1996) o Ten questions ◦Alternate negative and positive ◦Give a single number for usability o Bangor compared scores for hundreds of systems

15. 76 71 63

16. SUS Score Comparison

17. Success? o Results are very promising ◦Very positive reception ◦Users indicated they wanted to begin using it o Not without problems o Small number sent e-mail without encryption o Participants were confused about security ◦Wanted to see more details ◦Unsure of who could read e-mails

18. Where to go from here? o Simple solutions was to fix UI issues o One author (Nathan Kim) had a different idea ◦Manual encryption ◦Decoupled interface o Mocked up these ideas ◦Message Protector (MP) ◦Simple Interface ◦Direct handling of ciphertext ◦Implied key management

19. MP: Walkthrough

23. First MP User Study o Evaluated MP using SUS o Compared against Encipher.it ◦Bookmarklet-based encryption system ◦Works in Gmail and Facebook o Evaluation ◦Pre- and post-survey questionnaire ◦Monitored participants actions for unrecognized mistakes ◦Post-survey interviews ◦The system usability scale o Evaluated comprehension ◦Survey included questions about comprehension ◦How to use the system ◦Who could read messages

24. 61 72

25. Second MP User Study o Surprising usability results ◦Participants had a positive reaction to seeing ciphertext ◦Similar SUS score to MP o Ran a second study comparing MP to Pwm ◦Modeled after the first MP study

26. 76 74

27. SUS Score Comparison

28. Other results o MP improved users comprehension ◦Clearly understood how to use system ◦Clearly understood who could read messages o Usability scores nearly identical to Pwm o Participants preferred manual encryption of MP o Participants preferred tight integration of Pwm

29. Study limitations o MP studies ignore bootstrapping new users ◦Studies assumed software pre-installed ◦Bootstrapping is a key component of Pwm’s design ◦Not fully representative of overall usability o Short-term studies o SUS question unclear ◦“I think that I would like to use this system frequently.” ◦Participants ranked low even when enthusiastic about the system ◦Relevant to security studies

30. Related Works o “Johnny 2: a user test of key continuity management with s/mime and outlook express.” ◦Garfinkel and Miller, SOUPS 2005 ◦Applied automatic key management to e-mail ◦Allowed great success o “Helping Johnny 2.0 to encrypt his Facebook conversations.” ◦Fahl, et al., SOUPS 2012 ◦Encrypted Facebook communication ◦Explored manual vs. automatic encryption ◦Invisibility security not trusted by users

31. Conclusion o Pwm was a success ◦Participants largely succeeded at using encrypted e-mail ◦Participants had high praise for Pwm ◦Succeeding in being easy for new users o Pwm wasn’t perfect ◦Security was too transparent ◦Caused users to be confused and make mistakes o Mocked up a system using manual encryption ◦Users enjoyed manual encryption ◦Wished it was tightly integrated with the browser o A combination of approaches is needed to solve the problem

32. Conclusion: Future Work o Manual encryption in Pwm ◦Don’t automatically send encrypted email ◦“Encrypt” button which puts ciphertext in compose window o Sidebar ◦Browser sidebar allowing for manual encryption ◦Can be used on any site ◦Fallback for when Pwm has an error o Long-term studies ◦Larger populations ◦Real tasks

33. Questions?