Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet Security Research Lab Computer Science Department Brigham Young University

Confused Johnny o encryption for the masses o We developed a system maximizing usability ◦Made everything transparent o Johnny became confused o Designed another system with manual encryption ◦This helped Johnny gain clarity

Encrypted o Exists, but largely goes unused o S/MIME, PGP ◦Tools available o “Why Johnny can't encrypt: A usability evaluation of PGP 5.0” ◦Whitten and Tygar, 8th USENIX Security Symposium (1999) ◦Later research confirmed findings o What can be done?

Usability Issues o Users resist change ◦Users are using webmail ◦If security is difficult users will forgo it o Key management is confusing ◦Hierarchical, web-of-trust ◦Recipient must already have key ◦Chicken and egg problem o Cryptography is complicated ◦Unclear which properties are provided ◦Unclear which properties are needed

Private Webmail (Pwm) o Pronounced “Poem” o Adds end-to-end encryption to existing webmail systems ◦Gmail, Hotmail, Yahoo! Mail ◦Runs on all modern browsers o Designed to maximize usability o Provide good-enough security ◦Improvement for those already sending sensitive

Users Resist Change o Security overlays ◦Integrates tightly with existing webmail systems ◦Users do not need to learn yet-another-system o Tightly integrates with existing systems ◦Replaces small portions of the interface ◦Displayed using iFrames o Functionally transparent ◦Low barrier to adoption o Visually distinctive ◦Easy to identify

Usability Fixes o Users resist change ◦Focus on bootstrapping first-time users ◦Helpful instructions in ◦Bookmarklet-based installation o Key management is confusing ◦Key escrow based on IBE ◦Simple Authentication for the Web (EBIA) ◦No user interaction required o Cryptography is complicated ◦Encryption is automatically handled by Pwm ◦Users never interact with ciphertext

Pwm: Walkthrough

Pwm User Studies o Two studies o First study measured usability of Pwm ◦Also evaluated bookmarklets for use during installation o Second study compared Pwm to Voltage Secure Mail Cloud ◦Voltage Secure Mail Cloud is an existing depot-based secure system ◦Pwm was run using a browser extension o Evaluation ◦Pre- and post-survey questionnaire ◦Monitored participants actions for unrecognized mistakes ◦Post-survey interviews

System Usability Scale o Brook (1996) o Ten questions ◦Alternate negative and positive ◦Give a single number for usability o Bangor compared scores for hundreds of systems

SUS Score Comparison

Success? o Results are very promising ◦Very positive reception ◦Users indicated they wanted to begin using it o Not without problems o Small number sent without encryption o Participants were confused about security ◦Wanted to see more details ◦Unsure of who could read s

Where to go from here? o Simple solutions was to fix UI issues o One author (Nathan Kim) had a different idea ◦Manual encryption ◦Decoupled interface o Mocked up these ideas ◦Message Protector (MP) ◦Simple Interface ◦Direct handling of ciphertext ◦Implied key management

MP: Walkthrough

First MP User Study o Evaluated MP using SUS o Compared against Encipher.it ◦Bookmarklet-based encryption system ◦Works in Gmail and Facebook o Evaluation ◦Pre- and post-survey questionnaire ◦Monitored participants actions for unrecognized mistakes ◦Post-survey interviews ◦The system usability scale o Evaluated comprehension ◦Survey included questions about comprehension ◦How to use the system ◦Who could read messages

61 72

Second MP User Study o Surprising usability results ◦Participants had a positive reaction to seeing ciphertext ◦Similar SUS score to MP o Ran a second study comparing MP to Pwm ◦Modeled after the first MP study

76 74

SUS Score Comparison

Other results o MP improved users comprehension ◦Clearly understood how to use system ◦Clearly understood who could read messages o Usability scores nearly identical to Pwm o Participants preferred manual encryption of MP o Participants preferred tight integration of Pwm

Study limitations o MP studies ignore bootstrapping new users ◦Studies assumed software pre-installed ◦Bootstrapping is a key component of Pwm’s design ◦Not fully representative of overall usability o Short-term studies o SUS question unclear ◦“I think that I would like to use this system frequently.” ◦Participants ranked low even when enthusiastic about the system ◦Relevant to security studies

Related Works o “Johnny 2: a user test of key continuity management with s/mime and outlook express.” ◦Garfinkel and Miller, SOUPS 2005 ◦Applied automatic key management to ◦Allowed great success o “Helping Johnny 2.0 to encrypt his Facebook conversations.” ◦Fahl, et al., SOUPS 2012 ◦Encrypted Facebook communication ◦Explored manual vs. automatic encryption ◦Invisibility security not trusted by users

Conclusion o Pwm was a success ◦Participants largely succeeded at using encrypted ◦Participants had high praise for Pwm ◦Succeeding in being easy for new users o Pwm wasn’t perfect ◦Security was too transparent ◦Caused users to be confused and make mistakes o Mocked up a system using manual encryption ◦Users enjoyed manual encryption ◦Wished it was tightly integrated with the browser o A combination of approaches is needed to solve the problem

Conclusion: Future Work o Manual encryption in Pwm ◦Don’t automatically send encrypted ◦“Encrypt” button which puts ciphertext in compose window o Sidebar ◦Browser sidebar allowing for manual encryption ◦Can be used on any site ◦Fallback for when Pwm has an error o Long-term studies ◦Larger populations ◦Real tasks

Questions?