Presentation on theme: "With your instructor, Jeremy Hyland"— Presentation transcript:
1 With your instructor, Jeremy Hyland User Studies IIWith your instructor, Jeremy Hyland
2 Plan for Today Discuss the reading: Do a little testing of our own… Why Johnny Can’t EncryptJohnny 2: Judgment DayDo a little testing of our own…
3 Why Johnny Can’t Encrypt Who’s Johnny and why can’t he encrypt?Posner saysWhat’s Johnny trying to hide?
4 Why Johnny Can’t Encrypt Whitten and Tygar, 1999A Usability Evaluation of PGP 5.0
5 Why Johnny Can’t Encrypt “Security mechanisms are only effective when used correctly”So:If Usable thenelse
6 Why Johnny Can’t Encrypt Defining Usable Security SoftwareWhitten and Tygar:Security software is usable if the people who are expected to use it:are reliably made aware of the security tasks they need to perform.are able to figure out how to successfully perform those tasksdon't make dangerous errorsare sufficiently comfortable with the interface to continue using it.
7 Why Johnny Can’t Encrypt Why is usable security hard?McNealy saysYou have no usable security, get over it.
8 Why Johnny Can’t Encrypt Why is usable security hard?Five reasons:1. The unmotivated users“Security is usually a secondary goal”2. Policy AbstractionProgrammers understand the representation but normal users have no background knowledge.
9 Why Johnny Can’t Encrypt Why is usable security hard?Five reasons:3. The lack of feedbackWe can’t predict every situation.4. The proverbial “barn door”Need to focus on error prevention.5. The weakest linkAttacker only needs to find one vulnerability
10 Why Johnny Can’t Encrypt Usability EvaluationPGP 5.0Pretty Good PrivacySoftware for encrypting and signing dataPlug-in provides “easy” use with clientsModern GUI, well designed by most standards
11 Why Johnny Can’t Encrypt Usability EvaluationWhitten and Tygar focus their evaluation on a question based off their definition of usable secure software:If an average user of feels the need for privacy and authentication, and acquires PGP with that purpose in mind, will PGP's current design allow that person to realize what needs to be done, figure out how to do it, and avoid dangerous errors, without becoming so frustrated that he or she decides to give up on using PGP after all?Loaded question?
12 Why Johnny Can’t Encrypt Usability EvaluationCognitive walk throughMentally step through the software as if we were a new user. Attempt to identify the usability pitfalls.Focus on interface learnablity.
13 Why Johnny Can’t Encrypt Usability EvaluationCognitive walk through results:Visual metaphorsPublic vs. Private keysSignatures and verification
14 Why Johnny Can’t Encrypt Usability EvaluationCognitive walk through results:Different key typesCompatibility increases complexityKeys listed as users
16 Why Johnny Can’t Encrypt Usability EvaluationCognitive walk through results:Key serverHidden?What is it doing?Revocation not automaticWould that help?
17 Why Johnny Can’t Encrypt Usability EvaluationCognitive walk through results:Key management policyUnneeded confusionWhat’s the difference between trust and validity?
18 Why Johnny Can’t Encrypt Usability EvaluationCognitive walk through results:Irreversible actionsNeed to prevent costly errorsConsistency“Encoding”?!?Too much informationMore unneeded confusionShow the basic information, make more advanced information available only when needed.
19 Why Johnny Can’t Encrypt Usability EvaluationUser TestPGP 5.0 with Eudora12 participants all with at least some college and none with advanced knowledge of encryptionParticipants were given a scenario with tasks to complete within 90 minTasks built on each otherParticipants could ask some questions through
20 Why Johnny Can’t Encrypt Usability EvaluationUser Test Results:3 users accidentally sent the message in clear text7 users used their public key to encrypt and only 2 of the 7 figured out how to correct the problemOnly 2 users were able to decrypt without problemsOnly 1 user figured out how to deal with RSA keys correctly.A total of 3 users were able to successfully complete the basic process of sending and receiving encrypted s.One user was not able to encrypt at all
21 Why Johnny Can’t Encrypt ConclusionIf an average user of feels the need for privacy and authentication, and acquires PGP with that purpose in mind, will PGP's current design allow that person to realize what needs to be done, figure out how to do it, and avoid dangerous errors, without becoming so frustrated that he or she decides to give up on using PGP after all?NopeIs this a failure in the design of the PGP 5.0 interface or is it a function of the problem of traditional usable design vs. design for usable secure systems?Security as the primary function vs. a secondary function
22 Johnny 2 Garfinkel and Miller, 2005 Follow-up to Why “Johnny Can’t encrypt”Test of new encryption technologyKey Continuity ManagementS/MIME certificatesBetter interfaceSimple buttons
23 Johnny 2 Garfinkel and Miller: Johnny couldn’t encrypt because of the key architecture behind PGP.“….the fundamental usability barriers that Whitten identified could be overcome by replacing the underlying third-party certification model with Key Continuity Management.”
24 Johnny 2User TestTried to stay as close to the Johnny experiment as practicalSame methods of user solicitation/selectionSame basic scenarioSimilar user tasksAdded attackers
25 Johnny 2 User Test Attacks: new key attacknew identity attackunsigned message attackHow well does the interface enable users to respond to these attacks?
26 Johnny 2 User Test Test application: CoPilot “Wizard of Oz” prototype S/MIME certificate handling:First time = YellowTrusted certificate = GreenChanged certificate = RedUnsigned message = WhiteUnsigned message from a sender that normal sends signed messages = GrayBetter tools allow for a more automated and scientific test
27 Johnny 2 User Test 43 test subjects Three groups: No KCM Color Color+Briefing
28 Johnny 2 User Test Results: Users generally understood the basics Little understanding of signature integrity guaranteesVerifying attack message authenticity was difficult for most usersNo group resisted attacks 100% of the timeColor and Color+Briefing resisted new key attack and the unsigned message attack better then No KCMThe interface did not help against new identity attacks
29 Johnny 2 User Test: Conclusions A few surface interface issues Do not trust buttonMisconceptions about the security of sealed messagesGenerally, the new interface simplifies encryptionStill problems with determining certificate trust, however some of these problems may be unavoidable.
31 User Test 3 groups: Take a few minutes to create a simple user test Cell PhoneCD playerCalculatorTake a few minutes to create a simple user testOne member of each group switches to be a tester
32 User Test Guidance: Decide whose going to do what! Create a Use Case ScenarioDefine user tasks for completion of the scenarioSet up metrics for results evaluationWhat qualifies as success vs. failure?