1. With your instructor, Jeremy Hyland
User Studies II
With your instructor, Jeremy Hyland

2. Plan for Today Discuss the reading: Do a little testing of our own…
Why Johnny Can’t Encrypt
Johnny 2: Judgment Day
Do a little testing of our own…

3. Why Johnny Can’t Encrypt
Who’s Johnny and why can’t he encrypt?
Posner says
What’s Johnny trying to hide?

4. Why Johnny Can’t Encrypt
Whitten and Tygar, 1999
A Usability Evaluation of PGP 5.0

5. Why Johnny Can’t Encrypt
“Security mechanisms are only effective when used correctly”
So:
If Usable then
else

6. Why Johnny Can’t Encrypt
Defining Usable Security Software
Whitten and Tygar:
Security software is usable if the people who are expected to use it:
are reliably made aware of the security tasks they need to perform.
are able to figure out how to successfully perform those tasks
don't make dangerous errors
are sufficiently comfortable with the interface to continue using it.

7. Why Johnny Can’t Encrypt
Why is usable security hard?
McNealy says
You have no usable security, get over it.

8. Why Johnny Can’t Encrypt
Why is usable security hard?
Five reasons:
1. The unmotivated users
“Security is usually a secondary goal”
2. Policy Abstraction
Programmers understand the representation but normal users have no background knowledge.

9. Why Johnny Can’t Encrypt
Why is usable security hard?
Five reasons:
3. The lack of feedback
We can’t predict every situation.
4. The proverbial “barn door”
Need to focus on error prevention.
5. The weakest link
Attacker only needs to find one vulnerability

10. Why Johnny Can’t Encrypt
Usability Evaluation
PGP 5.0
Pretty Good Privacy
Software for encrypting and signing data
Plug-in provides “easy” use with clients
Modern GUI, well designed by most standards

11. Why Johnny Can’t Encrypt
Usability Evaluation
Whitten and Tygar focus their evaluation on a question based off their definition of usable secure software:
If an average user of feels the need for privacy and authentication, and acquires PGP with that purpose in mind, will PGP's current design allow that person to realize what needs to be done, figure out how to do it, and avoid dangerous errors, without becoming so frustrated that he or she decides to give up on using PGP after all?
Loaded question?

12. Why Johnny Can’t Encrypt
Usability Evaluation
Cognitive walk through
Mentally step through the software as if we were a new user. Attempt to identify the usability pitfalls.
Focus on interface learnablity.

13. Why Johnny Can’t Encrypt
Usability Evaluation
Cognitive walk through results:
Visual metaphors
Public vs. Private keys
Signatures and verification

14. Why Johnny Can’t Encrypt
Usability Evaluation
Cognitive walk through results:
Different key types
Compatibility increases complexity
Keys listed as users

15. Why Johnny Can’t Encrypt
Keys listed as users

16. Why Johnny Can’t Encrypt
Usability Evaluation
Cognitive walk through results:
Key server
Hidden?
What is it doing?
Revocation not automatic
Would that help?

17. Why Johnny Can’t Encrypt
Usability Evaluation
Cognitive walk through results:
Key management policy
Unneeded confusion
What’s the difference between trust and validity?

18. Why Johnny Can’t Encrypt
Usability Evaluation
Cognitive walk through results:
Irreversible actions
Need to prevent costly errors
Consistency
“Encoding”?!?
Too much information
More unneeded confusion
Show the basic information, make more advanced information available only when needed.

19. Why Johnny Can’t Encrypt
Usability Evaluation
User Test
PGP 5.0 with Eudora
12 participants all with at least some college and none with advanced knowledge of encryption
Participants were given a scenario with tasks to complete within 90 min
Tasks built on each other
Participants could ask some questions through

20. Why Johnny Can’t Encrypt
Usability Evaluation
User Test Results:
3 users accidentally sent the message in clear text
7 users used their public key to encrypt and only 2 of the 7 figured out how to correct the problem
Only 2 users were able to decrypt without problems
Only 1 user figured out how to deal with RSA keys correctly.
A total of 3 users were able to successfully complete the basic process of sending and receiving encrypted s.
One user was not able to encrypt at all

21. Why Johnny Can’t Encrypt
Conclusion
If an average user of feels the need for privacy and authentication, and acquires PGP with that purpose in mind, will PGP's current design allow that person to realize what needs to be done, figure out how to do it, and avoid dangerous errors, without becoming so frustrated that he or she decides to give up on using PGP after all?
Nope
Is this a failure in the design of the PGP 5.0 interface or is it a function of the problem of traditional usable design vs. design for usable secure systems?
Security as the primary function vs. a secondary function

22. Johnny 2 Garfinkel and Miller, 2005
Follow-up to Why “Johnny Can’t encrypt”
Test of new encryption technology
Key Continuity Management
S/MIME certificates
Better interface
Simple buttons

23. Johnny 2 Garfinkel and Miller:
Johnny couldn’t encrypt because of the key architecture behind PGP.
“….the fundamental usability barriers that Whitten identified could be overcome by replacing the underlying third-party certification model with Key Continuity Management.”

24. Johnny 2
User Test
Tried to stay as close to the Johnny experiment as practical
Same methods of user solicitation/selection
Same basic scenario
Similar user tasks
Added attackers

25. Johnny 2 User Test Attacks:
new key attack
new identity attack
unsigned message attack
How well does the interface enable users to respond to these attacks?

26. Johnny 2 User Test Test application: CoPilot “Wizard of Oz” prototype
S/MIME certificate handling:
First time = Yellow
Trusted certificate = Green
Changed certificate = Red
Unsigned message = White
Unsigned message from a sender that normal sends signed messages = Gray
Better tools allow for a more automated and scientific test

27. Johnny 2 User Test 43 test subjects Three groups: No KCM Color
Color+Briefing

28. Johnny 2 User Test Results: Users generally understood the basics
Little understanding of signature integrity guarantees
Verifying attack message authenticity was difficult for most users
No group resisted attacks 100% of the time
Color and Color+Briefing resisted new key attack and the unsigned message attack better then No KCM
The interface did not help against new identity attacks

29. Johnny 2 User Test: Conclusions A few surface interface issues
Do not trust button
Misconceptions about the security of sealed messages
Generally, the new interface simplifies encryption
Still problems with determining certificate trust, however some of these problems may be unavoidable.

30. So Now What?
Now its time to do your own test!

31. User Test 3 groups: Take a few minutes to create a simple user test
Cell Phone
CD player
Calculator
Take a few minutes to create a simple user test
One member of each group switches to be a tester

32. User Test Guidance: Decide whose going to do what!
Create a Use Case Scenario
Define user tasks for completion of the scenario
Set up metrics for results evaluation
What qualifies as success vs. failure?

33. User Test
Results!?