1. User studies

2. Why user studies? How do we know security and privacy solutions are really usable? Have to observe users! –you may be surprised by what users really do –you are not your users

3. Typical Security Evaluation Does indicator behave correctly when not under attack? Does indicator behave correctly when not under attack? –No false positives or false negatives Does it behave correctly when under attack? Does it behave correctly when under attack? Can it be spoofed or obscured? Can it be spoofed or obscured? Wrong indicator Attacker redirects Correct indicator

4. Usability evaluation questions Do users notice it? Do users notice it? –“What lock icon?”

5. Do users know what it means? Netscape SSL icons Cookie flag IE6 cookie flag Firefox SSL icon

6. Do users know what to do when they see it?

7. Other usability questions Are they motivated to take action? Are they motivated to take action? And do they actually do it? And do they actually do it? How about over the long term? How about over the long term?

8. Why Johnny Can’t Encrypt Whitten and Tygar, 1999 Whitten and Tygar, 1999 A Usability Evaluation of PGP 5.0 A Usability Evaluation of PGP 5.0 –Pretty Good Privacy –Software for encrypting and signing data –Plug-in provides “easy” use with email clients –Modern GUI, well designed by most standards

9. Evaluation Methodology Motivation: Security software may require additional usability considerations Motivation: Security software may require additional usability considerations Question: Is PGP usable by everyday users? Question: Is PGP usable by everyday users? Method: Cognitive Walkthrough + User Study Method: Cognitive Walkthrough + User Study Goal: demonstrate usability problems Goal: demonstrate usability problems Question: is method appropriate?

10. Defining usable security software Security software is usable if the people who are expected to use it: Security software is usable if the people who are expected to use it: 1.are reliably made aware of the security tasks they need to perform. 2.are able to figure out how to successfully perform those tasks 3.don't make dangerous errors 4.are sufficiently comfortable with the interface to continue using it.

11. The studies Cognitive Walkthrough: – –Tasks: encrypting and signing email, decrypting, etc. User Study –PGP 5.0 with Eudora –12 participants all with at least some college and none with advanced knowledge of encryption –Participants were given a scenario with tasks to complete within 90 min –Tasks built on each other –Participants could ask some questions through email

12. Cognitive Walkthrough results Visual metaphors Visual metaphors –Public vs. Private keys –Signatures and verification Key server Key server –Hidden? What is it doing? –Revocation not automatic Several irreversible actions Several irreversible actions –Can cause serious errors Consistency Consistency Too much information Too much information –More unneeded confusion

13. User Study Results 3 users accidentally sent the message in clear text 3 users accidentally sent the message in clear text 7 users used their public key to encrypt and only 2 of the 7 figured out how to correct the problem 7 users used their public key to encrypt and only 2 of the 7 figured out how to correct the problem Only 2 users were able to decrypt without problems Only 2 users were able to decrypt without problems Only 1 user figured out how to deal with RSA keys correctly. Only 1 user figured out how to deal with RSA keys correctly. A total of 3 users were able to successfully complete the basic process of sending and receiving encrypted emails. A total of 3 users were able to successfully complete the basic process of sending and receiving encrypted emails. One user was not able to encrypt at all One user was not able to encrypt at all

14. Conclusion None of their defined usability goals were met. None of their defined usability goals were met. Question: Is this a failure in the design of the PGP 5.0 interface or is it a function of the problem of traditional usable design vs. design for usable secure systems?

15. Kazaa File Sharing Study Motivation: Lots of people use P2P file sharing Motivation: Lots of people use P2P file sharing Problem: Seems like lots of people sharing files accidentally. Why? Problem: Seems like lots of people sharing files accidentally. Why? Method Method –Cognitive walkthrough –User study 12 users, 10 had used file sharing before 12 users, 10 had used file sharing before Questionnaire for file sharing understanding Questionnaire for file sharing understanding Task: figure out what files are being shared by Kazaa (Answer: Download files set to C:\ so all files on the C:\ drive) Task: figure out what files are being shared by Kazaa (Answer: Download files set to C:\ so all files on the C:\ drive)

16. Their usability criteria Peer-to-peer file sharing software is safe and usable if users: Peer-to-peer file sharing software is safe and usable if users: –Are clearly made aware of what files are being offered for others to download –Are able to determine how to share and stop sharing files successfully –Do not make dangerous errors that can lead to unintentionally sharing private files –Are comfortable with what is being shared with others and confident that the system is handling this correctly

17. Cognitive Walkthrough Results Multiple names for similar things Multiple names for similar things –My Shared Folder, My Media, My Kazaa, Folder for downloaded files Downloaded files are also shared Downloaded files are also shared Kazaa recursively shares sub-folders Kazaa recursively shares sub-folders Easy to add directories to share, difficult to remove Easy to add directories to share, difficult to remove

18. User Study Results 5 people thought it was “My Shared Folder” 5 people thought it was “My Shared Folder” –which one UI did suggest 2 people used Find Files to find all shared files 2 people used Find Files to find all shared files –This UI had no files checked, thus no files shared? 2 people used help, said “My Shared Folder” 2 people used help, said “My Shared Folder” 1 person couldn’t figure it out at all 1 person couldn’t figure it out at all Only 2 people got it right Only 2 people got it right

19. Generalizing results Design suggestions: Design suggestions: –Only allow sharing of multimedia files –Better feedforward –Allow exceptions to recursively shared folders

20. A very different study Motivation: Online social networking widespread. Motivation: Online social networking widespread. Problem: People sharing large amounts of personal information, which puts them at risk for variety of problems Problem: People sharing large amounts of personal information, which puts them at risk for variety of problems Questions: Questions: –how and why do users share and protect their information? –how do they form impressions of other profiles? Goal: Identify requirements, issues and challenges in improving privacy in online communities Goal: Identify requirements, issues and challenges in improving privacy in online communities

21. Method User study of Facebook.com User study of Facebook.com –16 college participants from psych pool –Logged into own profile, information and privacy settings noted –Interviewed about their own profile: their motivations for entering information, how they formed their social networks, their concerns over others viewing their profiles, etc. their motivations for entering information, how they formed their social networks, their concerns over others viewing their profiles, etc. –View 4 other profiles, and interviewed about impressions

22. Big picture results Users thinking about their privacy mainly during initial activation – while filling out initial profile information Users thinking about their privacy mainly during initial activation – while filling out initial profile information Neglect privacy implications of later interactions – interacting with friends, not thinking about the broader audience at that point… Neglect privacy implications of later interactions – interacting with friends, not thinking about the broader audience at that point… …Until a negative experience occurs …Until a negative experience occurs Need new mechanisms to increase awareness of the accessibility of their profile and their risks – especially during everyday activities. Need new mechanisms to increase awareness of the accessibility of their profile and their risks – especially during everyday activities. Need new ways to more easily adjust privacy settings during those everyday activities. Need new ways to more easily adjust privacy settings during those everyday activities.

23. Summing it Up Examples of how to run user studies Examples of how to run user studies –Not the most rigorous studies, but good enough to demonstrate main point Tradeoffs of various methods? Tradeoffs of various methods? How to choose methods? How to choose methods?

24. Your Observations Where did you observe? Where did you observe? What were some general observations? What were some general observations? What problems did people have? What problems did people have? Any privacy or security implications? Any privacy or security implications? What did you think of being an observer? What did you think of being an observer?

25. Now let’s practice 4 groups: 4 groups: –Voice recorder –Camera –PDA (2) Take a few minutes to design a simple user study Take a few minutes to design a simple user study –What questions to you have? –What are your usability goals? –What methods? –Use one member as tester if you can

26. User Test Results!? Results!?