1. CS2003 Usability Engineering Usability Evaluation Dr Steve Love

2. Slide 2 Overview Continuing theme of usability evaluation techniques Think aloud protocol Observation Interviews Diary studies Laboratory setting Field studies Summary References Further reading

3. Think aloud protocol The idea of using verbal protocols as a form of usability evaluation was put forward by Ericsson and Simon (1985). Potentially provides information on what the user is actually thinking about when they are interacting with a service. Get feedback on users’ feelings at particular points in the interaction, especially if the system does something they didn’t expect or they ended up somewhere that they didn’t want to be. May find out that the navigation structure is not as clear-cut as you originally thought it was. The key to success in using this method is practice. You must give your participants practice in using the method before data collection begins. How many windows do you have in your house? Slide 3

4. Observation In an observational study, the emphasis is on observing the naturally occurring behaviour of people (with or without them knowing about it). In an HCI context, observational techniques have been used as part of the data collection process in the lab. For example, you can watch people’s reactions as they interact with a new smartphone interface, observing where they have difficulties or which parts of the interface they do not like. Two main types of observation: controlled observation: users are in a controlled lab environment (see example above) naturalistic observation : observing behaviour as it naturally occurs in its normal context of use (e.g. mobile phone use in public places). Slide 4

5. Observation techniques Audiotape recording your own verbal record (recorded onto audiotape) of the behaviour of the people you are observing as it occurs. have to make sure that your recording was unobtrusive and did not distract users from carrying out the tasks that you had asked them to do. people may become uneasy about you looking at them while you are speaking into a small tape recorder at the same time. Videotape recording could provide you with a very rich source of data observing behaviour and reactions in real time however, how would you feel seeing someone pointing a video camera at you in a public place? Slide 5

6. Constructing an observational coding form be clear about what it is you are interested in observing (e.g. mobile phone use in public places.) identify what you think are the dimensions or categories of behaviour that you expect to observe. test it out to see if it is covering the main behaviours that you have highlighted this process will also allow you to add new behaviour categories that you had not previously thought Another way to test the reliability of your coding form is to get a few of your friends to use it (if you persuade them to help you) and record their observations and see how it compares with your own Obviously you would all have to observe the same social scene to do this, at the same time. Slide 6

7. Interviews Questioning an individual can be a quick way of getting information from them on usability issues, requirements gathering and a whole host of mobile HCI research interests. Interviews can be used in the lab at the end of a formal testing session to help find out some more information on what participants thought about the experiment that they had just taken part in. It can also be used to investigate how participants’ interview answers compare to their behaviour which you will have observed during the experiment and also, the performance data that you may have logged during the experiment (e.g. timings for completing each task you asked them to undertake as part of the study). Interviews come in various formats and the following are probably the most commonly used in HCI to illicit information from participants. Slide 7

8. Interview formats Informal interviews totally unstructured with the aim of the interview being to try and collect as much data as possible. Semi-structured interviews you have a set number of questions which are presented in the same order These types of questions could come at the end of a Likert- type questionnaire Structured interviews all the questions are fixed and set beforehand (e.g. how many text messages do you send per week?) There are a set number of responses that respondents can choose from (e.g., ‘please specify how many calls you make per week using your mobile phone: 0–10, 11–20, 21–30, 31–40, other...) Slide 8

9. Diary studies Ask participants to keep a personal record of their thoughts and feelings about interacting with a particular piece of technology; for example, over a certain period of time This can provide a rich source of textual data for you to analyse. The key thing to remember here is that you have to give participants in your diary study specific instructions of what you want them to describe and write about Slide 9

10. Exercise Develop a set of guidelines for participants taking part in a 4 week diary study looking into people’s use of smartphones What are the important aspects of user behaviour that you want participants to record in their diaries? Work in groups for the next 10 minutes Slide 10

11. Lab settings The traditional approach to usability evaluation You can control the environment in order to test your specific hypotheses Allows you to see how they react to using a new piece of mobile technology without anything distracting them from the task in hand. Are there any problems with this approach? Slide 11

12. Field studies does not give you control over all your variables and settings in the same way as a laboratory experiment it can be a very effective form of research method for mobile HCI research mobile devices and applications are used in dynamic social environments (e.g. in a café, on the train). it is important for researchers (and designers) to assess how context of use affects people’s performance and attitude towards using mobile devices and applications. Slide 12

13. Summary There are a variety of usability evaluation methods It is important you choose the method that will be the most appropriate for the aims of your usability study A triangulation approach is often the best way to proceed Slide 13

14. Questions? Slide 14

15. References and further reading Reference Ericsson, K.A. and Simon, H.A. (1985) Protocol analysis: verbal reports as data. Cambridge MA: MIT Press. Further reading Faulkner, X. (2000),Usability Engineering, Chapter 6: Usability Evaluation Palen,L. and Salzman, M (2002) voice-mail diary studies for naturalistic data capture under mobile conditions. Proceedings of CSCW 2002, New Orleans, November 16-29, pp87-95 Slide 15