1. School of Engineering and Information and Communication Technology KIT305/607 Mobile Application Development Week 7: Usability (think-alouds) Dr. Rainer Wasinger, Dr. James Montgomery School of Engineering and ICT Acknowledgements: The material in these slides is based on a lecture by Prof. Judy Kay (The University of Sydney), as well as other experts in interaction design.

2. School of Engineering and Information and Communication Technology Think-aloud Usability Experiments Overview: Empirical methods Think-aloud Tasks –Abstract versus concrete –Task design and avoiding leading the user Benefits Disadvantages 2

3. School of Engineering and Information and Communication Technology Think-aloud protocols Ask user to “think aloud” as they use the interface Often used with video, audio taping Otherwise MUST make notes Helps observer interpret what is going on Gives qualitative data mainly 3-5 users may be enough (Nielsen)‏ for each stage of refining the prototype 3

4. School of Engineering and Information and Communication Technology Classification of think-aloud Formative versus summative Laboratory vs naturalistic Qualitative vs quantitative 4

5. School of Engineering and Information and Communication Technology Design cycle User Centred Design Define criteria for success Define concrete tasks that users should be able to do - use these in evaluation Prototype construction Usability study –Decide what data to collect –Test design of experiment for timing (trial it)‏ –Recruit users –Run study Iterate / go back to the top 5

6. School of Engineering and Information and Communication Technology Recruiting users How representative are your users? –Similarity to the intended user population –Age –Gender –Experience in area –Interest/motivation –Computer literacy What effect does the user population have for conclusions? 6

7. School of Engineering and Information and Communication Technology Stages of running an evaluation 1.Preparation 2.Introduction 3.The test 4.Questionnaire/interview 5.Debriefing 6.Analysis, reflection, summarising, reporting, conclusions for action Steps 1- 5 are done for each user test Step 6 is applied mainly after several users 7

8. School of Engineering and Information and Communication Technology 1. Preparation Set up the room Set up the computer Check the computer Check the user instructions Do a mental run-through Be sure not to waste the user's time because of your lack of preparation. 8

9. School of Engineering and Information and Communication Technology 2. Introduction Welcome the user; explain the purpose of the test Make clear that the system is being tested not the user Confidentiality Anonymity of reporting Allow the participant to opt out at any time Let them know what is being recorded Invite any other questions they might have Explain the procedure If appropriate, do a demo Invite questions 9

10. School of Engineering and Information and Communication Technology 3. The Test User works through the experiment.... Your role is to: Record Ensure the user feels supported Show pleasure at problems identified If appropriate, help the user if they get stuck 10

11. School of Engineering and Information and Communication Technology 4. Questionnaire/Interview Questionnaire/interview Open and closed questions. 11

12. School of Engineering and Information and Communication Technology What data should you collect? Observe Direct/indirect Take notes Video/audio/software monitor Software logs for timing Questionnaire: Open Closed 12

13. School of Engineering and Information and Communication Technology 5. Debriefing Thank the user Remind them of the usefulness of their results Pause to make sure all data is collected All notes written May ask user to confirm details collected 13

14. School of Engineering and Information and Communication Technology Pitfalls Defining the right concrete tasks –Test all key aspects –Multiple tasks for same aspects Instructions to the users –Do NOT lead the user –Take particular care not to use words that are identical to terms on the interface 14

15. School of Engineering and Information and Communication Technology Benefits of think-aloud “show what users are doing and why they are doing it while they are doing it in order to avoid later rationalisations” (Nielsen, Usability Engineering, Academic press 1993, p195)‏ Cheap Slows users down –Studies show users may work faster with fewer errors due to care on critical elements 15

16. School of Engineering and Information and Communication Technology Problems of think-aloud Not directly quantitative Adds cognitive load to users User's “theories” must be interpreted with care Slows users down Users are aware they are being observed so may behave accordingly 16

17. School of Engineering and Information and Communication Technology Facilitating think-aloud What are you thinking now? What do you think that message means? (only after the user has noticed the message and is clearly spending time on it)‏ Don't help the user except with How do you think you can do it? If the user appears surprised, Is that what you expected to happen? 17

18. School of Engineering and Information and Communication Technology Problems with observing users Hawthorne effect People rationalise (telling more than we can know)‏ Qualitative data 18

19. School of Engineering and Information and Communication Technology Summary Relatively inexpensive Can identify major flaws And may indicate causes of user problems May give access to a user's mental model 19

20. School of Engineering and Information and Communication Technology Case study Test usability of the lecturer’s data projection facilities in the classroom. What is the intended functionality? Formulate some concrete tasks – write these as instructions e.g.: –You want to display the screen of your mobile phone to the whole class. 20

21. School of Engineering and Information and Communication Technology Case study Work in groups to identify 3 tasks relevant to testing the teacher’s use of projection facilities in this room. 21

22. School of Engineering and Information and Communication Technology Call for volunteers to be users for this trial Have not used these facilities before Have used them in other places in the uni Call for volunteers to conduct the trial All will make helpful notes 22 Case study

23. School of Engineering and Information and Communication Technology Case study Single sentence statement for lecture room control system: –Lecturers, without prior training, can use the control interface to make effective use of the lecture room facilities, as needed in a lecture (within 2 minutes). 23

24. School of Engineering and Information and Communication Technology Requirements Lecturers, can, without prior training, and within 2 minutes, use the control interface to: 1.Display a physical resource (such as a sheet of paper) to the class 2.Increase the audio level of the lectern microphone 3.Set the lighting for showing a video 4.… 24

25. School of Engineering and Information and Communication Technology Example tasks Assume: initial state of theatre is with data projectors showing internal computer. props: a mobile phone the user is asked to pretend they brought to the class, and a book A. Show the class the screen of your mobile phone. B. Adjust the audio so that the class can hear you. C. Alter the lighting to show a video to best effect. 25

26. School of Engineering and Information and Communication Technology Example task design Task \ RequirementR1 show resource R2 audioR3 light A. Show the class the screen of your mobile phone. X B. Adjust the audio so that the class can hear you. X C. Alter the lighting to show a video to best effect. XX 26