1. Week 3 Judy Kay CHAI: Computer human adapted interaction research group School of Information Technologies

2. 1. Reflections on work so far Core + advanced readings

3. Progress on e-text Week 3: – Think-alouds as start of Assignment 1Assignment 1 Week 2: – Lab consolidation + create affinity diagram Week 1: – Studying use of e-textbook, based on auto- ethnography + observation of other user – System concept statement for e-textbook

4. 2. Reading discussion

5. Questions to consider Form groups – of ~5 all members – within tute class and degree level, – and then split into 2 subgroups Compare your concept maps within groups of 2-3 – Important common elements – Important differences – Annotate each map * for common !! for diferences Then repeat, sharing with other subgroup Be ready to share this in whole class activity

6. Think-alouds The usability tool

7. "Thinking aloud may be the single most valuable usability engineering method." I wrote this in my 1993 book, Usability Engineering, and I stand by this assessment today. The fact that the same method has remained #1 for 19 years is a good indication of the longevity of usability methods.Usability Engineering Thinking Aloud: The #1 Usability Tool by Jakob Nielsen on January 16, 2012Jakob Nielsen

8. When to Use Which User-Experience Research Methods by Christian Rohrer on October 12, 2014Christian Rohrer http://www.nngroup.com/articles/which-ux-research-methods/

9. Demonstrating think-aloud Demonstrate Thinking Aloud by Showing Users a Video by Jakob Nielsen on September 1, 2014Jakob Nielsen Topics: User TestingUser Testing

10. Services Userzoom “can run unmoderated task-based studies with geographically dispersed participants over any web- based interface (website, prototype, mock-up). Participants take the study simultaneously, in their natural context, using their own PC or device.” Users think User testing – Let’s watch their video, noting the tasks and critiquing them (just one in class) Let’s watch their video

11. Facilitating think aloud makes you – experimenter -- really valuable What are you thinking now? What do you think that message means? (only after the user has noticed the message and is clearly spending time on it)‏ don't help user except with How do you think you can do it? if user appears surprised, Is that what you expected to happen?

12. So now, onto Assignment 1

13. First things first Do we have the tasks right? Abstract tasks Concrete instances of them – Concrete? – Relevant? – Not lead the user? – Minimalist? – Good coverage?

14. More preparation For all user studies Later in the semester, we will revise these for Assignment 2 and a different interface

15. Recruiting users How representative are they? –similarity to intended user population –Age –Gender –experience in area –interest/motivation –computer literacy What effect does user population have for conclusions?

16. How many users? Why You Only Need to Test with 5 Users by Jakob Nielsen on March 19, 2000Jakob Nielsen http://www.nngroup.com/articles/why-you- only-need-to-test-with-5-users/

17. Insights: 0 users? 1 user? 2 users? http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

18. What does this really mean? If first test shows catastrophic problems, should you still do 5? Never bother with more than 5? 80% is good enough? NO! NO!

19. It is all about budgets, user groups…. If your interface is to be used by very different groups of people, you need to do the think- aloud with each group eg. – Children – Elderly – Different cultures, languages…… Iterate!!!!! – Use same budget of effort to test each iteraction

20. Other views Spool, J., & Schroeder, W. (2001, March). Testing web sites: Five users is nowhere near enough. In CHI'01 extended abstracts on Human factors in computing systems (pp. 285- 286). ACM.

21. Lewis, J. R. (2014). Usability: Lessons Learned… and Yet to Be Learned. International Journal of Human-Computer Interaction, 30(9), 663-684.” “The consistent finding … has been that observers, individually or in teams, who evaluated the same product discovered very different sets of usability problems.”

22. “In the early 1990s … fairly large-sample formative usability studies … determined that the … first five participants observed in a formative usability study should usually reveal about 85% of the problems available for discovery in that iteration, where the properties of the study (type of participants and tasks employed) place limits on what is discoverable. But over time, in the minds of many usability practitioners, the rule became simplified to “All you need to do is watch five people to find 85% of a product’s usability problems.”

23. Summative evaluation As in your Assignment 1 How many participants? Number of team members * 4 – Why? User populations? – Within community interested in UX – Those who are not – Ages, broad range

24. Stages of running an evaluation 1.Preparation 2.Introduction 3.The test 4.Questionnaire/interview 5.Debriefing 6.Analysis, reflection, summarising, reporting, conclusions for action Steps 1- 5 done for each user test, as run Step 6 is applied mainly after several users

25. Preparation Materials for consent Set up machine, room, environment Check all of them Check user instructions Do a mental run-through Be sure not to waste user's time because of your lack of preparation!

26. Introduction Welcome user, explain purpose of test –make clear system tested not user –confidentiality –anonymity of reporting –opt out at any time –what is recorded Invite any other questions to here –explain procedure –if appropriate, do demo –invite questions

27. The test User works through experiment.... –recording –ensure user feels supported –show pleasure at problems identified –critical to help user if stuck Questionnaire/interview –open and closed

28. What data should you collect? Observe –direct/indirect –take notes –video/audio/software monitor –software logs for timing Questionnaire: –Open –Best things about –If you could change one thing about, what would it be? –Closed (later in the semester only)

29. Debriefing Thank user Remind them of usefulness of results Pause to make sure all data collected All notes written May ask user to confirm details collected

30. Pitfalls Defining the right concrete tasks –Test all key aspects –Multiple tasks for same aspects Instructions to the users –Do NOT lead the user –Take particular care not to use words that are identical to terms on the interface

31. Benefits of think aloud “show what users are doing and why they are doing it while they are doing it in order to avoid later rationalisations” (Nielsen, Usability Engineering, Academic press 1993, p195)‏ Cheap Slows users down –studies show users may work faster with fewer errors due to care on critical elements

32. Problems of think aloud Unnatural context and situation (do you talkto yourself?) People filter, want to please, do not want to look foolish or inept Hawthorn effect Experimenter can bias results Directly eg via task choice Inadvertently eg gasp, brief frown Not directly quantitative Add cognitive load to users User's “theories” must be interpreted with care Slows users down Users are aware they are being observed so behave accordingly

33. Naturalised think-aloud Multi-user interaction –Two (or more) users work on task –Conversation is natural –Observer collects dialogue

34. Summary Top method for formative evaluations Relatively inexpensive Can identify major flaws And may indicate causes of user problems May give access to user's mental model Alters activity => meaningfulness This is a major part of your assignment

35. Homework Get started on Assignment 1 Homework is to work with your group to get started on the assignment Define the tasks Split the work Think-Alouds on e-textbook evaluation