COMP5047 Pervasive Computing: 2012 Think-aloud usability experiments or concurrent verbal accounts Judy Kay CHAI: Computer human adapted interaction research group School of Information Technologies

Overview Empirical methods Think-aloud Benefits Disadvantages Naturalised think-aloud

Postconditions for this week Describe the uses of Think Alouds Describe the processes for conducting one Describe advantages and limitations Ability to use Think-aloud as relevant and for your project Justify the use of Think Aloud in the overall testing of a pervasive computing application Be ready to do Think Aloud evaluations for your part of your project

Think aloud protocols Ask user to “think aloud” as they use the interface Often used with video, audio taping Otherwise MUST make notes Helps observer interpret what is going on Gives qualitative data mainly 3-5 users may be enough (Nielsen)‏ for each stage of refining the prototype

Case study Test usability of teacher’s data projection facilities in this classroom What is intended functionality? Formulate some concrete tasks – write these as instructions –eg Suppose you currently have the projector screen down but you want to use the whiteboard. So you want make the screen go up.

Case study Work in your groups to identify 3 tasks relevant to testing the teacher’s use of projection facilities in this room

Case study Call for volunteers to be users for this trial Have not used these facilities before Have used them in other places in the uni Call for volunteers to conduct the trial All will make helpful notes

Classification of think-aloud formative versus summative predictive v empirical laboratory v naturalistic qualitative v quantitative For the project, you goal will be?

Design cycle User Centred Design Define criteria for success Define concrete tasks users should be able to do - use these in evaluation Prototype construction Usability study –Decide just what data to collect –Test design of experiment for timing (trial it)‏ –Recruit users –Run study Goto top

Recruiting users How representative are they? –similarity to intended user population –Age –Gender –experience in area –interest/motivation –computer literacy What effect does user population have for conclusions?

Stages of running an evaluation 1.Preparation 2.Introduction 3.The test 4.Questionnaire/interview 5.Debriefing 6.Analysis, reflection, summarising, reporting, conclusions for action Steps 1- 5 done for each user test, as run Step 6 is applied mainly after several users

Preparation Set up machine, room, environment Check all of them Check user instructions Do a mental run-through Be sure not to waste user's time because of your lack of preparation!

Introduction Welcome user, explain purpose of test –make clear system tested not user –confidentiality –anonymity of reporting –opt out at any time –what is recorded Invite any other questions to here –explain procedure –if appropriate, do demo –invite questions

The test User works through experiment.... –recording –ensure user feels supported –show pleasure at problems identified –critical to help user if stuck Questionnaire/interview –open and closed

What data should you collect? Observe –direct/indirect –take notes –video/audio/software monitor –software logs for timing Questionnaire: –open –closed

Debriefing Thank user Remind them of usefulness of results Pause to make sure all data collected All notes written May ask user to confirm details collected

Pitfalls Defining the right concrete tasks –Test all key aspects –Multiple tasks for same aspects Instructions to the users –Do NOT lead the user –Take particular care not to use words that are identical to terms on the interface

Benefits of think aloud “show what users are doing and why they are doing it while they are doing it in order to avoid later rationalisations” (Nielsen, Usability Engineering, Academic press 1993, p195)‏ Cheap Slows users down –studies show users may work faster with fewer errors due to care on critical elements

Problems of think aloud Not directly quantitative Add cognitive load to users User's “theories” must be interpreted with care Slows users down Users are aware they are being observed so behave accordingly

Facilitating think aloud What are you thinking now? What do you think that message means? (only after the user has noticed the message and is clearly spending time on it)‏ don't help user except with How do you think you can do it? if user appears surprised, Is that what you expected to happen?

Naturalised think-aloud Multi-user interaction –Two (or more) users work on task –Conversation is natural –Observer collects dialogue

Problems with observing users Hawthorn effect People rationalise (Telling more than we can know)‏ Qualitative data

Class activity - foreshadowed Formulate three concrete tasks for testing the console of the lecture theatre Write these out ready for use in an experiment After next part of the seminar: each group will conduct one practice Think-Aloud with the rest of the class writing feedback on how this was done.

Summary Relatively inexpensive Can identify major flaws And may indicate causes of user problems May give access to user's mental model Alters activity => meaningfulness