Think-aloud usability experiments or concurrent verbal accounts Judy Kay CHAI: Computer human adapted interaction research group School of Information Technologies

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Overview Empirical methods Think-aloud Benefits Disadvantages Naturalised think-aloud

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Postconditions for this week (incl studio)‏ Describe the uses of Think Alouds Describe the processes for conducting one Describe advantages and limitations Ability to use Think-aloud as relevant and for your project Justify the use of Think-aloud in the overall testing of a pervasive computing application

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Think aloud protocols –ask user to `think aloud' as they use the interface –often used with video, audio taping –otherwise MUST make notes –helps observer interpret what is going on –gives qualitative data mainly –3-5 users may be enough (Nielsen)‏

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Case study Test usability of console in lecture theatre What is intended functionality? Formulate some concrete tasks – write these as instructions

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Classification of think-aloud formative versus summative predictive v empirical laboratory v naturalistic qualitative v quantitative

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Design cycle User Centred Design Prototype construction Usability study –Define concrete tasks to use in evaluation –Decide just what data to collect –Test design of experiment for timing (trial it)‏ –Recruit users –Run study Goto top

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Recruiting users How representative are they? –similarity to intended user population –Age –Gender –experience in area –interest/motivation –computer literacy What effect does user population have for conclusions?

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Stages of running an evaluation Preparation Introduction The test Questionnaire/interview Debriefing Analysis, reflection, summarising, reporting, conclusions for action Steps 1- 5 done for each user test, as run Step 6 is applied mainly after several users

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Preparation set up room set up machine check machine check user instructions do a mental run-through be sure not to waste user's time because of your lack of preparation

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Introduction Welcome user, explain purpose of test –make clear system tested not user –confidentiality –anonymity of reporting –opt out at any time –what is recorded Invite any other questions to here –explain procedure –if appropriate, do demo –invite questions

HAI2008. Lifelong ambient companions: challenges and steps to overcome them The test User works through experiment.... –recording –ensure user feels supported –show pleasure at problems identified –critical to help user if stuck Questionnaire/interview –open and closed

HAI2008. Lifelong ambient companions: challenges and steps to overcome them What data should you collect? Observe –direct/indirect –take notes –video/audio/software monitor –software logs for timing Questionnaire: –open –closed

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Debriefing Thank user Remind them of usefulness of results Pause to make sure all data collected All notes written May ask user to confirm details collected

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Pitfalls Defining the right concrete tasks –Test all key aspects –Multiple tasks for same aspects Instructions to the users –Do NOT lead the user

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Benefits of think aloud `show what users are doing and why they are doing it while they are doing it in order to avoid later rationalisations' (Nielsen, Usability Engineering, Academic press 1993, p195)‏ cheap slows users down –studies show users work faster with fewer errors due to care on critical elements

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Problems of think aloud not directly quantitative add cognitive load to users user's `theories' must be interpreted with care slows users down

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Facilitating think aloud What are you thinking now? What do you think that message means? (only after the user has noticed the message and is clearly spending time on it)‏ don't help user except with How do you think you can do it? if user appears surprised, Is that what you expected to happen?

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Naturalised think-aloud Multi-user interaction –two (or more) users work on task –conversation is natural –observer collects dialogue

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Problems with observing users Hawthorn effect People rationalise (Telling more than we can know)‏ Qualitative data

HAI2008. Lifelong ambient companions: challenges and steps to overcome them Summary Relatively inexpensive Can identify major flaws And may indicate causes of user problems May give access to user's mental model Alters activity => meaningfulness