Margje van de WielFaculty of Psychology, Universiteit Maastricht Thinking-Aloud & Protocol Analysis Margje van de Wiel Department of Experimental Psychology Universiteit Maastricht Unit Macro-Ergonomics: Evaluating human work

Margje van de WielFaculty of Psychology, Universiteit Maastricht Overview Research on cognitive processes underlying cognitive task performance: –Problem solving –Knowledge representation Focus on the think-aloud method –Collecting think-aloud protocols –Analyzing think-aloud protocols

Margje van de WielFaculty of Psychology, Universiteit Maastricht Applications of cognitive psy- chological research methods Fundamental psychological research –Explorative –Testing theoretical hypotheses –Compare tasks or groups Cognitive task analysis Building knowledge based systems Usability research

Margje van de WielFaculty of Psychology, Universiteit Maastricht Usability research How easy is it for people to set the time and alarm on a clock without a manual? How easy is it for them with the manual? Do people understand the manual? Do they use the manual? Video, computer program, web site etc. è Compare conditions or groups

Margje van de WielFaculty of Psychology, Universiteit Maastricht Problem solving Tower of Hanoi The objective of this game is to move all of the rings to the right most peg. You may only move one ring at a time; you must never allow a large ring to rest on a smaller ring.

Margje van de WielFaculty of Psychology, Universiteit Maastricht Problem space (Newell & Simon, 1972) Initial state Goal state Subgoal state Subgoal state Inter- mediate states Inter- mediate states Inter- mediate states Inter- mediate states

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research on problem solving Product evaluation Observation and registration of behavior (real-life, video, log files) But Gives no direct information on cognitive processes

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research on problem solving Introspection = Direct observation of mind But Participants may give their interpretation of their cognitive processes when they prefer during task performance

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research on problem solving Retrospection But Participants may give their interpretation of their cognitive processes after task performance

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research on problem solving Free recall Stimulated recall But Participants give their memories of the thoughts they had during problem solving after task performance

Margje van de WielFaculty of Psychology, Universiteit Maastricht Instructions Retrospection –How did you proceed in solving the problem? –What did you think during solving the problem? Stimulated / Free recall –Report everything you can remember about your thoughts during solving the problem

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research on problem solving Questioning and prompting But Participants may give their interpretation of their cognitive processes when prompted so that natural task performance may be disturbed

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research on problem solving Dialogues in explanation and collaboration But The cognitive processes might be different than in natural task performance

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research on problem solving Thinking aloud = Saying everything that comes to mind Advantages: Concurrent verbalization Close to natural task performance Discourages interpretation and explanation

Margje van de WielFaculty of Psychology, Universiteit Maastricht Theoretical framework / verbalization theory (Ericsson & Simon, 1993) =the information processing model LTM Verbal report STM Task performance Input info

Margje van de WielFaculty of Psychology, Universiteit Maastricht Theoretical framework Verbal report STM Task performance Verbal encoding Vocalization Type of task - modality - verbalizability - instruction

Margje van de WielFaculty of Psychology, Universiteit Maastricht The type of problem and the role of knowledge Well-defined problemsIll-defined problems Weak problem solving methods Strong problem solving methods NovicesExperts Knowledge lean taskKnowledge rich task

Margje van de WielFaculty of Psychology, Universiteit Maastricht Focus on knowledge representation The nature and organization of knowledge is central in problem solving and not the strategies used (e.g. Chi, 1997) è The representation of a problem determines the problem solving process

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research on knowledge representation Same methods as for investigating cognitive processes, but analysis of the knowledge used Free recall –Report everything you remember about... =Reflection of the representation of (prior) knowledge, a text or a problem in the mind

Margje van de WielFaculty of Psychology, Universiteit Maastricht The think-aloud method Collecting think-aloud protocols Transcribing the verbal protocols è Verbal reports as data Segmenting the transcribed protocols Analyzing the transcribed protocols

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research questions è Gaining insight in cognitive processes and knowledge structures Explorative Testing hypotheses Comparing types of problems Comparing groups

Margje van de WielFaculty of Psychology, Universiteit Maastricht Collecting think-aloud protocols What do you need? Experimenter Participant Problem Tape-recorder and tape Instructions

Margje van de WielFaculty of Psychology, Universiteit Maastricht Instructions Make participant feel comfortable Explain the purpose of the research Explain the task Explain thinking aloud Emphasize that it is no test situation Emphasize that the data are confidential Practice problem(s) and talking out loud Give feedback

Margje van de WielFaculty of Psychology, Universiteit Maastricht Transcribing the protocols Type out the protocols as literally as possible, including –Natural punctuation –Eh, ehs –Thinking pauses; with dots or even better to measure length –Indicate when something is said mumbling so you don’t understand well Indicate interventions of experimenter

Margje van de WielFaculty of Psychology, Universiteit Maastricht Example Tower of Hanoi (Anzai & Simon, 1979) I’m not sure, but first I’ll take 1 from A and place it on B. And I’ll take 2 from A and place it on C. And then, I take 1 from B and place it n C. Experimenter: If you can, tell me where you place it there? Because there was no place else to go, I had to place 1 from B and C. Then, next, I placed 3 from A to B. …..

Margje van de WielFaculty of Psychology, Universiteit Maastricht Segmenting the protocol Use natural pauses indicated by punctuation Indicate each segment by a line number No interpretation

Margje van de WielFaculty of Psychology, Universiteit Maastricht Example Tower of Hanoi (Anzai & Simon, 1979) 1. I’m not sure, 2. but first I’ll take 1 from A and place it on B. 3. And I’ll take 2 from A and place it on C. 4. And then, I take 1 from B and place it on C. Experimenter: If you can, tell me where you place it there? 5. Because there was no place else to go,.. …..

Margje van de WielFaculty of Psychology, Universiteit Maastricht Analyzing verbal protocols (Transcribed) and segmented verbal protocols Interpretation RAW DATA DATA

Margje van de WielFaculty of Psychology, Universiteit Maastricht Interpretation Qualitative analysis Analyzing qualitative data in an objective and quantifiable way è Compare experimental conditions –Groups –Tasks –Instructions

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research question --> qualitative analysis What comprehension strategies do skilled readers use in reading a text? è Thinking aloud while reading a text è Analysis: What strategies are used? –Exploratory: describe the strategies used by the participants

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research questions --> testing hypotheses (Rauenbush & Bereiter, 1991) What strategies do skilled readers use in dealing with degraded text? Are these comparable to comprehension strategies used in reading normal text? è Thinking aloud while reading text with 3 types of instruction: –Just reading –+ ‘Adults skim this type of material first’ –Skim text for 60 sec and report the topic

Margje van de WielFaculty of Psychology, Universiteit Maastricht Research questions --> testing hypotheses (Rauenbush & Bereiter, 1991) è Analysis: what strategies are used? Normal comprehension strategies –Rereading –Reading ahead –Summing up Other comprehension strategies –(Letter generating strategy) –(Determining word type)

Margje van de WielFaculty of Psychology, Universiteit Maastricht Analyzing verbal protocols 1 Hypotheses Coding scheme Verbal protocols Explore Test Psychological theory Cognitive task analysis

Margje van de WielFaculty of Psychology, Universiteit Maastricht Analyzing verbal protocols 1 Hypotheses Coding scheme Verbal protocols Problem solving Knowledge representation processes (E&S, 93) (Chi, 97)

Margje van de WielFaculty of Psychology, Universiteit Maastricht Cognitive task analysis What knowledge is available in the problem? What knowledge is necessary for problem solving? What steps need to be taken to solve the problem? What is the most efficient problem solving process? What strategies can be used?

Margje van de WielFaculty of Psychology, Universiteit Maastricht Coding scheme Operationalizing hypotheses in coding categories Clearly defining coding categories and providing examples from pilot protocols Categories may differ in level of detail –Segments –Aggregations of segments

Margje van de WielFaculty of Psychology, Universiteit Maastricht Analyzing verbal protocols 2 Code the verbal protocols Determine interrater reliability (% of correspondences, kappa, correlation) Depict the results Seek and interpret patterns in results Eventually Repeat process

Margje van de WielFaculty of Psychology, Universiteit Maastricht Example medical diagnosis Clinical case

Margje van de WielFaculty of Psychology, Universiteit Maastricht Example of a case A 70-year old female is admitted into hospital because of increasing shortness of breath. She has been very tired lately and tolerates her food badly. Sometimes she has chest pain, especially after dinner. Et cetera

Margje van de WielFaculty of Psychology, Universiteit Maastricht Example medical diagnosis Clinical case Thinking aloud while diagnosing the case –Problem solving process –The nature of the knowledge used

Margje van de WielFaculty of Psychology, Universiteit Maastricht Analysis of think-aloud protocol Problem solving process Data examination Data exploration Data explanation Hypothesis generation Hypothesis evaluation Discrepancy processing Meta-reasoning Summarization Nature of knowledge used Data interpretation Clinical reasoning Biomedical reasoning Hypotheses (C or B) (Repetitions) Plans for actions Metacognitive remarks

Margje van de WielFaculty of Psychology, Universiteit Maastricht Example medical diagnosis Clinical case Thinking aloud while diagnosing the case –Problem solving process –The nature of the knowledge used Free recall –Case representation

Margje van de WielFaculty of Psychology, Universiteit Maastricht Analysis of free recall protocols Proposition analysis Number of correct propositions Number of inferences (summarizing two or more propositions)

Margje van de WielFaculty of Psychology, Universiteit Maastricht Proposition analysis Proposition = single idea unit Concept Qualifier Causal Conditional Attribute Specification Is a

Margje van de WielFaculty of Psychology, Universiteit Maastricht Propositions in the case Female - 70-year old (attribute) - is admitted into hospital (attribute) - (because of) shortness of breath (attribute) - increasing (specification) - very tired (attribute) - lately (specification) - tolerates her food badly (attribute)

Margje van de WielFaculty of Psychology, Universiteit Maastricht Other literature Chi, M.T.H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The journal of the learning sciences, 6(3), SGT 3179 Ericcsson, K. A. & Simon, H. A. (1993). Protocol analysis: verbal reports as data. Cambridge, MA: MIT Press (ch 6 & 7). SL BF 180

Margje van de WielFaculty of Psychology, Universiteit Maastricht Other literature Reitman Olson, J. & Biolsi, K. J. (1991). Techniques for representing expert knowledge. In: K. A. Ericsson & J. Smith (Eds.) Toward a general theory of expertise: Prospects and Limits (pp ). Cambridge: Cambridge University Press. ME EAM 374 / Margje