1. Think-Aloud Protocols February 5, 2010

2. Today’s Class Probing Question Think-Aloud Protocols Assignments

3. Probing Question for Friday, February 5 Let’s say that the NSF has just given you a $100,000 grant to study student note-taking during classroom lecture – i.e. you can assign one minion to the task full-time for a year Which qualitative method would you prefer to use, and why? – What factors would you be especially attentive to, to ensure validity of your results?

4. Today’s Class Probing Question Think-Aloud Protocols Assignments

5. I need a volunteer A fast typist To try to record things people say while thinking aloud Only record the person thinking aloud

6. I need two more volunteers

7. Volunteer #1: Will think-aloud Volunteer #2: Will run the think-aloud

8. What should Volunteer #2 remember while running the think-aloud?

9. Use appropriate instructions Prompt the subject Write down interesting events Focus on what the subject does and says, not “why” Don’t help the subject

10. Long division Volunteer #1 will think-aloud Volunteer #2 will run the think-aloud So, Volunteer #2, give Volunteer #1 some instructions…

11. Long Division 16, 429, 782, 6007140

12. How did it go? Volunteer #1: How hard was the task? Did thinking aloud make it harder?

13. How did it go? Volunteer #2: How hard was it to get him/her to think-aloud? Did you worry about disrupting him or her? Audience: Any (friendly and constructive!) criticism for Volunteer #2?

14. Thoughts/comments From anyone On why this task might have been difficult/easy for the person thinking aloud, or the experimenter (or anything else)

15. Two more volunteers!

16. Drawing a scatterplot Volunteer #1 will think-aloud Volunteer #2 will run the think-aloud

17. Draw a scatterplot of this fake data CityPopulation (in 1000) Number of Brazilian Restaurants Worcester1554 Fitchburg650 Boston6506 Providence1500 Springfield701 Manchester1302 Hartford2204 New Haven1200 New Bedford553 Arapiraca, Brazil14080

18. How did it go? Volunteer #1: How hard was the task? Did thinking aloud make it harder?

19. How did it go? Volunteer #2: How hard was it to get him/her to think-aloud? Did you worry about disrupting him or her? Audience: Any (friendly and constructive!) criticism for Volunteer #2?

20. Thoughts/comments From anyone On why this task might have been difficult/easy for the person thinking aloud, or the experimenter (or anything else)

21. Two more volunteers!

22. The 9-Dot Problem (Kershaw & Ohlsson, 2004)

23. Volunteer #2 will connect all 9 points with just 4 straight lines (you don’t already know the answer, right?)

24. The 9-dot problem

25. How did it go? Volunteer #2: How hard was the task? Did thinking aloud make it harder?

26. How did it go? Volunteer #1: How hard was it to get him/her to think-aloud? Did you worry about disrupting him or her? Audience: Any (friendly and constructive!) criticism for Volunteer #1?

27. Thoughts/comments From anyone On why this task might have been difficult/easy for the person thinking aloud, or the experimenter (or anything else)

28. Insight Problems The biggest problem with thinking-aloud as a technique. When people are trying to figure something out, and it requires “insight”, thinking-aloud can be very disruptive. Therefore, if your participant is really baffled, Now might not be exactly the time to push them to “KEEP TALKING” In general, it’s best to avoid using think-aloud in these situations What else might you use to study process in these situations?

29. Two more volunteers! One more time

30. Trogdor! http://www.homestarrunner.com/trogdor.html

31. How did it go? Volunteer #1: How hard was the task? Did thinking aloud make it harder?

32. How did it go? Volunteer #2: How hard was it to get him/her to think-aloud? Did you worry about disrupting him or her? Audience: Any (friendly and constructive!) criticism for Volunteer #2?

33. Thoughts/comments From anyone On why this task might have been difficult/easy for the person thinking aloud, or the experimenter (or anything else)

34. OK enough for now What have we learned about think-alouds from all of this? Bonus points for expressing your thought as a Beyer & Holtzblatt-style aphorism

35. What could you do? If you want to do a think-aloud on some uninterruptable process?

36. Example Take, for example, a helicopter pilot who needs to drop a ladder to someone while still flying

37. Crashing computer OK, Crashing helicopter NOT OK

38. What could you do?

39. Task-splitting You might want to do a think-aloud of the pilot’s processes as (s)he does each of these tasks separately: ie, a think-aloud as (s)he lowers a ladder, and a think-aloud as (s)he flies the plane This would show you the pilot’s major sub- goals, decision-points, and evaluation criteria for each task

40. Retrospective Think-Aloud Videotape the participant as they are doing the task Take a 10 minute break Then show the video to the participant and have them do colour-commentary. Take notes on their colour-commentary (Or even tape-record it)

41. Retrospective Think-Aloud is Also Useful… When you are studying someone who is so expert at an action, that their actions are completely automatized. In this case, thinking-aloud can disrupt or change someone’s behavior.

42. Two more volunteers!

43. Volunteer #1 Volunteer #1: Please tell me the cash value of the change you have in your pocket. Don’t think aloud.

44. Volunteer #2 Volunteer #2: Please THINK ALOUD while you determine the cash value of the change you have in your pocket.

45. Volunteer #2 Volunteer #2: Please THINK ALOUD while you determine the cash value of the change you have in your pocket. Volunteer #1, is this the procedure you used?

46. Thoughts? Questions?

47. Analysis

48. Let’s take the longest of the protocols that our noble live transcriber has produced Can you email it to me? – I’ll put it up on the screen

49. Segmentation It’s better to do segmentation from an audio recording than a transcript in general, but for this time…

50. Segmentation Your thoughts on where we should segment?

51. Cognitive Processes What are the cognitive processes found in these utterances?

52. Cognitive Processes Do we have an “irrelevant to task” category? – Recommended by van Someren

53. Coding Let’s code the whole transcript with regard to the list of cognitive processes For each segment, quickly write down your code, and then read it out loud – What level of agreement do we obtain? – Why do we disagree?

54. Question Let’s think about the coding scheme we used How did the coding scheme in Gobert (1999) differ from this?

55. Question Let’s think about the coding scheme we used How did the coding scheme in Gobert (1999) differ from this? Atheoretical.vs. theoretical

56. Advantages/Disadvantages What are the advantages and disadvantages of – atheoretical coding schemes – theoretical coding schemes

57. Thoughts? Questions?

58. What kind of participants could you involve in a think-aloud?

59. Complete novice – to see learning Expert – to see how “it should be done” Recent learner – to see process of someone who has just acquired skill (may be more applicable for designing instruction)

60. For each think-aloud we did, what kind of participant did we have? Complete novice Expert Recent learner Other

61. Where do think-alouds fit on this diagram? ENTITATIVE HOLISTIC ESSENTIALISTEXISTENTIALIST

62. Janice Do you have anything you’d like to add, as one of the world’s foremost experts on think- alouds?

63. Today’s Class Probing Question Think-Aloud Protocols Assignments

64. Assignment #1 Very high quality submissions overall! Complete grading and commentary for assignment #1 will be a little bit delayed

65. Assignment #2 Any questions?

66. Assignment #3 Will be handed out Monday (by email)