1. Qualitative Methods Part Two January 20, 2010

2. Today’s Class Qualitative Methods Probing Question for next class Special session Assignment #2 Surveys

3. Clarification questions? Any questions on dialectic or phenomenology brewing in your minds since last class?

4. Contextual Inquiry

5. Contextual Inquiry is apprenticeship compressed in time One who visions must be steeped in the customer data Keep the customer concrete by exploring ongoing work He who questions training only trains himself at asking questions Partnership creates a sense of shared quest

6. Contextual Inquiry Contextual Inquiry is apprenticeship compressed in time One who visions must be steeped in the customer data Keep the customer concrete by exploring ongoing work He who questions training only trains himself at asking questions Partnership creates a sense of shared quest Which of these is not actually from Beyer & Holtzblatt?

7. Contextual Inquiry Contextual Inquiry is apprenticeship compressed in time One who visions must be steeped in the customer data Keep the customer concrete by exploring ongoing work He who questions training only trains himself at asking questions Partnership creates a sense of shared quest

8. Contextual Inquiry The relationships in Contextual Inquiry The Process of Contextual Inquiry How to Screw Up a Contextual Inquiry

9. What is Contextual Inquiry? “The core premise of Contextual Inquiry is very simple: go where the customer works, observe the customer as he or she works, and talk to the customer about the work. Do that, and you can’t help but gain a better understanding of your customer.” -- Beyer & Holtzblatt

10. Ethnographic method… A different one than Schofield’s But still fundamentally phenomenological – Goal of understanding the participant as he/she understands him/herself

11. Key aspect of Contextual Inquiry Relationship between researcher and participant Different from the methods we’ve studied up until this point

12. Subject or Participant? In most educational research, “subjects” are “subjected to” experiments or

13. Subject or Participant? In most educational research, “subjects” are “subjected to” experiments or are “the subjects” of studies,

14. Subject or Participant? In most educational research, “subjects” are “subjected to” experiments or are “the subjects” of studies, to help the researcher develop understanding

15. Subject or Participant? In most educational research, “subjects” are “subjected to” experiments or are “the subjects” of studies, to help the researcher develop understanding In contextual inquiry, “participants” “participate” in helping the researcher develop understanding

16. Note that… It is also often PC to call subjects participants However, in this case the difference actually means something

17. What is your relationship to the participant?

18. In a scientist/subject relationship: The scientist does stuff or asks questions The subject responds in some way The scientist collects data, goes back to their office, and analyzes the data to gain understanding of the subject

19. What is your relationship to the participant? In an interviewer/interviewee relationship: (like what was described in the R&R reading) The interviewer asks a question The interviewee responds immediately As soon as there is a pause, the interviewer asks another question from the list When all the questions have been answered, the interview is over. Beyer & Holtzblatt argue that this is only appropriate for gaining phenomenological understanding if you know what questions to ask in advance, and this requires already having phenomenological understanding

20. What is your relationship to the participant? Beyer & Holtzblatt argue that this is only appropriate for gaining phenomenological understanding if you know what questions to ask in advance, and this requires phenomenological understanding How do you think Schofield would respond to this claim?

21. What is your relationship to the participant? In a master/apprentice relationship: The master is doing stuff The master explains what they’re doing, to the apprentice The apprentice asks a clarification question The master answers Contextual inquiry attempts to simulate (in part) this relationship

22. What is your relationship to the participant? Photo of me in 2003 with Albert Corbett

23. What is your relationship to the participant? In a CI researcher/participant relationship: The participant is doing stuff The participant explains what they’re doing to the researcher The researcher offers an immediate interpretation The participant agrees or corrects The researcher’s goal is to develop understanding of what the participant is doing in partnership with the participant.

24. What is your relationship to the participant? How does this differ from Jean Lave’s more standard ethnographic approach? How did Lave relate to her participants? How did Lave interact with her participants?

25. Despite Beyer & Holtzblatt’s analogy It’s not quite a master/apprentice relationship The researcher’s goal is not to learn to do the task Instead, the researcher’s goal is to learn how the participant does the task, in order to understand the activity better And the researcher enlists the participant’s active assistance in understanding the activity

26. Partnership In Beyer & Holtzblatt’s view: Traditional interview models give too much control to the interviewer. A pure apprenticeship model gives too much control to the master. In Contextual Inquiry, the researcher and the participant are partners.

27. Partnership The participant knows their process better than the researcher The researcher has the distance to be able to see patterns and important features in the participant’s process and practice The researcher can draw the participant into a partnership in trying to richly understand the participant’s process This partnership can often result in insight for both parties (which is not a feature of either traditional interview or pure apprenticeship interactions)

28. Question Which approach is more likely to give phenomenological understanding? – Schofield? – Beyer & Holtzblatt?

29. Comments? Questions?

30. Contextual Inquiry The relationships in Contextual Inquiry The Process of Contextual Inquiry How to Screw Up a Contextual Inquiry

31. The Process of Contextual Inquiry Let’s contextualize the process of contextual inquiry with an example Let’s say we’re studying how a college student writes a term paper

32. The Process of Contextual Inquiry Let’s contextualize the process of contextual inquiry with an example Let’s say we’re studying how a college student, call her Beth, writes a term paper for her anthropology class How would we study Beth’s process?

33. The interviewer says… Why don’t you just ask Beth how she writes her paper? What arguments would Beyer & Holtzblatt give against doing this?

34. The interviewer says… Why don’t you just ask Beth how she writes her paper? Beth might not entirely know… When she summarizes how she does it for you, she might not remember vital details – Like the fact that she always has her roommate read her drafts she might gloss over important difficulties – Like the trouble she has searching for references on the web Many people remember summaries rather than exactly what happened

35. The ethnographic interviewer says… Why don’t you watch Beth write her paper, and then ask her questions about it afterwards? We’ll discuss this in a few minutes, when we discuss Retrospective Contextual Inquiry

36. In Contextual Inquiry You go to the actual setting where Beth is writing her term paper, and you interview her there – In this fashion, CI is like Lave’s methods

37. The Stages of a CI Observe Behavior Interview/ Warm-Up Share Interpretation Refine Interpretation Wrap-Up Withdrawal and return cycle

38. CI Stage 1: Conventional Interview Introduce yourself and explain the interview procedure (consent, recording, how CIs work). Ask Beth to summarize the goal she will be working towards during the CI – Writing her term paper Don’t take too long on this.

39. CI Stage 1: Conventional Interview Beyer & Holtzblatt emphasize that at this point The researcher should clearly and very explicitly state the rules of the CI. This is important, because if it’s not completely clear, the encounter may devolve into a traditional interview (since this relationship is more familiar to people)

40. CI Stage 2A: Observe Behavior Now ask Beth to go about her normal tasks, exactly as she would if you weren’t there – for example

41. CI Stage 2A: Observe Behavior Now ask Beth to go about her normal tasks, exactly as she would if you weren’t there – for example – Outlining her article

42. CI Stage 2A: Observe Behavior Now ask Beth to go about her normal tasks, exactly as she would if you weren’t there – for example – Outlining her article – Downloading references

43. CI Stage 2A: Observe Behavior Now ask Beth to go about her normal tasks, exactly as she would if you weren’t there – for example – Outlining her article – Downloading references – Copying text out of wikipedia

44. CI Stage 2A: Observe Behavior Now ask Beth to go about her normal tasks, exactly as she would if you weren’t there – for example – Outlining her article – Downloading references – Copying text out of wikipedia – Slightly changing the copied text while looking around furtively

45. CI Stage 2A: Observe Behavior And as she’s doing it, ask her to explain what she’s doing whenever it’s not 100.0% obvious “Beth, why did you just search on ‘trepanation’?” And take lots of notes.

46. CI Stage 2: Withdrawal and Return B&H recommend a pattern called withdrawal and return In withdrawal and return: The researcher observes something in the pattern of action that indicates there’s something meaningful going on The researcher asks about this, and the pair withdraw momentarily from the task at hand. The pair discuss the researcher’s issue. Afterwards, the participant returns to the task at hand.

47. During those Withdrawals The researcher should not only attempt to observe what’s going on. The researcher should try to interpret what it means. Then, the researcher should check these interpretations with the participant to see if the interpretation is correct

48. Checking an Interpretation: B&H Say “I believe X. Is that correct?”

49. Checking an Interpretation: B&H Say “I believe X. Is that correct?” “Beth, I believe you are searching for papers on trepanation because it was mentioned in the Wilkinson article, and that article did not explain the physiological effects sufficiently. Is that correct?” “Yes!” Or “No. I think that trepanation was a symptom of the moral decline of the Monte Alban culture, and want to see if it was associated with the decline of other civilizations.”

50. How does this differ? How does this differ from the types of questions asked by Lave and Schofield? How does this differ from the ways interpretations were checked in Lave and Schofield? Advantages/disadvantages?

51. Further notes on interpretation questions in CI

52. Does this method bias the data? B&H argue that people are more suggestible out of context, or when given general questions (note that Schofield uses general questions to *avoid* suggesting) In context, they are less likely to agree because the counterexample is ready-at-hand Beth knows why she just looked up trepanation

53. General Questions: A Contradiction What situations might B&H be correct in? What situations might Schofield be correct in?

54. When You’re Wrong CI practitioners find that participants can be uncomfortable flat-out disagreeing with a researcher. When doing a CI, it is important to be sensitive to two types of responses that indicate disagreement: Adjustments – where they repeat back to you something similar to what you said, but not identical. This usually means they think your interpretation is subtly incorrect. The words “Um…” or “Huh…” – this usually means you were wrong.

55. Example “Beth, I believe that you are now adding text to the introduction, because you have now completed all your reference-searching. Is that correct?” “Um, well, maybe. Yeah, I guess so…” (means NO!)

56. Checking Interpretation with Design Ideas An alternate interpretation-checking strategy is to offer a design idea that would make their task easier (if you interpreted correctly), and see their reaction Any ideas about how you could do this for Beth’s task?

57. CI Stage 3: Wrap-Up Continue the Contextual Inquiry as long as your participant completes the process of interest (or until you run out of time) After the process has been completed, skim back through your notes and summarize to the participant what you have learned. Focus on what you consider important about the work and the participant’s process Give them an opportunity to correct and elaborate on your understanding

58. Comments? Questions?

59. Contextual Inquiry The relationships in Contextual Inquiry The Process of Contextual Inquiry How to Screw Up a Contextual Inquiry

60. Caution! Contextual Inquiry is kind of famous as a design method that is easy to screw up (though not as much so as, say, profile-based design… now that is a method famous for producing disasters)

61. Caution! Contextual Inquiry is kind of famous as a design method that is easy to screw up (though not as much as, say, profile-based design…) The Pontiac Aztek – the car that turns into a tent!

62. How could you screw up a CI?

63. Disrupting The Task If you change the way the person performs the task with your behavior, your data is less useful Need to be sensitive to whether you are changing their behavior – in these situations, it is good to pull back – Retrospective methods can help in these situations; more on this in a bit – Asking people to take regularly scheduled discussion breaks is also sometimes useful

64. Multiple People Present People are less ready to be forthright about their processes with other individuals (especially powerful individuals) present True for all types of interview methods

65. Being Pushy With Your Interpretations If you seem to be ignoring the participant’s corrections, they will shut down

66. Doing a CI with the wrong person Find someone who is enthusiastic about the task, and interested in doing the CI Compliance is a much bigger issue in CI than other interview methods

67. Over-Focusing It’s perfectly reasonable – and good -- to have a focus on certain issues you expect to be important when you do a CI. But if you attend to events that violate your preconceptions about what’s important, you are likely to learn something interesting. If you’re ever surprised, or if you just don’t understand what the participant is saying, this is usually a cue that you could learn something. (Alexander Fleming) Oh, and be careful when you think you understand what’s going on!

68. Slipping Into Abstraction People like to make up stories about what they “usually do” – Even more of a problem with Schofield’s interview style “I usually look up all my references before starting to do any writing.” B&H argue that when you examine specific instances, “I usually…” can actually become “I wish I more frequently…”

69. Retrospective Contextual Inquiry Sometimes you want to study a process that occurs intermittently – or a process that you cannot be physically present for. You can’t hang out in a participant’s dorm room for 6 weeks straight to be around for 2 hours of data. – With student populations, there might be “eyebrows raised” For retrospective CI, you interview the participant, but ask them to go through a previous process step-by-step, repeatedly asking them to fill in the details they’re leaving out It’s helpful if the participant can bring out documents and objects relevant to that process and step you through what occurred – For instance, the papers Beth downloaded, her outlines, her drafts

70. A (Real) Example I was researching how software installers who work for educational technology projects coordinate with teachers and other school personnel (Baker, Wagner, Corbett, & Koedinger, 2004) It was not possible for me to be present during the coordination activity; I would have been very distracting. So I asked one installer (“Amber”) to print out all of her emails with different teachers, and we went through each email in order and discussed the interaction in the email and what had happened in between the emails (phone calls, school visits) It was possible to develop a fairly sophisticated model of the installer’s process this way.

71. Critical Incident Method A related approach is the Critical Incident Method (discussed in R&R) Ask participant to describe a particularly salient past experience, and then ask questions about it – “Think of the last time a classmate cheated off of you” – “Think of the last time you submitted a research paper for a class”

72. Advantages/Disadvantages of Retrospective CI? Compared to Schofield and to regular CI…

73. Today’s Class Qualitative Methods Probing Question for next class Special session Assignment #2 Surveys

74. Probing Question for Friday, February 5 Let’s say that the NSF has just given you a $100,000 grant to study student note-taking during classroom lecture – i.e. you can assign one minion to the task full-time for a year Which qualitative method would you prefer to use, and why? – What factors would you be especially attentive to, to ensure validity of your results?

75. Today’s Class Qualitative Methods Probing Question for next class Special session Assignment #2 Surveys

76. Special session Recall that we do not have class next Wednesday (I will be out of town at an AB meeting) There is a special (optional!) session "An Inappropriately Brief Introduction to Frequentist Statistics" Thursday, February 4, 4pm-6pm Higgins Laboratories 154

77. Today’s Class Qualitative Methods Probing Question for next class Special session Assignment #2 Surveys

78. Assignment #2 Will be handed out over the weekend – Check your email New due date: February 8 th (in schedule was February 5 th )

79. Today’s Class Qualitative Methods Probing Question for next class Special session Assignment #2 Surveys

80. The End