1. Observational Methods January 20, 2010

2. Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri, Jan. 22 Survey

3. Survey Results Only 3 responses  – I will be switching to a better technology for the next survey Mostly positive I’ve read through your comments, and will try to take them into account Quick comment – if you think a key term is being incompletely or unclearly defined, please let me know, and I will try to explain it better The concepts covered in the first class will be re-visited as we go, and perhaps will become clearer as we get more examples

4. Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri, Jan. 22 Survey

5. ENTITATIVE HOLISTIC ESSENTIALISTEXISTENTIALIST

6. Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri, Jan. 22 Survey

7. Observational methods: top level You don’t manipulate the research setting in any way You just look at what is happening, or what has happened, and you note it Later, you analyze the data

8. Observational Methods This class we will focus on quantitative observational methods – Where observations are noted in terms of a defined set of categories – Called “quantitative” (numerical) because of the analytic methods commonly used with this kind of data

9. Observational Methods This week we will focus on quantitative observational methods Next week we will focus on qualitative methods – Where observations are noted in a more free-form way that does not attempt to rigorously apply a defined set of categories, but attempts to understand cases in their full particularity

10. What is the research setting? Where the learning (or other phenomenon you want to study) is taking place

11. What is the research setting? Usually a real learning setting – Classroom – University computer lab – Dorm room – Workplace – Any others? Occasionally a lab setting – Needs to be justification for why the lab setting is valid

12. Types of observation Enhanced video coding Video coding Field observation Screen replay coding Text replay coding (and more)

13. High-level notes High-level methodological similarity Can be used for many of the same research goals Advantages and disadvantages to each

14. Types of observation (that we’ll focus on today) Video coding Field observation Text replay coding

15. Types of observation that we’ll focus on today Video coding Field observation Text replay coding

16. Let’s watch a video http://www.youtube.com/watch?v=lmoEWCej QIg&feature=related http://www.youtube.com/watch?v=lmoEWCej QIg&feature=related – From 0:00 to 1:15

17. What student behaviors did you observe? Please explain both what the behavior is, and how you could recognize it from the video Does anyone disagree with how a behavior is defined? Why?

18. List of behaviors Let’s briefly write down a list of behaviors seen

19. OK, let’s watch it again Please write every behavior you see that did not fit into the original coding scheme

20. What additional student behaviors did you observe? Please explain both what the behavior is, and how you could recognize it from the video Does anyone disagree with how a behavior is defined? Why?

21. What is the total list of behaviors?

22. Now let’s make a coding scheme A list of “interesting” behaviors, how they are defined, and a code letter Example: – Hint abuse (a type of gaming the system) – Clicking through hints at high speed until reaching the bottom-out hint, obtaining and entering the answer, and immediately moving on to the next problem step

23. Now let’s make a coding scheme Should not be infinitely long – eliminate – “Uninteresting” behaviors – be guided by your intuition prior theory and empirical results – “Extremely rare” behaviors – under approximately 1 in 200 – Behaviors occuring between 1-5% of the time are often very interesting behaviors! – Effects on learning are often disproportionate Gaming the system occurs 3-5% of the time, and correlates to learning at 0.3-0.4

24. Which behaviors in your list Are “uninteresting”? Are “extremely rare”?

25. Now let’s make a coding scheme And combine related behaviors – For example, different types of gaming the system (e.g. hint abuse, systematic guessing, intentional rapid mistakes) often get classified together common – antecedents (poorly known skill) – characteristics (very fast responses) – results (poor learning gains) differences are no longer considered theoretically interesting (some debate on this still) tend to co-occur

26. Which behaviors Should be combined?

27. Our Coding Scheme

28. OK, let’s watch it for the 3 rd time Each person in our class will be assigned a student in the video (minimum of 2 of us for each 1 of them) Every five seconds of video, I am going to stop the video, and you need to write down the best code for the previous 5 seconds – If 2 behaviors observed, pick the dominant one – If you are not sure, put down a “?” – Do NOT talk to your partner

29. OK, now work with your partner Find out what % of time you agreed Find out where and how you disagreed There are better ways to assess inter-rater reliability, we’ll discuss these next class

30. Each group’s % error

31. What categories did you disagree on?

32. OK, now let’s watch Same video, but times 1:06 to 2:10 Observe the same student Follow the same procedure as before

33. Were the behaviors the same in this segment? It is important to: – Develop your coding scheme based on sufficient initial qualitative observation (and/or theory and prior work) – Note behaviors that do not fit in your coding scheme

34. OK, now work with your partner Find out what % of time you agreed Find out where and how you disagreed

35. Across sessions Did your agreement go up or go down? By how much?

36. A good strategy Repeatedly code the same data separately and compare Until your inter-rater reliability is “high enough” (more next time) And then code different data separately

37. Incidentally There is nothing “magic” about the video coding method we used It was largely selected for tractability in class

38. Other things we could have done Watch the same time segment multiple times Advantages Disadvantages

39. Other things we could have done Watch the same time segment multiple times Advantages: Greater certainty in coding Disadvantages: Takes a lot longer; allows second-guessing and hair-splitting

40. Other things we could have done Code when the behavior changes, not on a pre-selected time interval Advantages Disadvantages

41. Other things we could have done Code when the behavior changes, not on a pre- selected time interval Advantages: Get better assessment of the duration of each behavior Disadvantages: Enforces coding one student at a time; some code switches do not have “hard” edges (especially in case of coding affect)

42. Other things we could have done Code the first action in a time window, not the predominant one Advantages Disadvantages

43. Other things we could have done Code the first action in a time window, not the predominant one Advantages: Less ambiguity and decision- making Disadvantages: May be less representative; very brief behaviors may be missed

44. Other things we could have done What else?

45. A limitation of one camera

46. What is she doing?

47. Other ways to use video Multiple cameras Addresses issue of the student off-screen Synchronization becomes a challenge Time to code increases with each camera you add

48. Other ways to use video Individual student video Often from web-cams on top of machines

49. Example

50. https://umdrive.memphis.edu/sdmello/memp his-affect-dialogue.wmv?ticket=t_dPrPbv8H https://umdrive.memphis.edu/sdmello/memp his-affect-dialogue.wmv?ticket=t_dPrPbv8H

51. Note that The specific D’Mello study we read was a lab study, but the same method has been used in classrooms (without the emote-aloud) (cf. Arroyo et al, 2009) I chose the D’Mello paper in part because of the very clear description and comparisons of alternatives

52. Other ways to use video Other thoughts?

53. Types of observation (that we’ll focus on today) Video coding Field observation Text replay coding

54. Methods Similar, but live and in the field rather than via video Different challenges Some advantages

55. Sampling One challenge that emerges in field observations is sampling

56. Sampling Usually not desirable to follow only one student throughout the entire observation period Usually not tractable to have one observer per student Usually not possible to fully code multiple students at the same time

57. Field Sampling Strategies Switch between students in pre-determined order according to pre-determined schedule Watch or public timekeeper (image from Rodrigo et al, 2008)

59. Field Sampling Strategies Focus on a specific behavior and code it whenever it occurs, including which student engaged in the behavior

60. Observer effects Another challenge in live observation is observer effects A person hovering over your shoulder will impact your behavior

61. Observer effects: What you can do Obfuscation strategies Desensitization strategies

62. Obfuscation strategies Don’t appear to be observing the student you are actually observing – Observation from a distance – Side glances – Peripheral vision observation

63. Obfuscation strategies Don’t appear to be observing the student you are actually observing – Observation from a distance – Side glances – Peripheral vision observation Easiest to do Depends on classroom layout Not possible to assess some things

64. Obfuscation strategies Don’t appear to be observing the student you are actually observing – Observation from a distance – Side glances – Peripheral vision observation Not always effective

66. Obfuscation strategies Don’t appear to be observing the student you are actually observing – Observation from a distance – Side glances – Peripheral vision observation Takes training, but is the most powerful

68. Desensitization strategies Spend a class period walking around the classroom and noting random symbols in your clipboard Whenever a student asks you what you’re doing, show them the clipboard and explain that you’re just there to study the software for your professor and won’t be giving any information to the teacher After a certain point, they will decide you’re not important and begin to behave in normal (frighteningly anti-social) ways

69. Advantages of field observations

70. Faster to conduct About 3x faster (Baker, Corbett, & Wagner, 2006)

71. Faster to conduct Go out to class Versus Go out to class, then analyze in lab Also – The ability to stop and re-watch is frequently used in video coding

72. No camera & microphone limitations You can stand close enough to really see & hear what’s going on If something is obscured you can change your position

73. What is she doing? What is his facial expression?

74. With video… You have to get the data collection right, even before you’ve had time to do analysis

75. With field observations… You can quickly see whether you’re missing key information, and adjust your method (Note: adjusting your method in the *middle* of a study is a bad idea. Better to adjust your method, and restart, and just lose the first day’s data)

76. Major limitation Impossible to change coding categories to follow up on a question that emerges when you analyze the data Impossible for data to be re-used by alternate research group with different coding scheme

77. Other thoughts on field observation?

78. Types of observation (that we’ll focus on today) Video coding Field observation Text replay coding

79. Text replays Pretty-prints of student interaction behavior from the logs

80. Examples

86. Sampling You can set up any sampling schema you want, if you have enough log data 5 action sequences 20 second sequences Every behavior on a specific skill, but other skills omitted

87. Sampling Equal number of observations per lesson Equal number of observations per student Observations that machine learning software needs help to categorize (“biased sampling”)

88. Major Advantages Both video and field observations hold some risk of observer effects Text replays are based on logs that were collected completely unobtrusively

89. Major Advantages Blazing fast to conduct – 8 to 40 seconds per observation

90. Notes Decent inter-rater reliability is possible (Baker, Corbett, & Wagner, 2006) (Baker, Mitrovic, & Mathews, under review) Agree with other measures of constructs (Baker, Corbett, & Wagner, 2006) Can be used to train machine-learned detectors (Baker & de Carvalho, 2008) (Baker, Mitrovic, & Mathews, under review)

91. Major Limitations Limited range of constructs you can code Gaming the System – yes Collaboration in online chat – yes (Prata et al, 2008) Frustration, Boredom – sometimes Off-Task Behavior outside of software – no Collaborative Behavior outside of software – no

92. Major Limitations Lower precision (because lower bandwidth of observation)

93. Other thoughts on text replays?

94. Other Quantitative Observation Methods

96. When? When might you want to use some of the other methods listed here?

97. Any thoughts? About some of these related methods?

98. Not as common I only know of a 2 papers using screen replay in education – That group (Helen Pain and her students) now uses transcript coding, field coding, and video coding, but don’t use screen replays anymore I don’t know of any group coding using Super- Fidelity data, although D’Mello and Arroyo both use that kind of data in machine learning

99. High-level thoughts on quantitative observation methods

100. Generalizability Important to think about where you data came from, and what conclusions you can draw

101. What can you conclude? Data on one class day Data on one lesson Data on full year Data from 1 student Data from 10 students Data from 1 class Data from 1 school Data from 3 schools with different profiles Data from 100 schools

102. Feasibility of collection for each method Data on one class day Data on one lesson Data on full year Data from 1 student Data from 10 students Data from 1 class Data from 1 school Data from 3 schools with different profiles Data from 100 schools

103. Things you can code Teacher behaviors Student individual behaviors Student collaborative behaviors Student affect What else?

104. What method(s) work for each of these? Teacher behaviors Student individual behaviors Student collaborative behaviors Student affect What else?

105. Coding teacher behavior http://www.youtube.com/watch?v=gHzTUYAO kPM http://www.youtube.com/watch?v=gHzTUYAO kPM

106. Standard coding schemes It is worth noting that many research areas have standard coding schemes that have been developed and refined across many research projects, and are used by many researchers – Lauren Resnick and Sarah Michaels’s accountable talk (strategies used by students and teachers) – Baker et al: engagement-related behaviors in classrooms using educational software – D’Mello et al: affect in educational settings

107. Advantages to using these schemes You don’t need to defend your choice of scheme in your write-up Easier to compare your results to previous results and identify what’s new/different

108. Disadvantages to using these schemes You insulate yourself to some degree from obtaining surprising/new/exciting results

109. Which types of research programs would use these methods?

110. ENTITATIVE HOLISTIC ESSENTIALISTEXISTENTIALIST

111. ENTITATIVE HOLISTIC ESSENTIALISTEXISTENTIALIST

112. Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri, Jan. 22 Survey

113. Probing Question For Friday, you will read D'Mello, S., Taylor, R.S., Graesser, A. (2007) Monitoring Affective Trajectories during Complex Learning. Proceedings of the 29th Annual Meeting of the Cognitive Science Society, 203-208 Which used data from a lab study If you wanted to study affective transitions in real classrooms, which of the methods we discussed today would be best? Why?

114. Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri, Jan. 22 Survey

115. Please complete this survey, and put it in this envelope

116. The End