2. Correlations with Learning in Spoken Tutoring Dialogues Diane Litman Learning Research and Development Center and Computer Science Department University of Pittsburgh

3. Outline  Introduction  Dialogue Acts and Learning Correlations  Learning, Speech Recognition and Text-to-Speech  Current Directions and Summary

4. Motivation  An empirical basis for optimizing dialogue behaviors in spoken tutorial dialogue systems  What aspects of dialogue correlate with learning? –Student behaviors –Tutor behaviors –Interacting student and tutor behaviors  Do correlations generalize across tutoring situations? –Human-human tutoring –Human-computer tutoring

5. Approach  Initial: learning correlations with superficial dialogue characteristics [ Litman et al., Intelligent Tutoring Systems Conf., 2004] –Easy to compute automatically and in real-time, but… –Correlations in the literature did not generalize to our spoken or human-computer corpora –Results were difficult to interpret »e.g., do longer student turns contain more explanations?  Current: learning correlations with deeper “dialogue act” codings [ Forbes-Riley et al., Artificial Intelligence and Education Conf., 2005]

6. ITSPOKE (Version1): Intelligent Tutoring SPOKEn Dialogue System  Back-end is text-based Why2-Atlas tutorial dialogue system (VanLehn et al., 2002)  Student speech digitized from microphone input; Sphinx2 speech recognizer  Tutor speech played via headphones/speakers; Cepstral text-to-speech synthesizer  Other additions: access to Why2-Atlas “internals”, speech recognition repairs, etc.

10. Two Spoken Tutoring Corpora  Human-Human Corpus –14 students –1 human tutor –128 physics problems (dialogues) –5948 student turns, 5505 tutor turns  Computer-Human Corpus –20 students –ITSPOKE (Version1) tutor –100 physics problems (dialogues) –2445 student turns, 2967 tutor turns

11. Outline  Introduction  Dialogue Acts and Learning Correlations  Learning, Speech Recognition and Text-to-Speech  Current Directions and Summary

12. Dialogue Acts  Dialogue Acts represent intentions behind utterances –Both domain-independent and tutoring-specific tagsets »e.g., Graesser and Person, 1994; Graesser et al., 1995; Chi et al., 2001 –Used in prior studies of correlations with learning »e.g., tutor acts in AutoTutor (Jackson et al., 2004), dialogue acts in human tutoring (Chi et al., 2001)  ITSPOKE Study –Student and tutor dialogue acts –Unigrams and bigrams of dialogue acts –Human and computer tutoring –Spoken input and output

13. Tagset (1): (Graesser and Person, 1994) Tutor and Student Question Acts Short Answer Question: basic quantitative relationships Long Answer Question: definition/interpretation of concepts Deep Answer Question: reasoning about causes/effects

14. Tagset (2): inspired by (Graesser et al., 1995) Tutor Feedback Acts Positive Feedback: overt positive response Negative Feedback: overt negative response Tutor State Acts Restatement: repetitions and rewordings Recap: restating earlier-established points Request/Directive: directions for argument Bottom Out: complete answer after problematic response Hint: partial answer after problematic response Expansion: novel details

15. Tagset (3): inspired by (Chi et al., 2001) Student Answer Acts Deep Answer: at least 2 concepts with reasoning Novel/Single Answer: one new concept Shallow Answer: one given concept Assertion: answers such as “I don’t know” Tutor and Student Non-Substantive Acts: do not contribute to physics discussion

16. Annotated Human-Human Excerpt T: Which one will be faster? [Short Answer Question] S: The feathers. [Novel/Single Answer] T: The feathers - why? [Restatement, Deep Answer Question] S: Because there’s less matter. [Deep Answer]  All turns in both corpora were manually coded for dialogue acts (Kappa >.6)

17. Correlations with Unigram Measures (student and tutor-centered analyses)  For each student, and each student and tutor dialogue act tag, compute –Tag Total: number of turns containing the tag –Tag Percentage: (tag total) / (turn total) –Tag Ratio: (tag total) / (turns containing tag of that type)  Correlate measures with posttest, after regressing out pretest

18. Human-Computer Results (20 students) Student Dialogue ActsMeanRp # Deep Answer11.90.48.04

19. Human-Computer Results (continued) Tutor Dialogue ActsMeanRp # Deep Answer Question9.59.41.08 % Deep Answer Question6.27%.45.05 % Question Act76.89%.57.01 (Short Answer Question)/Question.88-.47.04 (Deep Answer Question) /Question.08.42.07 # Positive Feedback76.10.38.10

20. Human-Human Results (14 students) Student Dialogue ActsMeanRp # Novel/Single Answer19.29.49.09 # Deep Answer68.50-.49.09 (Novel/Single Answer)/Answer.14.47.10 (Short Answer Question)/Question.91.56.05 (Long Answer Question) /Question.03-.57.04

21. Human-Human Results (continued) Tutor Dialogue ActsMeanRp # Request/Directive19.86-.71.01 %Request/Directive5.65%-.61.03 # Restatement79.14-.56.05 # Negative Feedback14.50-.60.03

22. Discussion  Computer Tutoring: knowledge construction –Positive correlations »Student answers displaying reasoning »Tutor questions requiring reasoning  Human Tutoring: more complex –Positive correlations »Student utterances introducing a new concept –Mostly negative correlations »Student attempts at deeper reasoning »Tutor attempts to direct the dialogue

23. Correlations with Bigram Measures (interaction-centered analyses)  For each student, and each tag sequence containing both a tutor and a student dialogue act, compute –[Student Act_n - Tutor Act_n+1] Totals »all bigrams constructed by pairing each Student Dialogue Act in turn n with each Tutor Dialogue Act in turn n+1 –[Tutor Act_n - Student Act_n+1] Totals »all bigrams constructed by pairing each Tutor Dialogue Act in turn n with each Student Dialogue Act in turn n+1  Correlate measures with posttest, after regressing out pretest

24. Bigram Results  Many bigrams incorporate, as either the first or second element, a dialogue act corresponding to one of the unigram results, e.g. –[Student Deep Answer – Tutor Deep Answer Question] –[Tutor Recap - Student Deep Answer]  Other dialogue acts only correlate with learning as part of a larger dialogue pattern, e.g. – [Student Shallow Answer - Tutor Restatement] –[Tutor Restatement – Student Shallow Answer]

25. Discussion  Computer Tutoring –n-grams seem able to capture effective learning patterns in this simpler corpus  Human Tutoring –despite mostly negative correlations, students are learning! –suggests effective learning patterns are too complicated to be captured with n-grams

26. Current Directions  “Correctness” annotation –Are more Deep Answers “incorrect” or “partially correct” in the human-human corpus? –Do correct answers positively correlate with learning?  Beyond the turn level –Correlations with larger dialogue act patterns (e.g., tri- grams, n-grams) –Computation and use of hierarchical discourse structure

27. Outline  Introduction  Dialogue Acts and Learning Correlations  Learning, Speech Recognition and Text-to-Speech  Current Directions and Summary

28. Research Question  Does the performance of a system’s speech recognizer and/or text-to-speech system relate to learning ? –Speech recognition accuracy correlates with user satisfaction in non-tutoring systems (Litman and Pan, 2002; Walker et al., 2002) –The nature of a computer’s voice relates to learning and motivation in pedagogical agents (Baylor et al., 2003; Atkinson)

29. ASR Performance Measures (1): Rejections ITSPOKE: Therefore, what is the magnitude of this gravitational force in the horizontal direction? STUDENT:significant ASR: significant (False Rejection) ITSPOKE: Could you please repeat that? STUDENT: great ASR: crate (True Rejection) ITSPOKE: I'm sorry, I'm having trouble understanding you. Please try again.

30. ASR Performance Measures (2): “Transcription” Misrecognitions ITSPOKE: Yeah. Does the packet have an acceleration? If yes, please specify its direction. STUDENT: yes downward ASR: is downward word error rate:50 % binary word error:True ITSPOKE: How would you describe the vertical component of the packet's velocity? (e.g., decreasing, zero, etc.) STUDENT: increasing ASR: decreasing word error rate:100% binary word error: True

31. ASR Performance Measures (3): “Semantic” Misrecognitions ITSPOKE: Yeah. Does the packet have an acceleration? If yes, please specify its direction. STUDENT: yes downward NLU(student): downward ASR: is downward NLU(ASR): downward word error rate:50 %semantic error rate: 0% binary word error: True binary semantic error: False ITSPOKE: How would you describe the vertical component of the packet's velocity? (e.g., decreasing, zero, etc.) STUDENT: increasing NLU(student): increase ASR: decreasing NLU(ASR): decrease word error rate: 100 % semantic error rate: 100% binary word error: True binary semantic error: True

32. Correlations with Learning?  Computed totals, percentages, and ratios of: –Rejections »False, True, Both –Misrecognitions »Transcription and Semantic »Word and Binary –ASR Problems (Rejections + Misrecognitions) »Transcription and Semantic »Word and Binary  Found no significant correlations or trends!

33. ITSPOKE (Version2) Prerecorded Output Synthesized Output (human voice) (text-to-speech) ITSPOKE: Terrific. Let's try the original question again. If gravity is the only force acting on an object, will it be moving or staying still? STUDENT:moving (ASR: moving) ITSPOKE: Yes. Not only are the person, keys, and elevator moving, they have only gravitational forces acting on them. When an object is falling and has only gravitational force on it, it is said to be in what? STUDENT: free fall(ASR: free fall) Prerecorded (human voice) Synthesized (text-to-speech)

34. New Computer Tutoring Experiment  Same subject pool, physics problems, web interface, and experimental procedure as before, except –ITSPOKE (Version2)  Pre-recorded voice condition –30 students (150 dialogues)  Text-to-speech condition –29 students (145 dialogues)

35. Outline  Introduction  Dialogue Acts and Learning Correlations  Learning, Speech Recognition and Text-to-Speech  Current Directions and Summary

36. Summary  Many dialogue act correlations – positive correlations with deep reasoning and questioning in computer tutoring – correlations in human tutoring more complex – student, tutor, and interactive perspectives all useful  No correlations with ASR problems  Stay tuned … –New dialogue act patterns and “correctness” analysis –Pre-recorded versus text-to-speech

37. Acknowledgments  The ITSPOKE Group –Staff:Scott Silliman –Research Associates:Kate Forbes-Riley Joel Tetreault Alison Huettner (consultant) –Graduate Students:Ai Hua Beatriz Maeireizo Amruta Purandare Mihai Rotaru Arthur Ward  Kurt VanLehn and the Why2 Team

38. Thank You! Questions?

39. Hypotheses  Compared to typed dialogues, spoken interactions will yield better learning gains, and will be more efficient and natural  Different student behaviors will correlate with learning in spoken versus typed dialogues, and will be elicited by different tutor actions  Findings in human-human and human-computer dialogues will vary as a function of system performance

40. Motivation  Working hypothesis regarding learning gains –Human Dialogue > Computer Dialogue > Text  Most human tutoring involves face-to-face spoken interaction, while most computer dialogue tutors are text-based –Evens et al., 2001; Zinn et al., 2002; Vanlehn et al., 2002; Aleven et al., 2001  Can the effectiveness of dialogue tutorial systems be further increased by using spoken interactions?

41. Potential Benefits of Speech  Self-explanation correlates with learning and occurs more in speech – Hausmann and Chi, 2002  Speech contains prosodic information, providing new sources of information for dialogue adaptation –Forbes-Riley and Litman, 2004  Spoken computational environments may prime a more social interpretation that enhances learning –Moreno et al., 2001; Graesser et al., 2003  Potential for hands-free interaction –Smith, 1992; Aist et al., 2003

42. Spoken Tutorial Dialogue Systems  Recent tutoring systems have begun to add spoken language capabilities –Rickel and Johnson, 2000; Graesser et al. 2001; Mostow and Aist, 2001; Aist et al., 2003; Fry et al., 2001; Schultz et al., 2003  However, little empirical analysis of the learning ramifications of using speech

43. Architecture Cepstral www server www browser java ITSpoke Text Manager Spoken Dialogue Manager essay dialogue student text (xml) tutor turn (xml) html xml text Speech Analysis (Sphinx) dialogue repair goals Essay Analysis (Carmel, Tacitus- lite+) Content Dialogue Manager (Ape, Carmel) Why2 tutorial goals text essay

44. Speech Recognition: Sphinx2 (CMU)  Probabilistic language models for different dialogue states  Initial training data –typed student utterances from Why2-Atlas corpora  Later training data –spoken utterances obtained during development and pilot testing of ITSPOKE  Total vocabulary – 1240 unique words  “Semantic Accuracy” Rate = 92.4%

45. Speech Synthesis: Cepstral  Commercial outgrowth of Festival text-to- speech synthesizer (Edinburgh, CMU)  Required additional processing of Why2-Atlas prompts (e.g., f=m*a)

46. Common Experimental Aspects  Students take a physics pretest  Students read background material  Students use web interface to work through up to 10 problems with either a computer or a human tutor  Students take a posttest –40 multiple choice questions, isomorphic to pretest

47. ITSPOKE Corpora Comparison Human-Human Human-Computer …1.3 minutes into session… Student: Can I ask you questions? Tutor: Yes Student: Is that ok? Tutor: Mm-hm Student: Um from what we learned from the last problem they're gonna have the same- the keys and you have the same acceleration right? Tutor: Yes Student: So that means they'll both hit the bottom at the same time. But I don't understand where- I don't understand if you're in the elevator- Tutor: You see Student: Where are you going to-? Tutor: The uh let me uh the key uh- the person holds the key in front of- Student: Their face yeah- Tutor: Uh his or her face and then lets it uh uh it is let go so the the the question relates to the relative position of the key and the face uh- Student: So the key and the face- Tutor: Uh how will they- as they go- as they both fall what will be their relative position? That is the question. Student: (sigh) …3.5 minutes into session… Tutor: I didn't understand. What force acts on all objects within close proximity to earth? Student: uh the force of gravity Tutor: Fine. Besides the Earth's gravitational force, are there any other forces acting on the person, keys, and elevator after the release? Student: no Tutor: Yep. Thus, keep in mind that after the release, the only force on the person, keys, and elevator is the force of gravity. If gravity is the only force acting on an object, will it be moving or staying still? Student: it will be moving Tutor: Fine. Not only are the person, keys, and elevator moving, they have only gravitational forces on them. When an object is falling and has only gravitational force on it, it is said to be in what? Student: freefall Tutor: Good. Thus, after the release the only force on the keys and man is the downward force of earth's gravity, so they are in freefall. Before the release, what is the relationship between the man's velocity and the keys' velocity? Student: they're equal Tutor: Correct. We know that after the release, the keys, the person, and the elevator are in freefall. Is freefall acceleration near earth a constant? Or does it vary depending upon which object is in freefall? Student: it's a constant

48. Learning Correlations after Controlling for Pretest Dependent Measure Human Spoken (14) Human Typed (20) RpRp Ave. Stud. Words/Turn-.209.49.515.03 Intercept: Stud. Words/Turn-.441.13.593.01 Ave. Tut. Words/Turn-.086.78.536.02

49. Learning Correlations after Controlling for Pretest Dependent Measure Spoken (ITSPOKE) Typed (Why2-Atlas) RpRp Tot. Stud. Words.394.10.050.82 Tot. Subdialogues/KCD-.018.94-.457.03

50. ASR-Learning Correlations Key ASR MIS: ASR Misrecognition SEM MIS: Semantic Misrecognition TRUE REJ: True Rejection FALSE REJ: False Rejection REJ: Total Rejections (true or false) ASR PROB: ASR Misrecognition or Rejection SEM PROB: Semantic Misrecognition TIMEOUT: Timeout

51. Correlation Results (20 students) LearningTime SP MeasureMeanRpRp # ASR MIS32.3.27.26.46.05 # SEM MIS6.7-.05.83.25.29 # TRUE REJ6.5-.22.36.18.46 # FALSE REJ1.7-.26.28.27.26 # REJ8.2-.24.31.21.38 # ASR PROB40.5.06.82.41.08 # SEM PROB14.9-.19.43.25.31 # TIMEOUT5.5.30.22.45.06

55. Current Directions  Online dialogue act annotation during computer tutoring –Tutor acts can be authored –Student acts need to be recognized  “Correctness” annotation –Are more Deep Answers “incorrect” or “partially correct” in the human-human corpus? –Do correct answers positively correlate with learning?  Beyond the turn level –Learning correlations with dialogue act patterns (e.g., bigrams) –Computation and use of discourse structure

56. Primary Research Question  How does speech-based dialogue interaction impact the effectiveness of tutoring systems for student learning?

57. Spoken Versus Typed Human and Computer Dialogue Tutoring (ITS 2004)  Human Tutoring: spoken dialogue yielded learning and efficiency gains –Many differences in superficial dialogue characteristics  Computer Tutoring: spoken dialogue made less difference  Learning Correlations: few results –Different dialogue characteristics correlate in human versus computer, and in spoken versus typed

58. Motivation  An empirical basis for authoring (or learning) optimal dialogue behaviors in spoken tutorial dialogue systems  Previous Approach: learning correlations with superficial dialogue characteristics –Easy to compute automatically and in real-time, but… –Correlations in the literature did not generalize to our spoken or human- computer corpora –Results were difficult to interpret »e.g., do longer student turns contain more explanations?  Current Approach: –learning correlations with measures based on deeper dialogue codings

59. Current Empirical Studies  Does learning correlate with measures of automatic speech recognition (ASR) performance? –ITSPOKE (Version1) corpus  Does the use of pre-recorded audio rather than text-to- speech improve learning and other measures? –ITSPOKE (Version 2) corpus  Can speech recognition “goats” be screened in advance?

60. User Survey: after (Baylor et al. 2003), (Walker et al. 2002) It was easy to learn from the tutor The tutor did not interfere with my understanding of the content The tutor believed I was knowledgeable The tutor was useful The tutor was effective on conveying ideas The tutor was precise in providing advice The tutor helped me to concentrate It was easy to understand the tutor I knew what I could say or do at each point in the conversations with the tutor The tutor worked the way I expected it to Based on my experience using the tutor to learn physics, I would like to use such a tutor regularly. Response options: almost always, often, sometimes, rarely, almost never

61. Summary  Dialog Act Annotation and Learning Correlations –Human tutoring and ITSPOKE (Version 1) corpora  Speech Recognition and Learning Correlations –ITSPOKE (Version 1) corpus  ITSPOKE (Version 2) and New Corpus Collection –Pre-recorded versus synthesized speech (in progress)