Evaluating the Effect of Social Conversation on Learning, Interaction, and Perceived Interdependence in a Collaborative Math Problem Solving Environment Rohit Kumar, Gahgene Gweon, Mhesh Joshi, Yue Cui, Adaeze Nwaigwe, Carolyn P. Rosé Language Technologies Institute and Human-Computer Interaction Institute Carnegie Mellon University

Why study help? Offering of deep help and elaborated explanations predicts post test performance (e.g., Webb et al., 2002) Help behavior mediates learning (e.g., Gweon et al., 2006; Gweon et al., 2007) Exchanging help in the context of collaborative learning increases identification with a learning community and motivation, and improves race relations (Sharan, 1980) Providing help leads to feelings empowerment for “low status” students (Elbers & Hann, 2004)

How can we prompt helping behavior?  Explicit prompts (Gweon et al., 2006)  Manipulating availability of help from problem solving environment (Gweon et al., 2007)  Girls offered more help with delayed feedback from environment  Boys offered more help with immediate help from the environment  Current study:  Will students follow the example of conversational agents offering conceptual help?  Will social prompts affect how students feel and behave?

Chat Agent Tutor: Another way to think of dividing by fractions is this: If I have 1 divided by 1/2, can I turn that into something divided by 1?. If I want to turn the denominator, which is 1/2, into 1, what do I need to do to it? Student 1: yep Tutor: Actually, you need to multiply by two. And If I multiply the denominator by 2, what do I have to do to the numerator? Student 1: 2 Tutor: Right. So 1 divided by 1/2 is the same as 2 divided by 1. So that is one reason why the flip and multiply strategy for dividing by fractions works. If you multiply the denominator by something to turn it into 1, you have to multiply the top by the same value. So let's try another one. If I want to divide something by 2/3, what do I need to multiply the numerator and denominator by? Student 1: 3 Tutor: Actually, it is 3/2. You should think about why it's 3/2.

Collaborative Problem Solving Environment Jan packed several books to amuse herself on along car ride to visit her grandma. After 1/5 of the trip she had already finished 6/8 of the books she brought. How many times more books should she have brought than what she packed?

Extraneous Entertainment?

Connection with VMT Work  Chat analysis from VMT session  “Class clown” at first appeared to be “goofing off”  Later reflection revealed he was a tone setter, putting his fellow students at ease  His light hearted behavior supported his fellow students in foraging ahead to solve difficult problems

Experimental Design  Participants: 30 6 th graders from a suburban elementary school  Learning in pairs with or without social prompts  2 Condition between subjects design  Dependent Measures  Tests and Quizzes  Questionnaire  Process analysis of conversational behavior

Experimental Procedure  Day 1  Pre-test  Day 2  45 minute lab session  Quiz 1  Day 3  45 minute lab session  Quiz 2  Questionnaire  Day 4: [separated from Day 3 by a weekend]  Post-test

Questionnaire  Perceived problem solving competence of self and partner  Perceived benefit of collaboration  Perceived help received  Perceived help provided  All questions were answered with a 6 point likert scale ranging from 0 (strongly dissagree) to 6 (strongly agree)

Questionnaire Results ControlExperimental Perceived Self Competence4.2 (.56)4.1 (.23) Perceived Partner Competence 4.3 (.62)3.9 (.49) Perceived Benefit of Collaboration 4.5 (.74)4.4 (.70) Perceived Help Received*1.8 (1.3)3.3 (.69) Perceived Help Provided*1.8 (1.1)3.1 (1.1)

Test Results  Very weak evidence in favor of Experimental condition  Consistent (but non-significant!) trend for students in the Experimental condition to learn more  Marginal advantage for Experimental condition on Lab day 2 on Interpretation problems (p=.06, effect size.55 s.d.)

Coding Scheme (Gweon et al., 2007)  Help provision mediates learning (Gweon et al., 2006; Gweon et al., 2007)  (R) Help Requests: “Help me”, “I’m stuck”, “I don’t get it.”  (P) Help Provisions: “Find the common denominator”, “Try the flip and multiply strategy”  (C) Can’t help: “I don’t know”, “I’m stuck too”  (D) Deny help: “ask the teacher”, “you’re an idiot”, “press the help button”  (N) Nothing

Help Request met with Provision of Help  Student 1: What operation do we do?    Student 2: We divide. Now look at the problem, what is the other fraction we must divide by?  Student 1: What do we put on top of the fraction?  Student 2: Did you find a common denominator?  

Help request met with Can’t Help  Student 1: Why 16?  Student 2: I don’t know.  Student 1: I need help.  Student 2: Same  Student 1: 23/2  Student 2: What’s 23/2?  Student 1: 11.5

Help request met with Deny Help  Student 1: I don’t get it  Student 2: hold on    Student 1: I don’t know what to do  Student 2: click on the help button

Help request met with No Response  Student 1: I don’t get it    Student 1: ?  

More Help Related Episodes per Problem in Experimental Condition F(1,15) = 16.8,p <.001 Mean Experimental =.69, Mean Control =.30 Effect size = 1 standard Deviation

Results from Corpus Analysis Experimental (Day 1) Experimental (Day 2) Control (Day 1) Control (Day 2) Total Episodes*47.1 (8.2)61.3 (12.3)33.8 (17.9)49.1 (26.9) Social Prompt Episodes* 24.1 (9.9)33.7 (16.2)0 (0) Help Episodes (P) Unsolicited.79 (1.6) 1.7 (2.1).36 (1.1) 3.2 (6.0) 1 (1.3) 2.1 (3.2) 1.4 (2.9) 1.9 (3.2) Unanswered Help Requests (C+R+N) 2.4 (2.7)1.4 (1.9)2.2 (1.9)1.4 (1.4) Non-Help Episodes*19.9 (5.6)35.8(9.3)30.6 (16.3)46.3 (25.1)

Proportion on Non-Help Related Episodes Experimental (Day 1) Experimental (Day 2) Control (Day 1) Control (Day 2) 80.0% 87.8% 85.2% 90.8%

What was happening in non-help episodes?  Coordination  Who should go next  Which parts of the problem each person should be responsible for  Regulation of speed (Go! Slow down! Hurry up!)  Complaining about being hungry or bored  Joking about the content of the problems  Cheering about finishing problems  More talk about winning or losing in the control condition  More references to “we”, “us”, and “our” in the experimental condition  Insulting each other  Insults like “loser”, “you stink”, and “stupid” only occurred in the Control condition!

Negative Affect in Control Condition  Student 1: finally  Student 2: Shut up  Student 1: oooooooooo burn  Student 2: I don't like you  Student 1: fine be that way  Student 2: how did you get that  Student 1: Guessing  Student 2: good, do you got it?  Student 1: no  Student 2: well too bad

What did we learn?  Social prompts affected student attitudes towards each other  More insults and competitive attitude in the Control condition  Social prompts fostered more perceived inter- dependence (based on questionnaire)  Social prompts increased the proportion of help related episodes per problem  Possible learning effect  Social factors affect learning

TagHelper tools  Supporting data analysis involving conversational data  Supporting on-line assessment  Triggering interventions TagHelper Labeled Texts Unlabeled Texts Labeled Texts A Model that can Label More Texts Time Help Student1: I don’t understand what to do next. Student2: You’re an idiot. Support Agent: Student2, it looks like your partner could use some help.

Supporting Data Analysis for helping behavior  Peer tutoring in Algebra LearnLab  Data coded for high-level- help, low-level-help, and no- help  Important predictor of learning (e.g., Webb et al., 2003)  TagHelper achieves agreement of.82 Kappa  Can be used for follow-up studies in same domain * Contributed by Erin Walker

Example of Triggering Interventions  Collaborative idea generation in the Earth Sciences domain  Chinese TagHelper learns hand-coded topic analysis  Human agreement.84 Kappa  TagHelper performance.7 Kappa  Trained models used in follow-up study to trigger interventions and facilitate data analysis

Example Dialogue SpeakerText Student 1 People stole sand and stones to use for construction. VIBRANT Yes, steeling sand and stones may destroy the balance and thus make mountain areas unstable. Thinking about development of mountain areas, can you think of a kind of development that may cause a problem? Student 2 Development of mountain areas often causes problems. Student 1 It is okay to develop, but there must be some constraints. * Feedback during idea generation increases learning and idea generation productivity (except during the first 5 minutes) (Wang et al., 2007) Pairs+ Feedback Individuals+ NoFeedback Pairs+ NoFeedback Individuals+ Feedback

Process Analysis Process loss Pairs vs Individuals: F(1,24)=12.22, p<.005, 1 sigma Process loss Pairs vs Individuals: F(1,24)=4.61, p<.05,.61 sigma Negative effect of Feedback: F(1,24)= 7.23, p<.05, sigma Positive effect of feedback: F(1,24)=16.43, p<.0005, 1.37 sigma Pairs+Feedback Individuals+NoFeedback Pairs+NoFeedback Individuals+Feedback Pairs+ Feedback Individuals+ NoFeedback Pairs+ NoFeedback Individuals+ Feedback

Thank you for listening!