Presentation is loading. Please wait.

Presentation is loading. Please wait.

Computer Supported Collaborative Learning Language Technologies Institute Carnegie Mellon University Wednesday, March 18, 2009: Speech and NLP for Educational.

Similar presentations


Presentation on theme: "Computer Supported Collaborative Learning Language Technologies Institute Carnegie Mellon University Wednesday, March 18, 2009: Speech and NLP for Educational."— Presentation transcript:

1 Computer Supported Collaborative Learning Language Technologies Institute Carnegie Mellon University Wednesday, March 18, 2009: Speech and NLP for Educational Applications

2 2/44 Research Group Rohit Kumar

3 3/44 Our work: Questions  Conversational Agents (Among other things)  To support human users at various tasks What kind of tasks? What kind of tasks?  Supporting Learning tasks What kind of support? (Agents ofcourse, but…) What kind of support? (Agents ofcourse, but…) What role? What role? Manifestation Manifestation How do we make sure the support is getting through? How do we make sure the support is getting through? What kind of users? What kind of users? Individuals / Pairs / Groups ? Individuals / Pairs / Groups ? Students ? Teachers/facilitators? Students ? Teachers/facilitators? What environment? What environment? How do we build this support? How do we build this support? How do we evaluate if the support helps? How do we evaluate if the support helps?

4 4/44 Our focus today: Rohit Kumar, Carolyn P. Rosé, Yi-Chia Wang, Mahesh Joshi, Allen Robinson Tutorial Dialogue as Adaptive Collaborative Learning Support Artificial Intelligence in Education 2007 Particularly interesting: Borderlining the transition of the CycleTalk project from Tutorial Dialog to CSCL

5 5/44 CycleTalk  Cycle Pad (Forbus et. al. 1999)  Thermodynamic cycles simulation environment  Designed to engage students in engineering design  Exploratory learning  Cycle Talk Goals:  Support engineering students learning the principles involved in designing thermodynamic cycles  Study the benefits of tutorial dialogue in an exploratory learning context

6 6/44 CycleTalk: History  Rosé et. al., CycleTalk: Towards a Dialogue Agent that Guides Design with an Articulate Simulator, 2004  Cognitive Task Analysis  Observation: Cycle Pad’s significant pedagogical potential tends to be underutilized when students do not receive tutorial guidance Build cycle Incorporate conceptual understanding Explore relationships between parameters Generate plan to improve cycle Assume component parameters Investigate variable dependencies Compare multiple cycle improvements Compare cycle to alternatives

7 7/44 CycleTalk: History  Rosé et. al., A First Evaluation of the Instructional Value of Negotiable Problem Solving Goals on the Exploratory Learning Continuum, conditions 3 conditions NPSG: Human Tutoring + written material (script) NPSG: Human Tutoring + written material (script) PS: Example tracing tutors (Aleven et. al.) + script PS: Example tracing tutors (Aleven et. al.) + script S: Script only S: Script only Overall order: PS < S < NPSG Overall order: PS < S < NPSG Significant difference between NPSG and PS Significant difference between NPSG and PS Marginal difference between PS and S Marginal difference between PS and S  By-product: Conversational data collected from human tutoring  Difference between tutors One of the tutors was very impatient and terse. One of the tutors was very impatient and terse. Most of his students learnt less than expected from pre-test! >> Content/presentation of tutorial dialog is critical to success

8 8/44 CycleTalk: History

9 9/44 CycleTalk: History  Since we had some conversational data now:  Analysis of Topics Using Info-Magnets Using Info-Magnets Arguello et. al  15 topics identified Each human-tutoring log, covered between 4 to 11 of these topics Each human-tutoring log, covered between 4 to 11 of these topics Correlation between coverage of these topics on learning from the human-tutoring data Correlation between coverage of these topics on learning from the human-tutoring data So, lets get the machine to do the talking. So, lets get the machine to do the talking.

10 10/44 CycleTalk: History  Developing a Tutorial Dialog Systems for CycleTalk  12 Knowledge Construction Dialogs authored based on this topics identified 7 related to optimization relationships 7 related to optimization relationships 5 related to theoretical concepts 5 related to theoretical concepts  Authored for TuTalk  When do we bring these up? Tutor: Let’s stop for a minute and reflect on what we have seen. Having seen how changing T-Max (i.e., the Maximum Temperature at the Heater Outlet) affects the efficiency of the cycle, do you think it is a good idea in general for the Maximum Temperature to be increased and kept high? Student: yes Tutor: Okay. Let’s think now about one important consequence of keeping T-Max high. What happens to the steam quality at the turbine exit if you in-crease the maximum temperature of the cycle? Student: Increases Tutor: Right. As increasing the temperature makes the efficiency and steam quality better, we should increase the maximum temperature of the cycle. But can we keep on increasing the maximum temperature at the heater outlet indefinitely, or are there limitations to keep in mind? Student: there are limitations such as material constraints Tutor: Great. we should try to keep T-Max (i.e., the maximum temperature at the Heater outlet of the cycle) as high as possible without risking safety concerns or exceeding the maximum temperature the material can withstand. Keeping T-Max high increases efficiency and keeps the steam quality high, which are both important. This is an important principle to keep in mind while optimizing the efficiency of your cycle.

11 11/44 CycleTalk: History  Developing a Tutorial Dialog Systems for CycleTalk  12 Knowledge Construction Dialogs authored based on this topics identified 7 related to optimization relationships 7 related to optimization relationships 5 related to theoretical concepts 5 related to theoretical concepts  Authored for TuTalk  Integrated with the example tracing tutor at relevant nodes  So, instead of hints, a window with the dialog agent would pops up Not a clean integration Not a clean integration

12 12/44 CycleTalk: History  Ran an experiment with our first CycleTalk with Tutorial Dialog Kumar et. al., Evaluating the Effectiveness of Tutorial Dialogue Instruction in an Exploratory Learning Context, 2006  3 conditions S – Script only S – Script only PSHELP – PS + Tutorial Dialog triggered in place of some hints PSHELP – PS + Tutorial Dialog triggered in place of some hints PSSUCCESS – PSHELP + Tutorial Dialog triggered on successful completion of certain trace nodes PSSUCCESS – PSHELP + Tutorial Dialog triggered on successful completion of certain trace nodes  Effect size CMU: 0.35σ comparing PSHELP to PSSUCCESS CMU: 0.35σ comparing PSHELP to PSSUCCESS USNA: 0.25σ comparing S to PSSUCCESS USNA: 0.25σ comparing S to PSSUCCESS  Average KCD launches: PSHELP - 1.8, PSSUCCESS – 2.7 PSHELP - 1.8, PSSUCCESS – 2.7 Human Tutoring – 4 to 11 Human Tutoring – 4 to 11

13 13/44 CycleTalk: Fall 2006  Collaborative Learning Setup: Implementations  Interaction Model: Keep the students more engaged: Hinting prompts Hinting prompts “Try to think of an idea related to manipulating a property of the pump.” Every one minute (too many!) Every one minute (too many!) Targeted prompting (based on contribution rate) Targeted prompting (based on contribution rate)  Dynamic/Adaptive support for collaboration vs. Scripting Topic Filter for triggering of dialogs Topic Filter for triggering of dialogs Trained models to classify turns into topics (Taghelper) Trained models to classify turns into topics (Taghelper) Training data from Human-Tutoring corpus Training data from Human-Tutoring corpus (Rosé et. al., 2005) 2 step classification 2 step classification 1.Topic worthiness: SVM (Q: How do we know if worthy?) 2.Topic labeling: TDIDF Scoring  Chatting software Agent as an observer/participant in chat Agent as an observer/participant in chat

14 14/44 CycleTalk: Implementations

15 15/44 CycleTalk: Procedure  15min: CyclePad training (led by experimenter)  70min: Work through material on domain content and using CyclePad  15min: Pre-Test  42 multiple choice, 8 open response questions  Contest announcement  25min: Review and write notes about what students learnt into the chat window   10min: Planning/Synthesis of 2 design plans  25min: Implementation of designs in CyclePad  15min: Post-Test  Questionnaire (Pairs only)

16 16/44 CycleTalk: Experimental Design  Manipulation in step 4: Review and write notes about what students learnt into the chat window  3x2 Full factorial design  Support None (N): No support None (N): No support Static (S): Written material (Script) Static (S): Written material (Script) Dynamic (D): Adaptive dialog agent Dynamic (D): Adaptive dialog agent  Collaboration: Alone (I) / Pair (P)  CMU Sophomores: 87 students over 4 days

17 17/44 CycleTalk: Outcome Metrics  Pre/Post Tests  Objective type questions  Open response questions  Ability to design, implement an efficient Ranking Cycle  Questionnaire

18 18/44 CycleTalk: Results: Collaboration  Effect size: 0.4σ (Q: Does this mean significant?)  Reflection in pairs is more effective than reflection alone

19 19/44 CycleTalk: Results: Support  Positive effect of Dynamic support  Dynamic Support > No support  Effect size: 0.7σ

20 20/44 CycleTalk: Results: Combined  Marginal Interaction, p=.07  Pair+Dynamic > Individual+No Support, 1.2 σ  Pair+Static > Individual+No Support,.9 σ  Individual+Dynamic > Individual+No Support, 1.06 σ

21 21/44 CycleTalk: More Results  Open response questions  Advantage for dialog based support, but not collaboration  Effect size: 0.5 σ (Simpler ANCOVA model)  Practical Assessment:  No significant differences  91% students built one fully defined cycle  64% built two  Questionnaire:  Dynamic support student rate high on benefit, but low on engagement when in pairs  Collaboration & Dynamic support not working together??  Desirable Difficulty? (Ref: Robert Bjork)

22 22/44 CycleTalk: Observations  Individual interaction with agents  Highly tutor directed  Students rarely ignore tutor prompts  More complicated dynamics with pairs  Students talk around tutor agent (Q)  Tutor agent is an interruption  Students treat agent contributions like hints  Evidence that they read although they often don’t respond  Frustration

23 23/44 AIED 2007: Your questions (Q)   Is this reflection?   Dialogs were about material already given to them which they review in step 2   Whether students were receiving instruction twice?   Multi-Party Turn-Taking   This was a problem back then Recent attempts to improve this And more needs to be done…   Why do students ignore the Tutors?   Pairwise Tukey Posthoc analysis   Difference of means/SE > q critical

24 24/44 That was Fall 2006: Since then…  Making the agent more “Social”  Kumar et. al., SLaTE 2007 Engaging students through small talk Engaging students through small talk Promoting Collaboration Promoting Collaboration  Unpublished Manuscript (Kumar et. al.) Attention Grabbing Attention Grabbing  Chaudhuri et. al., AIED 2009 Pointers Pointers  Basilica  Architecture for developing conversational agents  Virtual Environments  ConcertChat  SecondLife  More than 2 students?  Recent work in a freshmen Mechanical Engineering class

25 25/44 Kumar et. al., SLaTE 2007  Mathematics Problem Solving  Fractions Addition, Subtraction, Multiplication, Division Addition, Subtraction, Multiplication, Division Conversation Interface (shared by students & conversational agents) CTAT Problem Solving Interface with Cognitive Tutors based feedback [Aleven, Koedinger] (shared by students)

26 26/44 Kumar et. al., SLaTE 2007: Social Strategy Tutor: Student1, If you had to choose between a long flight and a long car ride, which seems less uncomfortable? Student1: I’ll take the car ride Tutor: Ok Student1. Tutor: Student2, Which are more entertaining – books or movies? Student2: definitely books! Tutor: Ok Student2. Tutor: Please work on the problem shown in the side panel. Motivated by the intention of engaging students  By showing interest in their personal preferences  Goal of this social conversation is to make the students feel that they worked together to construct the problem statement  Comes up every time the students are about to start solving a new problem  Related: (Bickmore & Cassell)  SmallTalk by Embodied Conversational Agent REA Example Jan packed several books to amuse herself on a long car ride to visit her grandma. After 1/5 of the trip, she had already finished 6/8 of the books she brought. How many times more books should she have brought than what she packed?

27 27/44 Kumar et. al., SLaTE 2007: Questionnaire ControlExperimental Perceived Self Competence 4.2 (.56) 4.1 (.23) ns Perceived Partner Competence 4.3 (.62) 3.9 (.49) ns Perceived Benefit of Collaboration 4.5 (.74) 4.4 (.70) ns Perceived Help Received* 1.8 (1.3) 3.3 (.69) Perceived Help Provided* 1.8 (1.1) 3.1 (1.1) Significant (Effect Size = 1.15) Students perceived higher help offering by their partners in the Experimental condition Significant (Effect Size = 1.18) Students perceived they offered more help to their partners in the Experimental condition

28 28/44  Observations from Conversation Analysis  Average number of Help Provisions not significantly different across conditions  More help related episodes per problem in the Experimental condition Mean (Control) = 0.30 Mean (Experimental) = 0.69 Mean (Control) = 0.30 Mean (Experimental) = 0.69 F(1, 15) = 16.8 p <  More episodes of Deny Help in Control condition Mean (Control) = 40.2 Mean (Experimental) = 24.7 F(1, 62) = 3.46 p =  Students displayed more negative attitude in Control conditions Insults (“you stink”, “stupid”) occurred only in Control condition Insults (“you stink”, “stupid”) occurred only in Control condition Kumar et. al., SLaTE 2007: Results

29 29/44 That was Fall 2006: Since then…  Making the agent more “Social”  Kumar et. al., SLaTE 2007 Engaging students through small talk Engaging students through small talk Promoting Collaboration Promoting Collaboration  Unpublished Manuscript (Kumar et. al.) Attention Grabbing Attention Grabbing  Chaudhuri et. al., AIED 2009 Pointers Pointers  Basilica  Architecture for developing conversational agents  Virtual Environments  ConcertChat  SecondLife  More than 2 students?  Recent work in a freshmen Mechanical Engineering class

30 30/44 CycleTalk Fall 2007

31 31/44 CycleTalk Fall 2007 Instructive Conversation with Attention Grabbing Now might be a good time for reflection. Tutor well the idea is to increase the heat in St13BP Power is generated when steam is cooled. If there is more heat input, will there be more or less potential for cooling? Tutor because that increases power St13BP There will be more potential for Cooling. If there is more potential for cooling, is there more or less potential for power generation? Tutor Consider the effect of increasing Qin which is heat input to a cycle. First, let’s consider what happens to power out when Qin is increased. What happens to power out when steam is cooled? Tutor Two Motivational Prompts Solving this problem can be a lot of fun if you cooperate well together Tutor At 2 minute mark Winning isn’t everything. Don’t worry. Be Happy. Tutor At 30 minute mark Instructive Conversation with Attention Grabbing Now might be a good time for reflection. Tutor well the idea is to increase the heat in St13BP Power is generated when steam is cooled. If there is more heat input, will there be more or less potential for cooling? Tutor because that increases power St13BP There will be more potential for Cooling. If there is more potential for cooling, is there more or less potential for power generation? Tutor Consider the effect of increasing Qin which is heat input to a cycle. First, let’s consider what happens to power out when Qin is increased. What happens to power out when steam is cooled? Tutor

32 32/44 CycleTalk Fall 2007: Conversation Analysis i n t r o d u c t i o n : e x p e r i m e n t 1 : e x p e r i m e n t 2 : d i r e c t i o n s Tutoring without Attention Grabbing Tutoring with Attention Grabbing Response to Instructional Turns 2.1%12.5% Relevant Response to Instructional Turns 1.1%8.6% More Attention to the Tutor More Relevant Responses Numbers here are: Average ( Percentage of Instructional turns each student responded to)

33 33/44 CycleTalk Fall 2007: Conversation Analysis i n t r o d u c t i o n : e x p e r i m e n t 1 : e x p e r i m e n t 2 : d i r e c t i o n s WithoutMotivationalPromptsWithMotivationalPrompts Reference to the Tutor 0.30 ± ± 1.25 Positive Reference to the Tutor 10%45%

34 34/44 CycleTalk Fall 2008 (Chaudhuri et. al. ‘09) B6: so i think the best thing to go with first is to increase the pressure entering the connector Tutor: You might want to review what you saw earlier in the sensitivity analysis investigating the effect of changing P-max on steam quality and efficiency, from the top half of page 13 in your book. Tutor: type HELP WITH CHANGEPMAX if you want to discuss it with me. B1: help with changepmax Tutor: Remember the sensitivity analysis you have completed where you manipulated P-Max and observed it's effect on the efficiency of the cycle. You can refer to the figure that I have pasted onto your whiteboard for your convenience. Having seen how changing P-Max, the Maximum Pressure at the Heater Inlet, effects efficiency and Steam, do you think as a general principle the Maximum pressure should be increased ? B6: Yes Tutor: So the Efficiency of a Cycle increases by increasing the maximum pressure of the cycle. In addition to the effect on efficiency, what happens to the steam quality at the turbine exit if you increase the maximum pressure of the cycle ? B6: Decreases Tutor: So let's think for a minute about these competing effects: Increasing P-Max makes the efficiency better but the steam quality degrades. Is the steam quality at the turbine exit acceptable at the maximum allowable pressure within safety limits?

35 35/44 CycleTalk Fall 2008: Experiment  Manipulation (2x2):  Pointer Hints: Yes/No  Dialog Support: Yes/No  Results  Higher learning gains for Pointer+Dialog condition Pointer+Dialog vs Dialog: 0.8 σ Pointer+Dialog vs Dialog: 0.8 σ Pointer+Dialog vs None: 0.6σ Pointer+Dialog vs None: 0.6σ Pointer vs None: 0.35σ Pointer vs None: 0.35σ  Few dialogs in Pointer + Dialog condition compared to Dialog only condition Too many dialogs distracting? Too many dialogs distracting?

36 36/44 That was Fall 2006: Since then…  Making the agent more “Social”  Kumar et. al., SLaTE 2007 Engaging students through small talk Engaging students through small talk Promoting Collaboration Promoting Collaboration  Unpublished Manuscript (Kumar et. al.) Attention Grabbing Attention Grabbing  Chaudhuri et. al., AIED 2009 Pointers Pointers  Basilica  Architecture for developing conversational agents  Virtual Environments  ConcertChat  SecondLife  More than 2 students?  Recent work in a freshmen Mechanical Engineering class

37 37/44 Basilica  Multi-Expert model of building conversational agents  Terminology:  Components, Actors, Filters, Events, Connections  Actors: Generate user perceivable events  Filters: (Everything else, mostly): Interpret events generated by other components  Data is encapsulated as Events

38 38/44 Basilica TuTalk Server ConcertChat Server Student 1 Student 2 Presence Actor TextMessageEvent x x Channel Filter TextMessageEvent OutGoingMessage SpoolingFilter TextMessageEvent CCText Filter x LaunchEvent Launch Filter TookTutorTurnEvent TextMessageEvent TakeTutorTurnEvent TurnTaking Filter GrabAttentionEvent TextMessageEvent GrabAttentionEvent AttentionGrabbed- Event Attention Grabbing Filter TakeTutorTurnEvent TookTutorTurnEvent LaunchEvent TutoringStartedEvent DoneTutoringEvent AttentionGrabbedEvent TakeTutorTurnEvent TookTutorTurnEvent StartTutoringEvent GrabAttentionEvent Tutoring Filter LaunchEvent TextMessageEvent ProduceHintEvent Hinting Filter Polling Prompting Actor LaunchEvent TextMessageEvent Hinting Actor ProduceHintEvent TextMessageEvent Tutoring Actor StartTutoringEvent TakeTutorTurnEvent TextMessageEvent TutoringStartedEvent TookTutorTurnEvent DoneTutoringEvent Attention Grabbing Actor GrabAttentionEvent TextMessageEvent

39 39/44 Basilica  Novel Architecture  Re-usable components  Rapid prototypes  Easy integration of the same agent with many environments  Incremental developments of components  Meta-architecture for bringing together other dialog management components

40 40/44 Conversational Agents in Second Life Session1Session2Session3Session4Session5… MIDDLEWARE TRANSLATIONTRANSLATION Object 1 Internal Representation Message Receiver Message Queue Object 2 Internal Representation Message Receiver Message Queue INTERFACEINTERFACE OBJECT1OBJECT1 INTERFACEINTERFACE OBJECT2OBJECT2 HTTP

41 41/44 That was Fall 2006: Since then…  Making the agent more “Social”  Kumar et. al., SLaTE 2007 Engaging students through small talk Engaging students through small talk Promoting Collaboration Promoting Collaboration  Unpublished Manuscript (Kumar et. al.) Attention Grabbing Attention Grabbing  Chaudhuri et. al., AIED 2009 Pointers Pointers  Basilica  Architecture for developing conversational agents  Virtual Environments  ConcertChat  SecondLife  More than 2 students?  Recent work in a freshmen Mechanical Engineering class

42 42/44 Supporting Groups  Mechanical Engineering Freshmen class  Sort of repeat the process of CycleTalk Initial data collection done with agents though Initial data collection done with agents though One advantage of Basilica here One advantage of Basilica here  Observations: (3 out of 6 sessions) New vocabulary collection (for dynamic triggers) New vocabulary collection (for dynamic triggers) Lots of Sarcasm, Teasing, Cursing, Discontent, Silliness Lots of Sarcasm, Teasing, Cursing, Discontent, Silliness Some positiveness too Some positiveness too Abuses towards Tutor Abuses towards Tutor Opportunities identified that next version of tutor can use Opportunities identified that next version of tutor can use Current irresponsiveness Current irresponsiveness frequent questions/concepts frequent questions/concepts  GRASP  Supporting teachers/facilitators for long-term group assessment

43 43/44 Our work: Questions  Supporting Learning tasks  What kind of tasks?  What kind of support? (Agents ofcourse, but…) What role? What role? Manifestation Manifestation How do we make sure the support is getting through? How do we make sure the support is getting through?  What kind of users? Individuals / Pairs / Groups ? Individuals / Pairs / Groups ? Students ? Teachers/facilitators? Students ? Teachers/facilitators?  What environment?  How do we build this support?  How do we evaluate if the support helps?

44 44/44 Done: Thanks for tuning in. Most questions are good questions. So, Please Ask. i n t r o d u c t i o n : e x p e r i m e n t 1 : e x p e r i m e n t 2 : d i r e c t i o n s


Download ppt "Computer Supported Collaborative Learning Language Technologies Institute Carnegie Mellon University Wednesday, March 18, 2009: Speech and NLP for Educational."

Similar presentations


Ads by Google