Adaptive Spoken Dialogue Systems & Computational Linguistics Diane J. Litman Dept. of Computer Science & Learning Research and Development Center University.

Slides:

Advertisements

Similar presentations

1 Free Help Live Chat Help 24 / 7 / 365 Anywhere, Anytime Answers Reference Research Help Information Easy to Use Fast.

Advertisements

Introducing yourself and Others

Non-Native Users in the Let s Go!! Spoken Dialogue System: Dealing with Linguistic Mismatch Antoine Raux & Maxine Eskenazi Language Technologies Institute.

Mihai Rotaru Diane J. Litman DoD Group Meeting Presentation

5/10/20151 Evaluating Spoken Dialogue Systems Julia Hirschberg CS 4706.

© British Council 2014 like Do sport watching on you television? television your programme? favourite What’s like Do dentist? to going you the films like?

WELCOME HI GOOD MORNING GOOD MORNING GOOD MORNING GOOD MORNING.

Uncertainty Corpus: Resource to Study User Affect in Complex Spoken Dialogue Systems Kate Forbes-Riley, Diane Litman, Scott Silliman, Amruta Purandare.

Machine Learning Case study. What is ML ?  The goal of machine learning is to build computer systems that can adapt and learn from their experience.”

CS 4705 Automatic Speech Recognition Opportunity to participate in a new user study for Newsblaster and get $25-$30 for hours of time respectively.

Detecting missrecognitions Predicting with prosody.

Part II Questions & Responses. Information Questions (1)Who is your sister in this picture? (A) She is generous. (B) There are three girls in this picture.

Frank L. H. WolfsDepartment of Physics and Astronomy, University of Rochester Physics 121. February 5, My favorite airline.

12.0 Computer-Assisted Language Learning (CALL) References: 1.“An Overview of Spoken Language Technology for Education”, Speech Communications, 51, pp ,

Predicting Student Emotions in Computer-Human Tutoring Dialogues Diane J. Litman and Kate Forbes-Riley University of Pittsburgh Pittsburgh, PA USA.

Introduction to machine learning

U.S.A. Learns A Web site to teach English to adult learners Leslie Petty Project IDEAL Support Center A Web site to teach English to adult.

 Today, we will Conduct an Investigation Take some notes Write a short reflection about what we learned  Please Do Now: get your lab notebook get out.

Interactive Dialogue Systems Professor Diane Litman Computer Science Department & Learning Research and Development Center University of Pittsburgh Pittsburgh,

৳ Look, I’ve got a leaflet about it.

Step 1. Find your child’s desk. Step 3. Please fill out the online form if you did not do so on Meet The Teacher Night Step 4. Step 2. Respond back to.

Presentations PLANNING PLAN PURPOSE PEOPLE PLACE AND EQUIPMENT PRESENTING YOURSELF TIMING VISUALS CONTENT STRUCTURE.

Presentations PLANNING PLAN PURPOSE PEOPLE PLACE AND EQUIPMENT PRESENTING YOURSELF TIMING VISUALS CONTENT STRUCTURE.

Spoken Dialogue for Intelligent Tutoring Systems: Opportunities and Challenges Diane Litman Computer Science Department & Learning Research & Development.

circle Adding Spoken Dialogue to a Text-Based Tutorial Dialogue System Diane J. Litman Learning Research and Development Center & Computer Science Department.

Learning, Adaptation and Personalization in Spoken Dialogue Systems Diane J. Litman* University of Pittsburgh Dept. of Computer Science & LRDC

Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun Spoken Dialogue System Diane Litman AT&T Labs - Research Florham.

Experiments with ITSPOKE: An Intelligent Tutoring Spoken Dialogue System Dr. Diane Litman Associate Professor, Computer Science Department and Research.

A Successful Dialogue without Adaptation S: Hi, this is AT&T Amtrak schedule system. This is Toot. How may I help you? U: I want a train from Baltimore.

On the phone In this unit you are going to: take and report telephone messages. make, accept ad refuse invitations. listen for details. read about telephone.

Unit 1 Sections 1-5 Sentence Frames

Collaborative Research: Monitoring Student State in Tutorial Spoken Dialogue Diane Litman Computer Science Department and Learning Research and Development.

1 Natural Language Processing Lecture Notes 14 Chapter 19.

Why predict emotions? Feature granularity levels [1] uses pitch features computed at the word-level Offers a better approximation of the pitch contour.

Using Word-level Features to Better Predict Student Emotions during Spoken Tutoring Dialogues Mihai Rotaru Diane J. Litman Graduate Research Competition.

ACE TESOL Diploma Program – London Language Institute OBJECTIVES You will understand: 1. The terminology and concepts of semantics, pragmatics and discourse.

Speech and Language Processing for Educational Applications Professor Diane Litman Computer Science Department & Intelligent Systems Program & Learning.

Spoken Dialogue in Human and Computer Tutoring Diane Litman Learning Research and Development Center and Computer Science Department University of Pittsburgh.

Spoken Dialog Systems Diane J. Litman Professor, Computer Science Department.

Unit One. To be ABSENT AFTERNOON AGAIN, REPEAT.

Experiences with Undergraduate Research (Natural Language Processing for Educational Applications) Professor Diane Litman University of Pittsburgh.

Using Prosody to Recognize Student Emotions and Attitudes in Spoken Tutoring Dialogues Diane Litman Department of Computer Science and Learning Research.

Misrecognitions and Corrections in Spoken Dialogue Systems Diane Litman AT&T Labs -- Research (Joint Work With Julia Hirschberg, AT&T, and Marc Swerts,

(Speech and Affect in Intelligent Tutoring) Spoken Dialogue Systems Diane Litman Computer Science Department and Learning Research and Development Center.

circle Spoken Dialogue for the Why2 Intelligent Tutoring System Diane J. Litman Learning Research and Development Center & Computer Science Department.

1 Spoken Dialogue Systems Error Detection and Correction in Spoken Dialogue Systems.

Unit 9 Have you ever been to an amusement park? Section B.

Curriculum Night. Welcome Parents!  This is your student’s Homeroom – we read for ten minutes in here every day, so please help ensure that they have.

Spoken Dialogue for Intelligent Tutoring Systems: Opportunities and Challenges Diane Litman Computer Science Department Learning Research & Development.

Improving (Meta)cognitive Tutoring by Detecting and Responding to Uncertainty Diane Litman & Kate Forbes-Riley University of Pittsburgh Pittsburgh, PA.

Designing and Evaluating Two Adaptive Spoken Dialogue Systems Diane J. Litman* University of Pittsburgh Dept. of Computer Science & LRDC

Experiments with ITSPOKE: An Intelligent Tutoring Spoken Dialogue System Diane Litman Computer Science Department and Learning Research and Development.

Detecting and Adapting to Student Uncertainty in a Spoken Tutorial Dialogue System Diane Litman Computer Science Department & Learning Research & Development.

Prosodic Cues to Disengagement and Uncertainty in Physics Tutorial Dialogues Diane Litman, Heather Friedberg, Kate Forbes-Riley University of Pittsburgh.

Mixed-Up Dialogues at home (slides 2-4) on transport (5-6) at school (7-11) on the weekend (12-15) being friendly (16 -25)

I have a birthday party. Task 2 accept and decline invitations.

FIGHTING WORDS n It is what you say u it triggers what they hear u it helps them set their minds.

Predicting and Adapting to Poor Speech Recognition in a Spoken Dialogue System Diane J. Litman AT&T Labs -- Research

Gravity and Projectile Motion

Chapter 6. Data Collection in a Wizard-of-Oz Experiment in Reinforcement Learning for Adaptive Dialogue Systems by: Rieser & Lemon. Course: Autonomous.

Towards Emotion Prediction in Spoken Tutoring Dialogues

Learning for Dialogue.

Dialogue-Learning Correlations in Spoken Dialogue Tutoring

UNIT 6 Travel Agencies.

Error Detection and Correction in SDS

Issues in Spoken Dialogue Systems

Spoken Dialogue Systems

Automatic Speech Recognition

Spoken Dialogue Systems

Joys and Pains of building a Dialog System

Presentation transcript:

Adaptive Spoken Dialogue Systems & Computational Linguistics Diane J. Litman Dept. of Computer Science & Learning Research and Development Center University of Pittsburgh Pittsburgh, PA USA \

Spoken Dialogue Systems Front end: ASR (automatic speech recognition) and TTS (text to speech) Back end: DB, web, tutor, chat room, etc. Middle: dialogue strategy (what action to take at each point) user ASR TTS backend spoken dialogue system

Overview of Issues Problem: building a spoken dialogue system is currently more of an art than a science design choices strongly influence performance most choices made by human intuition Hypothesis: performance can be improved by learning how to make optimal choices dynamic system adaptation when problems do occur

Why / When / How to Adapt Many types of problems –Task failure, poor speech recognition, poor natural language processing, user emotional state Many types of clues –Words, intonation, timing, emotion, logged information Many forms of adaptation –machine learning vs. user-led –What kind of learning algorithms? Many levels of granularity –dialogue vs. subdialogue vs. utterance

RLDS: Adaptive System I Optimization of dialogue strategy via reinforcement learning Parameters: linguistic state (to monitor), reward (to optimize) Example Result: task success increases from 51.5% to 63.5% TTS: Welcome to RLDS. Please say an activity name or say 'list activities' for a list of activities I know about. USER: I would like to visit a historic site in Stanhope this morning. TTS: Did you say you are interested in going to a zoo? USER: No. TTS: I know about amusement parks, aquariums, cruises, historic sites, museums, parks, theaters, wineries, and zoos. Please say an activity name from this list. USER: I would like to visit a historic site. TTS: Did you say you are interested in going to a historic site? USER: Yes. TTS: Please give me more information. USER:... TTS: Please tell me the location that you are interested in. You can also tell me the time. …. (8 turns omitted) TTS: Thank you for using the system. Please give me feedback by saying 'good', 'so-so', or 'bad'. USER: Bad. TTS: Goodbye and have a nice day! TTS: Welcome to RLDS. How may I help you? USER: I would like to um find about Stanhope historic sites. TTS: Did you say you are interested in Stanhope? USER: Yes. TTS: What time of the day do you want to go? USER: Uh...morning. TTS: Did you say you want to go in the morning? USER: Yes. TTS: I found 2 historic sites in Stanhope that are open in the morning. They are the Lenape Indian Village, and the Waterloo Village. Thank you for using the system. Please give me feedback by saying 'good', 'so-so', or 'bad'. USER: Good. TTS: Goodbye and have a nice day! Unoptimized System Optimized System

TOOT: Adaptive System II Dynamic system adaptation to particular users within particular dialogues via supervised learning Monitoring of subdialogues for ASR performance increases task success from 23% to 65% Monitoring of prosodic, lexical, and contextual features accurately predicts user error detection and recovery utterances

A Problematic Dialogue S: Hi, this is AT&T Amtrak schedule system. This is TOOT. How may I help you? U: I'd like to find a train from Ch- to Chicago from Baltimore. MISRECOGNITION S: Sorry, I can't understand you. Please repeat your utterance. U: I'd like to find a train from Baltimore to Chicago. MISRECOGNITION S: I heard you say “leave today around 5 thirty in the morning." Which city do you want to go to? U: I would like to leave Saturday at eight in the morning. MISRECOGNITION S: You can say the arrival city name, such as “New York City." U: Baltimore to Chicago. …

A Successful Dialogue after Adaptation S: Hi, this is AT&T Amtrak schedule system. This is Toot. How may I help you? U: I need to find a train going to New York City. S: Which city do you want to leave from? U: Baltimore. S: Do you want me to find the trains from Baltimore to DC on Thursday at anytime now? U: No. S: How may I help you? U: I need to leave on Sunday. **TOOT adapts to more conservative dialogue behaviors** S: I heard you say go to DC between 1 night and 8. Do you want me to find the trains from Baltimore to DC on Thursday between 1 night and 8 now? U: No.

S: You can say yes or no. U: No. S: How may I help you? U: I need to go to New York City from Baltimore on Sunday at 8:30 pm. S: I heard you say go to Chicago on Tuesday around 8:45 am. Do you want me to find the trains from Baltimore to Chicago on Tuesday around 8:45 am now? U: No. **TOOT adapts to even more conservative dialogue behaviors** S: Which city do you want to go to? U: New York City. S: Do you want to go to New York City? U: Yes. S: Which city do you want to leave from? U: Baltimore...

ITSPOKE: Adaptive System III Adaptation of intelligent tutoring system’s pedagogical dialogue strategies to student attitudes and emotions Tutor26: Let me ask my question in a more general way. What happens to the velocity of a body when there is no force acting on it? Student27:dammit (ASR: it is) Tutor28 :Could you please repeat that? Student29 :same (ASR: i same) Tutor30 :Right. And what happens to the HORIZONTAL velocity of a body when there is no HORIZONTAL force acting on it? Student31:zero (ASR: the zero) Tutor32 :Well... OK, let's consider how forces in the vertical direction affect the motion of a body in a horizontal direction in a different scenario Student33:oh gravity you already said this (ASR: beats gravity you're exert less) Tutor34:Fine. Are there any other forces acting on the apple as it falls? Student35:no why are you doing this again (ASR: no y and to it yes) Tutor36:Correct. The only force acting on the apple as it falls to the earth is the earth's gravitational force. In which direction does gravitational force act? Student37:downward you computer (ASR: downward you computer)

Summary Adaptation/Personalization via Learning can improve dialogue system performance RLDS: Given exploratory dialogues, reinforcement learning can be used for online system optimization TOOT: By learning from training dialogues, a system can dynamically look for problems and adapt ITSPOKE, Your System here: New types of voice-enabled systems pose many interesting opportunities