Presentation is loading. Please wait.

Presentation is loading. Please wait.

Spoken Language Generation: Information Presentation Johanna D. Moore University of Edinburgh Human Communication Research Centre School of Informatics.

Similar presentations


Presentation on theme: "Spoken Language Generation: Information Presentation Johanna D. Moore University of Edinburgh Human Communication Research Centre School of Informatics."— Presentation transcript:

1 Spoken Language Generation: Information Presentation Johanna D. Moore University of Edinburgh Human Communication Research Centre School of Informatics

2 Introduction Much of the research in SDS has focused on the information gathering phase of the dialogue Communicator evaluation indicates that task duration is negatively correlated with user satisfaction (r = -0.31, p <.001) (Walker, Passonneau, Boland, ACL 2001) What contributes most to dialogue duration?

3 Communicator 2001 Dataset SystemUser Utterances56%44% SystemReq_infoPresent_InfoOther Utterances 43%25%32% Time 31%54%15% Words 28%50%22% Time69%31% Words91%9%

4 Typical information presentation SYS:I found 9 round trips from Hartford to Orlando on 31st and returning Hartford on November 6th. For option 1, I have an outbound flight with US Air departing at 7 AM and arriving at 9 49 AM non-stop. A return flight is with U S Air departing at AM and arriving at AM with 1 stopover. Ticket price is 165 dollars. Please say “next option”, or, “flight details”, or, “I’ll take it”. USER: NEXT OPTION SYS: For option 2, I have an outbound flight with U S Air departing at 3 30 PM and arriving at 6 19 PM nonstop. A return flight is with U S Air departing at 8 25 PM and arriving at AM with 1 stopover. Ticket price is 165 dollars. Say “next option”, or, “flight details”, or, “I’ll take it”. USER: NEXT OPTION SYS:...

5

6 Current spoken dialogue systems Presents each options and selected attributes individually  No comparisons Relies on users to remember trade-offs between multiple options  Places large memory burden on users Potentially large payoff for improvements in information presentation

7 Improving Information Presentation Apply techniques from text generation  Allows customization to user(s) and discourse history  Improve quality of synthesis by using NL generator to provide info about both meaning and linguistic structure of utterance Overview of talk:  Case study: FLIGHTS system  Statistical approaches to generation See also Computer Speech and Language (2002) 16. Special Issue on Spoken Language Generation

8 What NLG Can Do For You… User:I want to travel from Edinburgh to Brussels, arriving by 5 pm. System:There’s a direct flight on BMI with a good price. It arrives at four ten p.m. and costs one hundred and twelve pounds. The cheapest flight is on Ryanair. It arrives at two p.m. and it costs just fifty pounds, but you’d need to connect in Dublin. System:You can fly business class on British Airways, arriving at four twenty p.m., but you’d need to connect in London Heathrow. There is a direct flight on BMI, arriving at four ten p.m., but there’s no availability in business class. For a starving student For a business traveller

9 Sentence Realizer Sentence Planner English Knowledge Sources Comm Goals Linguistic Knowledge Sources Discourse Strategies Aggregation Rules Referring Expression Generation Algorithm Lexicon Grammar Dialogue History Domain Model User Model Text Plan Sentence Plan(s) Content Selection Discourse Planning Aggregation Referring Expression Gen Lexical Choice Text Planner

10 FLIGHTS architecture Dialogue Manager (DIPPER) Semantic Interpretation Comm Goals Natural Language Understander (Word spotting) TTS (Festival) ASR (HTK) Text String User Input Text Planning (O-Plan) Sentence Planner (XSLT) Realizer (OpenCCG) System Response Response Generator Text String w/ APML Markup User Model Flight DB Content Selection

11 Customization happens everywhere Content selection: what flights and attributes to present to user Discourse planning: ordering of content, discourse relations Referring Expression Generation: e.g., The cheapest flight, the five-fifteen, a KLM flight Aggregation: grouping propositions into clauses and sentences, e.g., There’s a KLM flight arriving Brussels at ten to five, but business class is not available and you’d need to connect in Amsterdam Discourse cues: e.g., Although, because, but Scalar Adjectives: e.g., good price, just fifty pounds

12 Content Selection Need a domain (or genre) specific method for determining what to say In FLIGHTS:  Rank options based on predicted utility for the user  Select all options whose value is over a threshold  Select attributes that contribute most to value of selected options (Moore, Foster, Lemon & White, FLAIRS 2004, Carenini & Moore, AI Journal, 2006)

13 Discourse Planning Using discourse strategies for producing user-adapted recommendations, comparisons Produces text plans consisting of basic dialogue acts and rhetorical relations Orders presentation of options Groups attributes into positive and negative lists for contrasts Selects attributes to identify flights  a direct flight, the cheapest flight, the KLM flight Marks items as theme/rheme for information structure

14 Information Structure Theme/Rheme  Theme: part of utterance that connects it to prior discourse  Rheme: part of utterance that advances the discussion by contributing novel information  Theme and rheme phrases marked by distinctive combinations of pitch accents and boundary tones Focus/Background  Focus: words whose interpretations contribute to distinguishing the theme or rheme from other contextually available alternatives; marked by pitch accents  Background: the unmarked parts of themes and rhemes (Steedman )

15 Examples Ex 1: I know when the Ryanair flight LEAVES, but when does it ARRIVE? (The Ryanair flight ARRIVES focus ) theme (at FIVE focus ) rheme L+H* LH% H* LL% Ex 2: I know the KLM flight arrives at FOUR, but which flight arrives at FIVE? (The RYANAIR focus flight) rheme (arrives at FIVE focus ) theme H* LL% L+H* LH%

16 Assigning theme/rheme in FLIGHTS First option all rheme Subsequent items:  Identifying, contrastive information is theme  Implements notion of an implicit Question Under Discussion, e.g., After presenting a flight that’s not direct, there’s an implicit question: Are there any direct flights? You can fly business class on British Airways, arriving at four twenty p.m., but you’d need to connect in Manchester. [There’s a DIRECT flight] theme on BMI, arriving at four ten p.m., but there’s no availability in business class.

17 Controlling Intonation with OpenCCG OpenCCG realizer adapts previous work on chart realization to CCG, enabling CCG’s unique accounts of coordination and intonation to be employed in NLG systems Uses information structure to determine types and locations of pitch accents and boundary tones Measures similarity of realizations to n-gram language model Treats agenda as priority queue ordered by n-gram scores Yields best-first anytime algorithm: returns best scoring realization at “any time”, for interactive applications (White and Baldridge, EWNLG9 2003; White, INLG 2004; White, RLaC 2006)

18 The cheapest (L+H*) flight (LH) is on Ryanair (H* LH). It arrives at two p.m (H* LH) and it costs just fifty (H*) pounds (H* LH), but you’d need to connect (H*) in Dublin (H* LL). unit selection limited domain with APML markup Even though the first (L+H*) flight is not on BMI (L+H* LH), it is the cheapest (H*) one available (LH). unit selection limited domain with APML markup Examples

19 Q:I'd like a cheap flight from Frankfurt to Geneva, please. And I'd prefer to fly direct. A:There's a direct flight on Lufthansa with a good price, arriving in Geneva at ten thirty nine am and it costs two hundred and fifty five pounds. The cheapest flight is on Air France arriving at one twenty five pm and it costs only one hundred and five pounds, but it requires a connection in Paris Charles de Gaulle. limited domain limited domain with APML markup Examples

20 Is Tailoring Effective? Evaluation in MATCH Project: Restaurant recommendation system built using same user modeling techniques Subjects heard dialogues where recommendations and comparisons were based on own user model or random other model Subjects judge tailored responses significantly higher  Information quality: System’s response is easy to understand and provides exactly the information I am interested in when choosing a restaurant.  Ranking confidence: Recommended restaurant is somewhere I would like to go. (Walker, Whittaker, Stent, Maloor, Moore, Johnson, Vassiredy, Cognitive Science 28, 2004)

21 Does Intonation Matter? Affects meaning  “She only ATE the banana” vs.  “She only ate the BANANA” Human judgements of output in travel domain show that overall German speech produced with GToBI markup judged better than default intonation (Kruijff-Korbayova, EACL03) Naturalness (Ease of comprehension)

22 Evaluation Compared three synthesizers  Unit Selection Multisyn  Limited Domain  Limited Domain APML Hypotheses:  LD_APML >> USM  LD_APML > LD (Neide Franca Rocha, MSc, 2004)

23 Results: US vs. LD_APML

24 Results: LD vs. LD_APML

25 N-gram scoring & synthesis

26 Using N-gram LM in Generation

27 UM Approach to Info Pres +UM provides information users want to make choices with high confidence +Enables concise presentation of options and their tradeoffs +Users prefer recommendations tailored to their model - Doesn’t scale to large number of options - Does not provide users with an overview of options - Users may (perceive) that they’ve missed out on options

28 Summarize-and-Refine Approach Clusters options in database  Based on attributes that yield smallest number of clusters Summarize these clusters  enumerate, majority, count, singleton User provides additional constraints (Polifroni, Chung & Seneff, Eurospeech 2003, Chung ACL 2004)

29 Example User:What restaurants do you have? Sys:I found 983 restaurants. Most of them are located in Boston and Cambridge. There are 32 choices for cuisine. I also have information about price range. User:Okay tell me about the ones in Boston. Sys: I found 401 restaurants in Boston. There are 29 choices for cuisine. User:Do you have any that serve seafood? Sys:I have found 19 seafood restaurants. They are predominantly in the Back Bay, the North End, South Boston and the South End. User:In the South End. Sys:I have found 3 restaurants. Grill Fish on …

30 SR Approach: +Useful when users don’t know the structure of the database + Domain independent  no fixed set of attributes  make dynamic decisions based on analysis of data - Suboptimal choice of attribute(s) for summarization -Exploration of tradeoffs difficult -May include irrelevant entities

31 The Combined UM+SR Approach select relevant options structure them based on ranking of attributes in user model automatically determine tradeoffs tailor summaries improve overview of options space by briefly summarizing irrelevant options (Demberg and Moore, EACL 2006)

32 Example Dialogue Turn User: I'd like to book a flight from Edinburgh to Brussels for tomorrow. Sys:Ryan Air offers the cheapest flights to Brussels. They cost just 49 pounds but you need to connect in Dublin. There are two flights to choose from. There's an early flight leaving Edinburgh at 8:05 am and arriving at 12:45 pm. To leave later, you can take the 1pm flight arriving Brussels at 5:30 pm. If you want to fly direct, there's a flight on BMI that leaves Edinburgh at 12 pm. It arrives at 1:35 pm and costs 112 pounds. All other flights are more expensive. User Profile “student”: 1 price 2 number of legs departure time arrival time travel time 6 airline fare class layover airport

33 Evaluation Within-participants laboratory experiment  38 subjects  Participants read 6 dialogue pairs (UM+SR vs. SR) Four questions after each dialogue: 1. Did the system give the information in a way that was easy to understand? 2. Did the system give X a good overview of the available options? 3. Do you think there were better options the system did not tell X about? 4. How quickly did the system allow X to find the optimal flight? Forced-choice question after each pair:  Which system would you recommend to a friend?

34 Results - Forced Choice Q.

35 Results - Likert Scale Questions Q2: Under- standability Q3: Overview Q4: Con- fidence Q5: Quick access (1-3 scale) UM+SRSR Mean Likert Scale Value Significance levels using two-tailed paired t-test Q2: p = 0.97 Q3: p < Q4: p < Q5: p < 0.001

36 Exp 2: Overhearer mode Q2: Understandability Q3: Overview Q4: Confidence Q5: Quick Access Mean Likert Scale Value Significance levels using two-tailed paired t-test Q2: p = 0.24 Q3: p < 0.01 Q4: p < Q5: p < 0.10

37 Summary Integration of UM and Clustering allows system to navigate through a large set of options  structure options according to users' valuations  present relevant options only automatically present tradeoffs between options Results in increased overall user satisfaction better overview of options increased users' confidence in system

38 Learning Content Selection Rules Content selection rules for biographical summaries (Duboue & McKeown, EMNLP 2003)  Uses a corpus of textual biographies and corresponding frame-based knowledge representation  Anchor-based alignment of extracted facts with sentences in text corpus  Learns whether semantic unit should be included in biography »Recall 94%, F-score 51%  Induce rules from included material

39 Learning Content Selection Rules Collective classification for content selection (Barzilay and Lapata, HLT/EMNLP 2005) Again, a binary classification task All candidates considered simultaneously Improves coherence because semantically related items often selected together Evaluation: Aligned newswire summaries of NFL games with database of events »Recall 76.5%, F-score 60.15% Include chosen events in summary (as in extractive summarization)

40 Learning for Sentence Planning & Realization SPaRKy (Stent, Prasad & Walker, ACL 2004)  Input: content plan, a set of dialogue acts and rhetorical relations among them  Learns sentence plans from set of human- ranked training examples Oh & Rudnicky, CS&L, 2002  Produces surface realizations for sentence plans based on n-gram statistics Achieves performance comparable to hand-crafted versions

41 Credits: The FLIGHTS System Fancy Linguistically Inspired Generation of Highly Tailored Speech Rob Clark Steve Conway Mary Ellen Foster Kallirroi Georgila Oliver Lemon Michael White Thanks to: UK Engineering and Physical Science Research Council

42 Q:I'd like a cheap flight from Amsterdam to Madrid, please. And I'd prefer to fly direct. A:There's a direct flight on KLM with a good price, arriving in Madrid at twelve twenty one pm and it costs two hundred and seventeen pounds. The cheapest flight is on Air France arriving at eleven forty five am and it costs only one hundred and eighty pounds, but it requires a connection in Paris Charles de Gaulle. unit selection limited domain with APML markup Examples


Download ppt "Spoken Language Generation: Information Presentation Johanna D. Moore University of Edinburgh Human Communication Research Centre School of Informatics."

Similar presentations


Ads by Google