Presentation is loading. Please wait.

Presentation is loading. Please wait.

Learning to Generate Utterances for Spoken Dialogue Systems by Mining User Reviews Prof. Marilyn Walker (Higashinaka, Prasad and Walker, COLING/ACL 2006)

Similar presentations


Presentation on theme: "Learning to Generate Utterances for Spoken Dialogue Systems by Mining User Reviews Prof. Marilyn Walker (Higashinaka, Prasad and Walker, COLING/ACL 2006)"— Presentation transcript:

1 Learning to Generate Utterances for Spoken Dialogue Systems by Mining User Reviews Prof. Marilyn Walker (Higashinaka, Prasad and Walker, COLING/ACL 2006)

2 Cognitive Systems University of Sheffield Background Statistical methods are predominant in Spoken Dialogue Systems (SDS) and are quite mature for speech recognition, language understanding, and dialogue act detection tasks (Young 2004) Statistical methods also commonly used in NLP tasks such as information retrieval, automatic summarization, information extraction and machine translation These methods provide scalability and portability across domains

3 Cognitive Systems University of Sheffield Spoken Dialogue System Components

4 Cognitive Systems University of Sheffield Spoken Language Generation DM and SLG typically handcrafted Template Based Generation (usual) o Used in most dialogue systems, generate responses simply matching a template with the current dialogue context  Pros: efficient, highly customized  Cons: not portable cross domain, hard to encode linguistic constraints (subject/verb agreement), non scalable when more than few hundred templates Natural Language Generation (recently) o Clear separation between 1) text (or content) planning, 2) sentence planning, 3) surface realization. Uses general rules for each generation module  Pros: some aspects are portable cross domain and dialogue context  Cons: Specific rules are usually needed to tune the quality of the general rules, could be slow for real-time systems

5 How to say it NLG System Components DM Communicative Goals Text Planner What to say Sentence Planner Surface Realizer Prosody Assigner TTS NLG

6 Cognitive Systems University of Sheffield Spoken Dialogue Generation Dialogue Manager: Select communicative goals Text Planning: Refine communicative goal into structured atomic communicative goals, select content to be expressed Sentence planning: choose linguistic resources (lexicon, syntax) to achieve atomic communicative goals Realization: use grammar (syntax, morphology) to generate surface sentence(s) (Rambow & Korelsky, 1992; Reiter, 1994): “Consensus Architecture”

7 Cognitive Systems University of Sheffield Statistical Methods in SLG/NLG Statistical Surface Realizers: overgenerate and rank using parameters trained from corpora (Halogen, Langkilde 2002; Fergus, Bangalore and Rambow 2000; Chen etal 2002 ) Trainable Sentence Planning: learn which combination operations for aggregation and content ordering produce highest quality output (SPoT, Rambow etal 2002, Walker etal 2004; SPaRKy, Stent etal. 2004) Prosody Assignment: learn from labelled data to assign appropriate prosody (Hirschberg90, Pan etal. 2003)

8 Cognitive Systems University of Sheffield Problem Even the ‘trainable’ sentence planner requires a domain-specific handcrafted generation dictionary to specify the mapping between syntactic realizations and the text plan propositions (e.g., X has good food, X has good service) Mappings are created by hand  It is costly, needed for each domain Variation limited by original mappings and combination operations  Utterances can be unnatural

9 Assert-food_quality(Babbo, superb) Babbo has superb food and decor. Assert-décor(Babbo, superb) Joint relation (RST) Semantic Representations [have [ I proper_noun X] [ II common_noun food [ ATTR adjective superb]]] Assert-food_quality(X, superb)  [have [ I proper_noun X] [ II common_noun decor [ ATTR adjective superb]]] Assert-décor(X, superb)  Dictionary [have [ I proper_noun X] [ II common_noun food [ COORD and [II common_noun décor]] [ ATTR adjective superb]]] Utterance Example: SLG

10 Cognitive Systems University of Sheffield Solution? In many domains, there are web pages that describe and evaluate domain entities These web pages may include: o Textual reviews of domain entities o Scalar ratings of specific attributes of domain entities on a per review or per entity basis o Tabular data with values for particular attributes o Domain or product specific ontologies Is it possible to mine these corpora to bootstrap a spoken language generator?

11 Restaurant Domain

12 Sample Restaurant Review

13 Cognitive Systems University of Sheffield Hotel Domain a little gem!! Submitted by: kathy b. of cincinnati, oh usa ; May 03, 2006 Date of visit: 12/05 Traveler's Rating: 5 the history, ambience and old world charm of the algonquin are a unique combination that appeals very much. staff is very friendly and helpful; rooms small but restored to period charm. great lobby. say hello to matilda, the resident cat, another great algonquin tradition. Best Feature: staff & ambience Needs Improvement: not a thing Amenities rated on a 1-5 scale: (1=Lowest 5=Highest N/A=Not Rated) Rooms = 4 Dining = 5 Public Facilities = 5Sports/Activities = N/A Entertainment = 4 Service = 5

14 Cognitive Systems University of Sheffield What can be used Textual reviews of domain entities Scalar ratings of specific attributes of domain entities on a per review or per entity basis Tabular data with values for categorial attributes Specified attributes => partial ontology

15 Cognitive Systems University of Sheffield Bootstrapping SLG Automatically acquire dictionary entries from user reviews on the web Dictionary entry is a triple (U,R,S): o U (Utterance), R (Semantic representation), and S (Syntactic structure) Use user ratings and categorial attribute values to pinpoint semantic representation Use Minipar parser and DSyntS converter to produce dependency syntactic representation (Lavoie and Rambow 98, Mel’čuk, 1988)

16 Cognitive Systems University of Sheffield Related Work Create dictionary from parallel corpus (Barzilay et al., 2002)  Requires a corpus of parallel semantic representation/syntactic realizations Information Extraction: Find phrases/patterns expressing particular relations  Don’t know entities that realize the relations, want most frequent/general pattern Find opinion expressions in reviews o Adjectives for products (Hu and Liu, 2005) o Product features and adjectives with polarity (Popescu and Etzioni, 2005)  Do not focus on creating dictionary

17 Cognitive Systems University of Sheffield Method Create a population of utterances U from user reviews For each U Derive semantic representation R Derive syntactic structure S Filter inappropriate mappings Add remaining mappings to dictionary

18 Cognitive Systems University of Sheffield Experiment Obtaining dictionary for restaurant domain Data collected from we8there.com o 3,004 user reviews on 1,810 restaurants o 18,466 review sentences o 451 mappings after filtering Objective evaluation Subjective evaluation

19 Cognitive Systems University of Sheffield Collect user reviews Select review websites with individual ratings for review entities Collect review comments and ratings Collect tabular data Tabular Data Name, Food Type, Location Ratings Food, Service, Value, Atmosphere, Overall

20 Cognitive Systems University of Sheffield Derive Domain Ontology Assume meronymy relation between: o Any attribute that the user rates o Any attribute for which categorical values are specified on the web page Relations: RESTAURANT has foodquality RESTAURANT has servicequality RESTAURANT has valuequality RESTAURANT has atmospherequality RESTAURANT has overallquality RESTAURANT has FOODTYPE RESTAURANT has LOCATION

21 Cognitive Systems University of Sheffield Hypothesis/Assumptions Closed Domain: At least some of the utterances in reviews realize the relations in the domain ontology (identify these utterances) Hypoth: If an utterance U realizes named entities corresponding to the domain entity and the distinguished attributes, then R for that utterance includes the relation concerning that attribute in the domain ontology.

22 Cognitive Systems University of Sheffield Specify Lexicalizations of attributes AttributeLexicalizations foodfood, meal serviceservice, staff, waitstaff, wait staff, server, waiter, waitress atmosphereatmosphere, décor, ambience, decoration valuevalue, price, overprice, pricey, expensive, inexpensive, cheap, affordable, afford overallrecommend, place, experience, establishment

23 Cognitive Systems University of Sheffield Create and Label Named Entities Scrape web pages structured data for named entities for categorial attributes o Foodtype => Spanish, Italian, French, … o Location => New York, San Francisco, London Run Gate on U to label o Named entities o Lexicalizations of attributes

24 Cognitive Systems University of Sheffield Derive Syntactic Representation Run Minipar parser Convert Minipar output to DSyntS for RealPro (Lavoie and Rambow 98)

25 The best Spanish food in New York. The best {NE=foodtype, string=Spanish} {NE=food, string=food, rating=5} in {NE=location, string=New York}. RESTAURANT has FOODTYPE RESTAURANT has foodquality=5 RESTAURANT has LOCATION DSyntS (S) Review Sentence (U) NE-tagged Review Sentence Semantic Representation (R) Ratings Food=5, Service=5, Value=5, Atmosphere=5, Overall=5 Review Comment The best Spanish food in New York. I am from Spain and I had my 28th birthday… DSyntS converter

26 Cognitive Systems University of Sheffield Filter dictionary entries No Relations Filter, Other Relations Filter  Check whether a mapping has exactly the relations expressed in the ontology Contextual Filter  Check whether U can be uttered independently of the context (looks for context words) Parsing Filter  Check whether S( U ) regenerates U Unknown Words Filter (typos, common nouns, etc.)

27 Cognitive Systems University of Sheffield Filtering Statistics filteredretained No Relations Filter7,94710,519 Other Relations Filter5,3515,168 Contextual Filter2,9732,195 Unknown Words Filter1, Parsing Filter Duplicates Filter61 451

28 U: The river was beautiful and the food okay. We had a wonderful time. What an awful place. R: RESTAURANT has foodquality=3 Filtered by No Relations Filter RESTAURANT has overallquality=1 Filtered by Unknown Words Filter (“river” is a common noun) S: Filtered by Parsing Filter This DSyntS generates “What awful place.” Filtering Examples

29 Cognitive Systems University of Sheffield Objective Evaluation Domain coverage o How many relations are covered by the dictionary? Linguistic variation o What do we gain over handcrafted dictionary? Generativity o Can the dictionary entries can be used in conventional sentence planner ?

30 Cognitive Systems University of Sheffield Domain Coverage Total food service atmosphere value overall Total Distribution of single scalar-valued relation mappings

31 Cognitive Systems University of Sheffield Multi-Relation Entries (122 in all) Food-service: 39 Food-value: 21 Atmosphere-food: 14 Atmosphere-service: 10 Atmosphere-food-service: 7 Food-Foodtype: 4 Atmosphere-food-value: 4 etc

32 Cognitive Systems University of Sheffield Linguistic Variation 137 syntactic patterns 275 distinct lexemes, 2-15 lexemes per DSyntS (mean 4.63) 55% syntactic patterns ~: ATTR is AP o The atmosphere is wonderful. o The food is excellent and the atmosphere is great 45% are not. o An absolutely outstanding value with fantastic foodtype food.

33 Cognitive Systems University of Sheffield Examples Food Adjectival phrases: Attribute specificity RATING: ADJECTIVE 1:awful, bad, cold, burnt, very ordinary 2: acceptable, bad, flavored, not enough very bland, very good 3: adequate, bland and mediocre, flavorful but cold, pretty good, rather bland, very good 4: absolutely wonderful, awesome, decent, excellent, good and generatou, very fresh and tasty 5: absolutely delicious, ample, well seasoned and hot, delicious but simple, delectable and plentiful, fancy but tasty, so very tasty

34 Cognitive Systems University of Sheffield Example Service Adjectival phrases RATING: ADJECTIVE 1: awful, bad, forgetful and slow, marginal, young, silly and inattentive 2: overly slow, very slow and inattentive 3: bland and mediocre, friendly and knowledgeable, pleasant, prompt 4: all very warm and welcoming, attentive, extremely friendly and good, great and courteous, swift and friendly, very friendly and accommodating 5: polite, great, all courteous, excellent and friendly, fabulous, impeccable, intrusive, legendary, very friendly, very friendly and totally personal, very helpful, very timely

35 Cognitive Systems University of Sheffield Example Atmosphere Adjectival phrases RATING: ADJECTIVE 2: eclectic, unique and pleasant 3: busy, pleasant but extremely hot 4: fantastic, great, quite nice and simple, typical, very casual, very trendy 5: beautiful, comfortable, lovely, mellow, nice and comfortable, very cozy, very intimate, very relaxing, warm and contemporary

36 Cognitive Systems University of Sheffield Generativity Incorporate the learned mappings into SPaRKy generator (Stent et al., 2004) Combination operations need extension because of assumption that restaurant name is the subject of the utterance: ORIGINAL: Because it has excellent food, superb service and excellent décor, Babbo has the best overall quality among the selected restaurants. MODIFIED: Because the food is excellent, the wait staff is professional and the decor is beautiful and very comfortable, Babbo has the best overall quality among the selected restaurants

37 Cognitive Systems University of Sheffield Subjective Evaluation 10 native English speakers Compare baseline and learned mappings o 27 hand-crafted mappings from SPaRKy o 451 learned mappings Evaluation criteria: o Consistency between semantic representations and realizations o Naturalness/colloquialness of realizations o 1-5 Likert scale

38 Cognitive Systems University of Sheffield Results baselinelearned Consistency baselinelearned Naturalness Consistency is significantly lower, but still high Naturalness is significantly higher

39 Cognitive Systems University of Sheffield Conclusion A new method for automatically acquiring a generation dictionary in spoken dialogue systems Applied to hotel domain and had results in one day Reduce the cost involved with hand-crafting a spoken language generation module Achieve more natural system utterances using attested language examples Results suggest that this approach is promising

40 Cognitive Systems University of Sheffield Future Work Issues of ‘meaning’ vs. polarity, not substitutable Handcrafted lexicalizations: need to automatically generate lexicalizations for domain concepts (increase recall?) Method for extending domain ontology (food is plentiful, delicious, beautifully prepared) More complex domains


Download ppt "Learning to Generate Utterances for Spoken Dialogue Systems by Mining User Reviews Prof. Marilyn Walker (Higashinaka, Prasad and Walker, COLING/ACL 2006)"

Similar presentations


Ads by Google