1. Daniel Sonntag |1 LREC 2010 Speech Grammars for Textual Entailment Patterns in Multimodal Question Answering Daniel Sonntag, Bogdan Sacaleanu, DFKI 21/05/2010

2. Daniel Sonntag |2 Outline »Semantic Dialogue Shell »Textual Entailment »Processing Example »Conclusions

3. Daniel Sonntag |3 Acknowledgements »Thanks go out to Robert Nesselrath, Yajing Zang, Günter Neumann, Matthieu Deru, Simon Bergweiler, Gerhard Sonnenberg, Norbert Reithinger, Gerd Herzog, Alassane Ndiaye, Tilman Becker, Norbert Pfleger, Alexander Pfalzgraf, Jan Schehl, Jochen Steigner, and Colette Weihrauch for the implementation and evaluation of the dialogue infrastructure. Robust Question Understanding Ease the interface to external third-party components. SPARQL

4. Daniel Sonntag |4 Semantic Dialogue Shell

5. Daniel Sonntag |5 Dialogue Shell Workflow Speech Interpretation Text Interpretation Gesture Interpretation Graphic Generation Text Generation Speech Interpretation Modality Fusion Presentation Planning Personalisation Dialogue and Interaction Management Interactive Semantic Mediator Interactive Service Compo- sition eTFS/SPARQL SPARQL OWL-API Visualisation Service External Information Sources Semantiic (Meta) Services RDF KOIOS (Yago Ontology) Remote Linked Data Services OWL AOIDE (Music Ontology) Text Summarisation - Domain Model - Context Model - User Model

6. Daniel Sonntag |6 THESEUS’s Semantic Dialogue Shell: Goals and Requirements »Multimodal interaction with the Semantic Web and the Internet of Services »Components customisable to different use case scenarios »Flexible adaptation to »Input and output modalities »Interaction devices »Knowledge bases »To understand a greater number of queries: »Robust question understanding (NLU) when using both speech and written text input »Semantic (i.e., a RDF or OWL based) query interpretation »The combination of robust question understanding and ontology-based answer retrieval

7. Daniel Sonntag |7 SPARQL Query Editor SPARQL

8. Daniel Sonntag |8

9. Daniel Sonntag |9 Speech Grammar zeige ?mir den [werdegang lebenslauf] [von zu] PERSON sage ?mir mehr über den [werdegang lebenslauf] von PERSON wie ist der [werdegang lebenslauf] von PERSON 1.0 mediator:summarizer

10. Daniel Sonntag |10 Textual Entailment Our idea is that an NLU grammar for speech input can be reused to build more robust multimodal text-based question understanding by automatically generating textual entailment patterns.

11. Daniel Sonntag |11 Textual Entailment & Information Access Request Method1 Method2 Information (RDF) Implicit Mapping Method1: Speech / Semantic Grammars RDF/OWL reasoning Method2: RTE textual reasoning Ontology (RDF/OWL) Conceptual Textual Reasoning

12. Daniel Sonntag |12 Textual Entailment through Alignments »For textual entailment to hold we need: »text AND background knowledge  hypothesis »but background knowledge should not entail hypothesis alone »Background Knowledge »Unsupervised acquisition of linguistic and world knowledge from general corpora and web »Acquiring larger entailment corpora »Manual resources and knowledge engineering »Alignment-based TE and Background Knowledge »Preprocessing: POS, morphology, cognates »Representation: bag-of-words »Knowledge Sources: WordNet, Roget‘s Thesaurus, Wehrle Thesaurus

13. Daniel Sonntag |13 Argumentation »Input modalities are usually interpreted according to separate models and aligned to a shared model (often coarse-grained). »Present a method of interpretation based on a common model (propagated changes to multiple modalities). »Built on the grammar for speech inputs = Leveraging Existing Speech Grammar Knowledge

14. Daniel Sonntag |14 Processing Example

15. Daniel Sonntag |15 Entailment Patterns and Possible Hypotheses

16. Daniel Sonntag |16 Entailment Patterns and Alignment Engine »Association-based word alignment. Three steps: »lexical segmentation, when boundaries of lexical items are identified; »correspondence, when possible similarities are suggested in line with some correspondence measures; »alignment, when the most likely semantically similar word is chosen.

17. Daniel Sonntag |17 Entailment Patterns and Alignment Techniques »Question: What is the birthplace of Angela Merkel? »Pattern: Where is Angela Merckel born? »Filters on a full alignment.

18. Daniel Sonntag |18 Entailment Patterns and Alignment Techniques POS Filter: Exclude unlikely alignments based on POS. Allow for the additional mappings: verb to noun (i.e., born vs. birthplace) Lexical Semantic Resource Filter: WordNet (synonyms); Roget Thesaurus (conceptually related words) String Similarity Filter: Dice coefficient, Longest common subsequence ratio; submatches, misspellings System of weights: nouns, verbs, and adjectives are better scored than function words.

19. Daniel Sonntag |19 Dialogue Example »(1) U: “Open my personal address book. What do you know about Claudia?” »(2) S: “There’s an entry: Claudia Schwartz. The personal details are shown below. She lives in Berlin.” + Google Map Display of street coordinates. »(3) U: “Which is Claudia’s favorite kind of music? Do you know the bands she likes most?” »(4) S: “Nelly Furtado” + Displays videos obtained from YouTube. (Rest API) »(5) U: “How did experts rate her last album?” »(6) S: Shows an expert review according to the BBC Linked Data Set. »(7) U: “Show me other news.” »(8) S: Opens a browser + Text field and a new agency Internet page (featuring Angela Merkel) »(9) U writes: “Where was Angela Merkel born? / In which town was Angela Merkel born?” etc. »(10) S: “She was born in Hamburg.” »(11) U speaks again: “And Barack Obama?” »(12) S: “He was born in Honolulu.” »(13) U: “Show me Angela Merkel’s career.”

20. Daniel Sonntag |20 Touchscreen Installation

21. Daniel Sonntag |21 Image Analysis in Biomedicine MEDICO Retrieval and examination of 2D picture series

22. Daniel Sonntag |22 Conclusions

23. Daniel Sonntag |23 Conclusions »We described a multimodal dialogue shell for QA and focussed on the robust multimodal question understanding task. »The textual interpretation is based on automatically generated textual entailment patterns. »As a result, we can deal with written text input and different surface forms more flexibly according to the derived entailment patterns.

24. Daniel Sonntag |24 Method Comparison »Method 1: Speech Grammars »Speech grammars are verbose »Requires full coverage of expected input »Hard-coded reasoning in rules »Example: »Show me all pictures of X. »What pictures does X have? »Show me all images of X. »Method 2: NLU Grammars »Use of Textual Entailment »NLU grammars are compact »Requires partial coverage of possible input »Example: »Show me all pictures of X. »Entailed utterances: »What pictures does X have? »Show me all images of X. 2 1