1. AQUAINT Mid-Year PI Meeting – June 2002 Integrating Robust Semantics, Event Detection, Information Fusion, and Summarization for Multimedia Question Answering Vasileios Hatzivassiloglou, Kathleen R. McKeown Columbia University Dan Jurafsky, Wayne H. Ward, James H. Martin University of Colorado

2. AQUAINT Mid-Year PI Meeting – June 2002 Our focus – Question type (I) Distinguish between questions answerable with –Unique facts (TREC-like) –Facts but not absolute facts; depend on source; perspective; time –Opinions / subjective answers When was Mullah Omar born? vs. Who controls Jalalabad? vs. Will the King’s return be good for Afghanistan?

3. AQUAINT Mid-Year PI Meeting – June 2002 Our focus – Question type (II) Questions with multiple answers Questions with long answers –Definitions –Descriptive answers (different levels of detail) –Lists of related facts

4. AQUAINT Mid-Year PI Meeting – June 2002 Our focus – Multiple sources Integrate answers from multiple sources Use similarities across sources to locate core part of the answer Highlight important differences between sources

5. AQUAINT Mid-Year PI Meeting – June 2002 Our focus – Answer form Answer contains –Core part where sources agree –Differences in perspective –Trends in time Text is not copied verbatim Text generation allows for concise combination of materials from multiple sources

6. AQUAINT Mid-Year PI Meeting – June 2002 Our focus – Q&A Environment Spoken and written questions Specialized language model for accepting questions in realistic, noisy environments Context management system allows for –clarifications –follow-up questions

7. AQUAINT Mid-Year PI Meeting – June 2002 Technology Innovations Specialized speech recognition and dialog management Semantic parsing of questions and source text Event recognition Information fusion

8. AQUAINT Mid-Year PI Meeting – June 2002 Progress in the first six months Revised architecture Software support for integrated system System modules prototyped –Baseline Q&A system –Question hierarchy –Semantic Parser –Event Detection Questions of different types collected

9. AQUAINT Mid-Year PI Meeting – June 2002 Revised Architecture Answer planning Web Question classification Specialized language model Spoken question Speech recognition Recognition feedback Typed question Recognized question Semantic parser MG Google Local collections, TREC Answer extraction and combination Answer strategy selector Query manager Event detection Information fusion Context/dialog manager Short answers Long answers Learned answer plans

10. AQUAINT Mid-Year PI Meeting – June 2002 Integrated System Support Defined APIs for communication between system modules Added ability to communicate via data structures in memory (interprocess calls) Ability to read/write XML Implemented common system foundation and module manager

11. AQUAINT Mid-Year PI Meeting – June 2002 Baseline Q&A System Question is transformed to a search engine query Answers are retrieved from the web or a fixed collection Sentences or paragraphs containing the answer are extracted

12. AQUAINT Mid-Year PI Meeting – June 2002 Prototype System metal company bus

13. AQUAINT Mid-Year PI Meeting – June 2002 Question Analysis Tokenization Part-of-speech assignment Named entity extraction Syntactic parsing Recognition of key and target phrases –How many cities in the US have a stadium?

14. AQUAINT Mid-Year PI Meeting – June 2002 Semantic Parser What effect does a prism have on light? A prism has ____ effect on light [ cause prism] has [ result ____ ] on [ theme light] [ agent Newton] split [ theme white light] [ result into its spectrum of colours] [ cause by beaming it through a prism] Implemented domain-independent version of semantic parser (40% recall, 60% precision)

15. AQUAINT Mid-Year PI Meeting – June 2002 Question Classification By question type (detailed hierarchy has been built) Distinguish between descriptive and non- descriptive answers –Who is the President of the United States? –How do I become President?

16. AQUAINT Mid-Year PI Meeting – June 2002 Question Mining Collection of questions and answers –From FAQs (mostly descriptions) –From trivia sites (mostly non-descriptive facts) Implemented classifier between these types –Features: words, length, part-of-speech

17. AQUAINT Mid-Year PI Meeting – June 2002 Answer Generation Bottom-up, from the data –Clustering organizes similar answers together –Fusion matches common parts –Generation combines answer fragments in a concise response Top-down, using question-specific plans –Appropriate for lists of facts

18. AQUAINT Mid-Year PI Meeting – June 2002 Event Recognition Atomic events vs. TDT events vs. topics Events as a basis for segmenting documents and classifying document fragments as matching a question Event algebra will allow –grouping sub-events –linking related events –detecting updates

19. AQUAINT Mid-Year PI Meeting – June 2002 Detecting an event Events can be detected on –participants (named entities, semantic roles) –time –location –limited verbal features Collected data and human annotations Implemented event detection system (80% accuracy)

20. AQUAINT Mid-Year PI Meeting – June 2002 Goals for the first six months Initial FrameNet parser (limited coverage) Identification of participants, time, location Identifying paraphrases from comparable news reports on the same event Adapting information fusion from summarization to question-answering Building prototype Q&A system

21. AQUAINT Mid-Year PI Meeting – June 2002 Goals for the first six months Initial FrameNet parser (limited coverage) Identification of participants, time, location Identifying paraphrases from comparable news reports on the same event Adapting information fusion from summarization to question-answering Building prototype Q&A system

22. AQUAINT Mid-Year PI Meeting – June 2002 Progress on Other Items Initial syntactic question paraphrasing Hierarchy of question types Tools for searching a specific collection Event recognition prototype built Started analysis of data sources for questions with multiple or long answers

23. AQUAINT Mid-Year PI Meeting – June 2002 Goals for the next six months Full syntactic and lexical question paraphrasing Classifier for choosing appropriate question type Integrate semantic labels into question analysis Answer strategy selector Initial context management module Participation in TREC Data collection and classification of questions with multiple or long answers