1. 1 Answering Questions through Understanding and Analysis (AQUA) Ralph Weischedel and Scott Miller BBN Technologies 3 December 2001

2. 2 Linguistic Processor -Entity Determination -Proposition Recognition -Proposition Matching Question Interpretation User Preferences Catalog of Query Types Analyzed Query History Answer Determination Answer Formulation Analyzed Archive Analysts JFMAMJJA Time Line JFMAMJJA Maps Mary Smith is fluent in Mythian and Legendian Text Tables Vision / Goal Foreign Language Payoff Treebanks and Proposition Banks under development at UPenn Toward a Total System Initial Focus: English text

3. 3 Approach l Implement all core algorithms as l Statistical language modeling, augmented by l Lexical knowledge (e.g., COMLEX, WordNet, gazetteers) l Interpret all source documents and questions as l Entities (which are mentioned in various documents) and l Propositions (which are relations among entities) l Given a proposition over entities, find related propositions l Formulate answer from retrieved entities and propositions l Support user definition of terms via examples

4. 4 Existing Elements for Document Understanding representatives Yugoslav President Slobodan Milosevic received on Thursday the of the Association of Yugoslav Banks, headed by its president Milos, who is also the general director of JugoBanka Milosevic PersonORG Person GPE Name finding via hidden Markov models NPA NP NPA NP PP SBAR PP VP S S WHNP Parsing via lexicalized probabilistic CFGs Co-reference via Bayesian classifier Person:Slobodan Milosevic Position:president Organization: Yugoslavia Person:Milos Milosevic Position:president Organization: Association of Yugoslav Banks Person:Milos Milosevic Position:general director Organization: JugoBanka

5. 5 In-Document Entity Tracking from ACE BAGHDAD, Iraq (AP) _ Iraq's deputy foreign minister attacked U.S. National Security Advisor Sandy Berger Friday, accusing him of ``lies and deception.'' Riyadh al-Qaysi picked his way through Berger's press conference in Washington hours earlier, criticizing the security advisor's assertion that Iraq had been repeatedly in ``material breach'' of U.N. Security Council resolutions. UN Security Council Washington AP US Organization Iraq Berger Baghdad Al-Qaysi Geo-Political EntityPerson

6. 6 To be Investigated l Recognition of all relations (the propositions) in the text l Identification of entities across documents l Recognition of related propositions l Efficient inference (from evolving state of the art)

7. 7 From Trees to Propositions l To be added to UPenn Treebank: l Pronoun co-reference completed by BBN l Predicate-argument structure underway at UPenn The S China VP NP reached have for the VP PP NP of an crew NP PPNP Americanspy theplane NP and States United release agreement Act: reach tense: present perfect Log-Subj: Log-Obj: Act: agree tense: noun Log-Subj: Log-Obj: Act: release tense: noun Log-Subj: Log-Obj: Predicate and arguments Argument Co-reference l To be developed: l A proposition recognition algorithm (both predicate- argument and co-reference)

8. 8 Cross-Document Entity Tracking l Given the mentions of an entity within a document l Connect those mentions to a known data base entity or l Create a new data base entity DB Entity 100347: Names: George Bush, Bush, … Descriptions: President (1989), President (1990), President (1991), President (1992), … Relation pointers: DB Entity 110300: Names: George W. Bush, Bush, Descriptions: President (2001), Governor (2000), the former governor of Texas (2001), … Relation pointers: Highly abridged view of two entities in the entity database Bush announced that Tom Ridge will head a new Department of Homeland Defense. … Highly abridged new document

9. 9 Entity-Relation Example Iraq’s deputy foreign minister attacked U.S. National Security Advisor Sandy Berger Friday, accusing him of “lies and deception.” E1 Type: person Name: Sandy Berger E2 Type: person Name: Riyadh al-Qaysi E3 Type: GPE nation Name: Iraq E4 Type: GPE nation Name: United States DATE Day: Friday Date: 12/27/91 Entities National Security Advisor a1: a2: Deputy Foreign Minister a1: a2: Attack: a1: a2: date: Accuse: a1: a2: a3: date: lie a1: deceive a1: Relations

10. 10 Component Dependency l Foundation of interpretation l Cross-document entity detection and tracking l Proposition recognition l Proposition matching estimates the probability that one proposition may be expressed as another l Example-driven definition uses proposition matching as a basis for finding related examples given user-supplied ones Cross-document Entity Detection & Tracking (CEDT) Proposition Matching (PM) Example-driven Definition (EDD) Proposition Recognition (PR)

11. 11 Proposition Matching [China said Wednesday] it would free the crew of a US spy plane… free China crew US China plane Nationality: US crew reached agreement release The US and China have reached an agreement for the release of the American spy plane crew… plane Nationality: US

12. 12 Example-Driven Definition (today) l Preliminary rapid training strategy successfully demonstrated for description classification l Job titles l Locations l Nuclear materials Trainer Extractor Model Answers Language Input Annotated Examples Annotated Examples Automatically Generalize Patterns & Find Examples Automatically Generalize Patterns & Find Examples Seed Examples

13. 13 Example-Driven Definition (proposed) Define: a coup Example: Nawaz Sharif, who led Pakistan, was ousted October 12 by Pervez Musharraf, Pakistani Army General. Troops loyal to Musharraf surrounded the home of Sharif, and seized the state television network. YES SOMETIMES YES SOMETIMES User Review Person-1 leads country Person-2 ousts Person-1 on date Military group surrounds facility-1 Military group surrounds facility-2 led-3 Agent: Nawaz Sharif Object: Pakistan ousted-1 Agent: Pervez Musharraf Object: Nawaz Sharif Time: October 12 surrounded-1 Agent: Troops Object: home of Sharif seized-4 Agent: Troops Object: state television Propositions (machine-internal) User does not see this! Internal to machine only

14. 14 Conclusion l Goal: Develop comprehensive system l Tested on English text l Portable to other languages l Key to approach l Use statistical language models, augmented by (lexical) knowledge sources and simple formal reasoning l Represent questions and text as (cross-document) entities and relations (propositions) l Find answers (and related questions) via proposition matching l Answers will be drawn from across documents and sources