Presentation is loading. Please wait.

Presentation is loading. Please wait.

Structured Probabilistic Inference in an Embodied Construction Grammar and Jerome Feldman International Computer Science Institute U. California at Berkeley.

Similar presentations


Presentation on theme: "Structured Probabilistic Inference in an Embodied Construction Grammar and Jerome Feldman International Computer Science Institute U. California at Berkeley."— Presentation transcript:

1

2 Structured Probabilistic Inference in an Embodied Construction Grammar and Jerome Feldman International Computer Science Institute U. California at Berkeley Berkeley, CA jfeldman@icsi.berkeley.edu Srini Narayanan International Computer Science Institute Berkeley, CA snarayan@icsi.berkeley.edu

3 Language, Learning and Neural Modeling Scientific Goal Understand how people learn and use language Practical Goal Build systems that analyze and produce language Approach Embodied linguistic theories with advanced biologically-based computational methods

4 State of the Art Limited Commercial Speech Applications transcription, simple response systems Statistical NLP for Restricted Tasks tagging, parsing, information retrieval Template-based Understanding programs expensive, brittle, inflexible, unnatural Essentially no NLU in QA Systems

5 Hypothesis NLU is essential to large, open domain QA. One line dismissals (lack of world knowledge) are out of date (by at least 5 yrs.) Substantial Progress in Enabling Technologies –Knowledge Representation/Inference Techniques Active Knowledge Dealing With Uncertainty. Simulation Semantics –Scaling Up CYC, Wordnet, Term-bases FrameNet, Semantic Web, MetaNet –Extraction of Semantic Relations Empirical Linguistic The goal of NLU can be realized

6 Hypothesis NLU is essential to large, open domain QA. One line dismissals (lack of world knowledge) are out of date (by at least 5 yrs.) Substantial Progress in Enabling Technologies –Knowledge Representation/Inference Techniques Active Knowledge Dealing With Uncertainty. Simulation Semantics –Scaling Up CYC, Wordnet, Term-bases FrameNet, Semantic Web, MetaNet –Extraction of Semantic Relations Empirical Linguistic The goal of NLU can be realized, perhaps! Anyway, it is time to try again.

7 Complex questions and semantic information (SH, UTD) Complex questions are not characterized only by a question class (e.g. manner questions) –Example: How can a biological weapons program be detected ? –Associated with the pattern “How can X be detected?” –And the topic X = “biological weapons program” Processing complex questions is also based on access to the semantics of the question topic –The topic is modeled by a set of discriminating relations, e.g. Develop(program); Produce(biological weapons); Acquire(biological weapons) or stockpile(biological weapons) –Such relations are extracted from topic-relevant texts

8 The need for Semantic Inference in QA Some questions are complex! Example (from UTD CNS QA database): –What is the evidence that IRAQ has WMD? –Answer: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors.

9 ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium Answer Structure

10 ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: possess(Iraq, fuel stock(purpose: power launchers)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium possess(Iraq, delivery systems(type : scud missiles; launchers; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium Answer Structure (continued) hide(Iraq, Seeker: UN Inspectors; Hidden: CBW stockpiles & guided missiles) Justification: DETECTION Schema Inspection status: Past; Likelihood: Medium

11 Semantic inference for Q/A The problem of classifying questions –E.g. “manner questions”: –Example “How did Hitler die?” The problem of recognizing answer types/structures –Should “manner of death” by considered an answer type? –What other manner of event/action should be considered as answer types? The problem of extracting/justifying/ generating answers to complex questions –Should we learn to extract “manner” relations? –What other types of relations should we consider? –Is relation recognition sufficient for answering complex questions? Is it necessary?

12 Outline Part I. Introduction: Semantic Inference for QA Part II. Cognitive Primitives –Embodiment in Language –Conceptual Relations Cross-linguistic Relations –Image Schemas –Active Schemas: X-schemas –Domain Specific Knowledge Cultural Frames and FrameNet Ontologies

13 Embodiment Of all of these fields, the learning of languages would be the most impressive, since it is the most human of these activities. This field, however, seems to depend rather too much on the sense organs and locomotion to be feasible. Alan Turing (Intelligent Machines,1948)

14 NTL Manifesto Basic concepts and words derive their meaning from embodied experience. Abstract and Theoretical concepts derive their meaning from metaphorical maps to more basic embodied concepts. Structured Connectionist Models can capture both of these processes nicely.

15 General and Domain Knowledge Conceptual Knowledge and Inference –Embodied –Language and Domain Independent –Powerful General Inferences –Ubiquitous in Language Domain Specific Frames and Ontologies –Framenet Metaphor links domain specific to general –E.g., France slipped into recession.

16 Conceptual schemas Much is known about conceptual schemas –Image Schemas –X-schemas –Frames –Constructions Recently we have developed a scalable formalism for representing conceptual schemas We are testing and refining both the content and formalism based on other languages.

17 What are Image schemas? –Regularities in our perceptual, motor and cognitive systems –Structure our experiences and interactions with the world. –May be grounded in a specific cognitive system, but are not situation-specific in their application (can apply to many domains of experience)

18 Basis of Image schemas Perceptual systems Motor routines Social Cognition Image Schema properties depend on –Neural circuits –Interactions with the world

19 semantic schema Container roles: interior exterior portal boundary Representing image schemas Interior Exterior Boundary Portal Source Path Goal Trajector These are abstractions over sensorimotor experiences. semantic schema Source-Path-Goal roles: source path goal trajector

20 Schema Formalism SCHEMA SUBCASE OF EVOKES AS ROLES : CONSTRAINTS :: :: |

21 A Simple Example SCHEMA hypotenuse SUBCASE OF line-segment EVOKES right-triangle AS rt ROLES Comment inherited from line-segment CONSTRAINTS SELF rt.long-side

22 Source-Path-Goal SCHEMA: spg ROLES: source: Place path: Directed Curve goal: Place trajector: Entity

23 Translational Motion SCHEMA translational motion SUBCASE OF motion EVOKES spg AS s ROLES mover s.trajector source s.source goal s.goal CONSTRAINTS before:: mover.location source after:: mover.location goal

24 Extending Inferential Capabilities Given the formalization of the conceptual schemas –How to use them for inferencing? Earlier pilot systems –Used metaphor and Bayesian belief networks –Successfully construed certain inferences –But don’t scale to large open domains New approach –Probabilistic relational models –Support an open ontology

25 Frames Frames are conceptual structures that may be culture specific Words evoke frames –The word “talk” evokes the Communication frame –The word buy (sell, pay) evoke the Commercial Transaction (CT) frame. –The words journey, set out, schedule, reach etc. evoke the Journey frame. Frames have roles and constraints like schemas. –CT has roles vendor, goods, money, customer. Words bind to frames by specifying binding patterns –Buyer binds to Customer, Vendor binds to Seller.

26 The FrameNet Project C Fillmore PI (ICSI) Co-PI’s: S Narayanan (ICSI, SRI) D Jurafsky (U Colorado) J M Gawron (San Diego State U) Staff: C Baker Project Manager B Cronin Programmer C Wooters Database Designer

27 Frames and Understanding Hypothesis: People understand things by performing mental operations on what they already know. Such knowledge is describable in terms of information packets called frames.

28 FrameNet in the Larger Context The long-term goal is to reason about the world in a way that humans understand and agree with. Such a system requires a knowledge representation that includes the level of frames. FrameNet can provide such knowledge for a number of domains. FrameNet representations complement ontologies and lexicons.

29 The core work of FrameNet 1.characterize frames 2.find words that fit the frames 3.develop descriptive terminology 4.extract sample sentences 5.annotate selected examples 6.derive "valence" descriptions

30 The Core Data The basic data on which FrameNet descriptions are based take the form of a collection of annotated sentences, each coded for the combinatorial properties of one word in it. The annotation is done manually, but several steps are computer- assisted.

31 Types of Words / Frames oevents oartifacts, built objects onatural kinds, parts and aggregates oterrain features oinstitutions, belief systems, practices ospace, time, location, motion oetc.

32 FrameNet Product For every target word, describe the frames or conceptual structures which underlie them, and annotate example sentences that cover the ways in which information from the associated frames are expressed in these sentences.

33 Complex Frames With Criminal_process we have, for example, – sub-frame relations (one frame is a component of a larger more abstract frame) and –temporal relations (one process precedes another)

34 Frame-to-Frame Relations: Crime_scenario

35 Frame Relations Frames are linked to each other by various types of relations, three of which are: –Inheritance: all FEs of parent frame have corresponding FEs in child (not necessarily with the same name) –Using: like inheritance, but not all FEs are bound –Subframe: used for sub-events of a complex event; may be temporally ordered (or not)

36 Applying Frame Structures to QA Parsing Questions Parsing Answers Result: exact answer= “approximately 7 kg of HEU” Q: What kind of materials were stolen from the Russian navy? FS(Q): What [GOODS: kind of nuclear materials] were [Target-Predicate:stolen] [VICTIM: from the Russian Navy]? A(Q): Russia’s Pacific Fleet has also fallen prey to nuclear theft; in 1/96, approximately 7 kg of HEU was reportedly stolen from a naval base in Sovetskaya Gavan. FS(A(Q)): [VICTIM(P1): Russia’s Pacific Fleet] has also fallen prey to [Goods(P1): nuclear ] [Target-Predicate(P1): theft]; in 1/96, [GOODS(P2): approximately 7 kg of HEU] was reportedly [Target-Predicate (P2): stolen] [VICTIM (P2): from a naval base] [SOURCE(P2): in Sovetskawa Gavan]

37 Language understanding via simulation Hypothesis: Linguistic input is converted into a mental simulation based on bodily grounded structures –Semantic schemas, including image schemas (Johnson 1987) and executing schemas (Bailey, Narayanan 1997) are abstractions over neurally grounded perceptual and motor representations –Linguistic units make reference to these structures in their semantic pole –Analysis produces a simulation specification linking these structures and providing parameters for a simulation engine Model requires: – schema representations (image, motor, frame, social, etc.) – lexical and phrasal construction representations that invoke those schemas and other cultural frames (FrameNet+ frames)

38 Simulation-based language understanding “Harry walked to the cafe.” SchemaTrajectorGoal walkHarrycafe Analysis Process Simulation Specification Utterance Simulation Cafe Constructions General Knowledge Belief State

39 Embodied Construction Grammar (Bergen & Chang 2002) Assumptions from Construction Grammar –Constructions are form-meaning pairs (Lakoff 1987, Goldberg 1995) –Constructions vary in degree of specificity and level of description (morphological, lexical, phrasal, clausal) Constructions evoke and bind semantic schemas Additional influences –Cognitive Grammar (Langacker 1987) –Frame Semantics (Fillmore 1982) –Structured Connectionism (Feldman 1987)

40 Simulation specification The analysis process produces a simulation specification that includes image-schematic, motor control and conceptual structures provides parameters for a mental simulation

41 Simulation Semantics BASIC ASSUMPTION: SAME REPRESENTATION FOR PLANNING AND SIMULATIVE INFERENCE – Evidence for common mechanisms for recognition and action (mirror neurons) in the F5 area (Rizzolatti et al (1996), Gallese 96, Buccino 2002, Tettamanti 2004) and from motor imagery (Jeannerod 1996) IMPLEMENTATION: –x-schemas affect each other by enabling, disabling or modifying execution trajectories. Whenever the CONTROLLER schema makes a transition it may set, get, or modify state leading to triggering or modification of other x-schemas. State is completely distributed (a graph marking) over the network. RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!

42 Outline Part III. Knowledge representation and inference –KR requirements Dynamic Context and the structure of Events Uncertainty in knowledge sources Representing Metaphors and Maps –Active Knowledge and Event Structure X-schemas A computational model of Aspect –Bayesian Models of Uncertainty Graphical Models Dynamic Bayes Nets for Modeling State –A Computational Model of Metaphor Interpretation DEMO

43 ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium Answer Structure

44 ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium Answer Structure

45 ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium Answer Structure Temporal Reference/Grounding

46 ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: possess(Iraq, fuel stock(purpose: power launchers)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium possess(Iraq, delivery systems(type : scud missiles; launchers; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium Answer Structure (continued) hide(Iraq, Seeker: UN Inspectors; Hidden: CBW stockpiles & guided missiles) Justification: DETECTION Schema Inspection status: Past; Likelihood: Medium Present Progressive Perfect Present Progressive Continuing

47 ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium Answer Structure Uncertainty and Belief

48 ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium Answer Structure Uncertainty and Belief Multiple partly reliable sources

49 ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium Answer Structure Event Structure Metaphor

50 Temporal relations in QA Results of the workshop are accessible from http://www.cs.brandeis.edu/~jamesp/arda/time/documentation/TimeML-use-in-qa- v1.0.pdf http://www.cs.brandeis.edu/~jamesp/arda/time/documentation/TimeML-use-in-qa- v1.0.pdf A set of questions that require the extraction of temporal relations was created (TimeML question corpus) –E.g.: “When did the war between Iran and Iraq end?” “Who was Secretary of Defense during the Golf War?” A number of features of these questions were identified and annotated –E.g.: Number of TEMPEX relations in the question Volatility of the question (how often does the answer change) Reference to repetitive events Number of events mentioned in the question

51 Event Structure for semantically based QA Reasoning about dynamics –Complex event structure Multiple stages, interruptions, resources, framing –Evolving events Conditional events, presuppositions. –Nested temporal and aspectual references Past, future event references –Metaphoric references Use of motion domain to describe complex events. Reasoning with Uncertainty –Combining evidence from multiple, unreliable sources –Non-monotonic inference Retracting previous assertions Conditioning on partial evidence –Linguistic Ambiguity –Figurative inference

52 Relevant Previous Work Event Structure Aspect (VDT, TimeML), Situation Calculus (Steedman), Frame Semantics (Fillmore), Cognitive Linguistics (Langacker, Talmy, Lakoff, Sweetser), Metaphor and Aspect (Narayanan) Reasoning about Uncertainty Bayes Nets (Pearl), Probabilistic Relational Models (Pfeffer, Koller), Graphical Models (Jordan), First Order Probabilistic Inference (Poole, Braz et al) Reasoning about Dynamics Dynamic Bayes Nets (Murphy, Friedman), Distributed Systems (Alur, Meseguer), Control Theory (Ramadge and Wonham), Causality (Pearl)

53 Active representations Many inferences about actions derive from what we know about executing them Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions Walking: bound to a specific walker with a direction or goal consumes resources (e.g., energy) may have termination condition (e.g., walker at goal ) ongoing, iterative action walker =Harry goal =home energy walker at goal

54 An Active Model of Events Computationally, actions and events are coded in active representations called x- schemas which are extensions to Stochastic Petri nets. x-schemas are fine-grained action and event representations that can be used for monitoring and control as well as for inference.

55 Model Review: Stochastic Nets 3 1 2 Basic Mechanism [1] Precondition arc Resource arc Inhibition arc [1] Firing function -- conjunctive -- logistic -- exponential family

56 3 1 2 Firing Semantics Model Review

57 1 1 1 1 2 Result of Firing Model Review

58 X-Schema Extensions to Petri Nets Parameterization –x-schemas take parameter values (speed, force) Walk(speed = slow, dest = store1) Dynamic Binding –X-schemas allow run-time binding to different objects/entities Grasp(cup1), push(cart1) Hierarchical control and durative transitions –Walk is composed of steps which are composed of stance and swing phases Stochasticity and Inhibition –Uncertainties in world evolution and in action selection

59 Event Structure in Language Fine-grained Rich Notion of Contingency Relationships. –Phenomena: Aspect, Tense, Force-dynamics, Modals, Counterfactuals Event Structure Metaphor: –Phenomena: Abstract Actions are conceptualized in Motion and Manipulation terms. –Schematic Inferences are preserved.

60 Aspect Aspect is the name given to the ways languages describe the structure of events using a variety of lexical and grammatical devices. –Viewpoints is walking, walk –Phases of events Starting to walk, walking, finish walking –Inherent Aspect run vs cough vs. rub –Composition with Temporal modifiers, tense.. Noun Phrases (count vs. mass) etc..

61 Phases, Viewpoints, and Aspects John is walking to the store. John is about to walk to the store. John walked to the store. John started walking to the store. John is starting to walk to the store. John has walked to the store. John has started to walk to the store. John is about to start walking to the store. John resumed walking to the store. John has been walking to the store. John has finished walking to the store. John almost walked to the store.

62 Aspectual Information in Questions The US continues to be concerned about Russian arms sales to Syria. Has the US taken any steps to try to stop the sale of Russian arms to Syria? The US is concerned about Russian arms sales to Syria. Has the US taken any steps to try to detect the sale of Russian arms to Syria? The US is starting to be concerned about Russian arms. Is the US ready to take steps to try to establish the sale of Russian arms to Syria?

63 Aspectual Types

64

65

66

67

68

69 Aspects of (Climb) Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate Energy Ready Standing On top Hold Find hold Pull(self) Stabilize BINDINGS

70 About to + (Climb) (Prospective) Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate Energy Ready Standing On top Hold Find hold Pull(self) Stabilize BINDINGS

71 Be + (Climb)-ING (Progressive) Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate Energy Ready Standing On top Hold Find hold Pull(self) Stabilize BINDINGS

72 Have + (Climb)-ed (Perfect) Energy Ready Standing On top Hold Find hold Pull(self) Stabilize Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate BINDINGS

73 Phasal Aspect Maps to the Controller Ready DoneStartProcessFinish Suspend Cancel interruptresume Iterate Inceptive (start, begin) Iterative (repeat) Completive (finish, end) Resumptive(resume)

74 Embedding: About to start (X) Ready DoneStartProcessFinish Suspend interrupt resume R DSPF S C i r X-Schema for X with bindings

75 Embedding: Has Started (to X) Ready DoneStartProcessFinish Suspend interrupt resume R DSPF S C i r X-Schema for X with bindings

76 Embedding: The end of the beginning Ready DoneStartProcessFinish Suspend interrupt resume R DSPF S C i r X-Schema for X with bindings

77 Embedding: The beginning of the end Ongoing FinishDone R DPF S C i r X-Schema for X with bindings S

78 Inherent Aspect Much richer than traditional Linguistic Characterizations (VDT (durative/atomic, telic/atelic)) Action patterns –one-shot, repeated, periodic, punctual –decomposition: concurrent, alternatives, sequential Goal based schema enabling/disabling Generic control features; –interruption, suspension, resumption Resource usage

79

80 Other Transitions in the Controller may be coded Lexical items may code interrupts –Stumble is an interrupt to an ongoing walk A combination of grammatical and aktionsart may code of the controller phases –Ready to walk : Prospective –Resuming his run: Resumptive –Has been running: Embedded progressive –About to Finish the painting: Embedded Completive. –Canceling the meeting vs. Aborting the meeting.

81 Simulation Semantics BASIC ASSUMPTION: SAME REPRESENTATION FOR PLANNING AND SIMULATIVE INFERENCE – Evidence for common mechanisms for recognition and action (mirror neurons) in the F5 area (Rizzolatti et al (1996), Gallese 96, Buccino 2002, Tettamanti 2004) and from motor imagery (Jeannerod 1996) IMPLEMENTATION: –x-schemas affect each other by enabling, disabling or modifying execution trajectories. Whenever the CONTROLLER schema makes a transition it may set, get, or modify state leading to triggering or modification of other x-schemas. State is completely distributed (a graph marking) over the network. RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!

82

83

84

85

86

87

88 A Precise Notion of Contingency Relations Activation: Executing one schema causes the enabling, start or continued execution of another schema. Concurrent and sequential activation. Inhibition: Inhibitory links prevent execution of the inhibited x-schema by activating an inhibitory arc. The model distinguishes between concurrent and sequential inhibition, mutual inhibition and aperiodicity. Modification: The modifying x-schema results in control transition of the modified xschema. The execution of the modifying x-schema could result in the interruption, termination, resumption of the modified x-schema.

89 Summary of Aspect Results Controller mediates between linguistic markings and individual event/verbal x-schemas (Cogsci99, Coling2002) Captures regular event structure; inspired by biological control theory Flexible: specific events may require only a subset of controller; interaction of underlying x-schemas, linguistic markers and hierarchical abstraction/ decomposition of controller accounts for wide range of aspectual phenomena. Important aspectual distinctions, both traditional and novel, can be precisely specified in terms of the interaction of x-schemas with the controller (CogSci97, CogSci 98, AAAI99, IJCAI99, CogSci04): stative/dynamic, durative/punctual: natural in x-schemas telic processes: depletion of resources continuous processes: consumption of resources temporary/effortful states; habituals dynamic interactions with tense, nominals, temporal modifiers incorporation of world knowledge, pragmatics

90 Event Structure in Language Fine-grained Rich Notion of Contingency Relationships. –Phenomena: Aspect, Tense, Force-dynamics, Modals, Counterfactuals Event Structure Metaphor: –Phenomena: Abstract Actions are conceptualized in Motion and Manipulation terms. –Schematic Inferences are preserved.

91 We use metaphors everyday The council attacked every weak point of his proposal. I don't know how to put my thoughts into words. My summer plans are still up in the air. France slipped into a recession. Something smells fishy, but I can't quite put my finger on it. The economy hasn’t turned the corner, it has just made a U-turn and reversed course (Kerry, 8/10/04)

92 What is the basis for metaphors? metaphor is understanding one thing in terms of another specifically, we reason about abstract concepts through our sensory-motor experience. that means we have: –correlation –inference

93 Metaphors, defined Formally, metaphors are mappings from a source domain to a target domain both the source and target domains are structured by schemas and frames Take a simple example: I've been feeling quite depressed of late. ( Happy is Up; Sad is Down )

94 SCHEMA Happiness SUBCASE OF Emotion ROLES Degree SCHEMA Verticality SUBCASE OF Orientation ROLES Scale MAP HappyIsUpSadIsDown map-type <- METAPHOR tgt src PAIRS

95 Event Structure Metaphor Maps motion and manipulation concepts to abstract actions. States are Locations Changes are Movements Causes are Forces –Force Dynamic patterns of causation Actions are Self-propelled Movements –Speed, step size are parameters that are mapped. Means are Paths –crossroads Difficulties are Impediments to Motion Long-term, Purposeful Activities are Journeys –Set out, back on track…

96 Metaphorical Inference There is a conceptual metaphor, Understanding Is Grasping, according to which one can grasp ideas. One can begin to grasp an idea, but not quite get a hold of it. If you fail to grasp an idea, it can go right by you — or over your head! If you grasp it, you can turn it over in your mind. You can’t hold onto an idea before having grasped it. In short, reasoning patterns about physical grasping can be mapped by conceptual metaphor onto abstract reasoning patterns. Metaphors project the products of simulations to the target (abstract domain)!

97 Event Structure for semantically based QA Reasoning about dynamics –Complex event structure Multiple stages, interruptions, resources, framing –Evolving events Conditional events, presuppositions. –Nested temporal and aspectual references Past, future event references –Metaphoric references Use of motion domain to describe complex events. Reasoning with Uncertainty –Combining evidence from multiple, unreliable sources –Non-monotonic inference Retracting previous assertions Conditioning on partial evidence –Linguistic Ambiguity –Figurative inference

98 Uncertainty and domain representation Factorized Representation of Domain uses Dynamic Belief Nets (DBN’s) –Probabilistic Semantics –Structured Representation

99 Qualitative part: Directed acyclic graph (DAG) Nodes - random vars. Edges - direct influence Quantitative part: Set of conditional probability distributions 0.90.1 e b e 0.20.8 0.01 0.99 0.90.1 be b b e BE P(A | E,B) Family of Alarm Earthquake Radio Burglary Alarm Call Compact representation of joint probability distributions via conditional independence Together: Define a unique distribution in a factored form What is a Bayes (belief) net? Figure from N. Friedman

100 What is a Bayes net? Earthquake Radio Burglary Alarm Call C || R,B,E | A A node is conditionally independent of its ancestors given its parents, e.g. Hence From 2 5 – 1 = 31 parameters to 1+1+2+4+2=10

101 Why are Bayes nets useful? - Graph structure supports - Modular representation of knowledge - Local, distributed algorithms for inference and learning - Intuitive (possibly causal) interpretation - Factored representation may have exponentially fewer parameters than full joint P(X 1,…,X n ) => - lower sample complexity (less data for learning) - lower time complexity (less time for inference)

102 What can Bayes nets be used for? Posterior probabilities –Probability of any event given any evidence Most likely explanation –Scenario that explains evidence Rational decision making –Maximize expected utility –Value of Information Effect of intervention –Causal analysis Earthquake Radio Burglary Alarm Call Radio Call Figure from N. Friedman Explaining away effect

103 Dynamic Bayes Nets Dynamic Bayesian Networks (D(T)BNs) are an extension of Bayesian networks for modeling dynamic systems. –state at time t is represented by a set of random variables. – The state at time t is dependent on the states at previous time steps. –first-order Markovian, and thus we need to represent the transition distribution P(Z t+1 | Z t ). This can be done using a two-time-slice Bayesian network fragment (2-TBN) B t+1, – variables from Z t+1 whose parents are variables from Z t and/or Z t+1, and variables from Z t without any parents. –Typically, we also assume that the process is stationary,

104 Economic State [recession,nogrowth,lowgrowth,higrowth] Goal Policy Outcome Difficulty A Simple DBN for the Economics Domain [Liberalization, Protectionism] [Free Trade, Protection ] [Success, failure] [present, absent] T0 T1

105 Probabilistic inference –Filtering P(X_t | o_1…t,X_1…t) Update the state based on the observation sequence and state set –MAP Estimation Argmax h1…hn P(X_t | o_1…t, X_1…t) Return the best assignment of values to the hypothesis variables given the observation and states –Smoothing P(X_t-k | o_1…t, X_1…t) modify assumptions about previous states, given observation sequence and state set –Projection/Prediction/Reachability P(X_t+k | o_1..t, X_1..t)

106 Answer Type to Inference Method ANSWER TYPEINFERENCEDESCRIPTION Justify (Proposition)MAPProposition is part of the MAP Ability (Agent, Act)Filtering; Smoothing Past/Current Action enabled given current state Prediction (State)P;R’ MAPPropogate current information and estimate best new state Hypothetical (Condition)S, R_ISmooth intervene and compute state

107 Logical Action Theories Connection to ARD (or other Action Languages): –The representation can be used to encode a causal model for a domain description D (in the Syntax of ARD) in that it satisfies all the causal laws in D. Furthermore, a value proposition of the form C after A is entailed by D iff all the terms in C are in Si; the state that results after running the projection algorithm on the action set A. (IJCAI 99) Executing representation, –frame axioms are encoded in the topology of the network and transition firing rules respect them. Planning as backward reachability or computing downward closure (IJCAI 99, WWW2002) Links to linear logic. Perhaps a model of stochastic linear logic? (joint work with Jose Meseguer).

108 Outline Part III. Knowledge representation and inference –KR requirements Dynamic Context and the structure of Events Uncertainty in knowledge sources Representing Metaphors and Maps –Active Knowledge and Event Structure X-schemas A computational model of Aspect –Bayesian Models of Uncertainty Graphical Models Dynamic Bayes Nets for Modeling State –A Computational Model of Metaphor Interpretation DEMO

109 Task: Interpret simple discourse fragments/ blurbs France fell into recession. Pulled out by Germany US Economy on the verge of falling back into recession after moving forward on an anemic recovery. Indian Government stumbling in implementing Liberalization plan. Moving forward on all fronts, we are going to be ongoing and relentless as we tighten the net of justice. The Government is taking bold new steps. We are loosening the stranglehold on business, slashing tariffs and removing obstacles to international trade.

110 Basic Result An implemented computational model that is –Neurally plausible (Feldman and Narayanan 2004) –Establishes that motion, manipulation and spatial concepts are used to convey important and subtle information about abstract domains such as International Economics. In 1991, India set out on a path of Liberalization. After making rapid strides in the first few years, the Government policy hit a first of a series of roadblocks in 1995. By 1998, the new BJP Government had reoriented the Government’s policy..

111 I/O as Feature Structures Indian Government stumbling in implementing liberalization plan

112 Basic Primitives An fine-grained executing model of action and events (X-schemas). A factorized representation of probabilistic domain knowledge (DBN’s) A controller X-schema that models inter- event relations and forms the basis of inference through simulation. A model of metaphor maps that project bindings from embodied to application domains.

113

114 KARMA DEMO SOURCE DOMAINS: MOTION, HEALTH TARGET DOMAINS: INTERNATONAL ECONOMICS METAPHOR MAPS: EVENT STRUCTURE METAPHOR

115 Results Model was implemented and tested on discourse fragments from a database of 50 newspaper stories in international economics from standard sources such as WSJ, NYT, and the Economist. Results show that motion terms are often the most effective method to provide the following types of information about abstract plans and actions. – Information about uncertain events and dynamic changes in goals and resources. (sluggish, fall, off-track, no steam) –Information about evaluations of policies and economic actors and communicative intent (strangle-hold, bleed). –Communicating complex, context-sensitive and dynamic economic scenarios (stumble, slide, slippery slope). –Commincating complex event structure and aspectual information (on the verge of, sidestep, giant leap, small steps, ready, set out, back on track). ALL THESE BINDINGS RESULT FROM REFLEX, AUTOMATIC INFERENCES PROVIDED BY X-SCHEMA BASED INFERENCES.

116 Scaling Up –Scaling Up Language Embodied Construction Grammar FrameNet for Scenarios –Scaling Up Inference Coordinated Probabilistic Relational Model (CPRM) An Application to QA inference –Scaling Up Ontological Knowledge Semantic Web –OWL and OWL-S –An Event Ontology in OWL

117 Outline Part IV. Scaling Up –Scaling Up Language Embodied Construction Grammar FrameNet for Scenarios –Scaling Up Ontological Knowledge Semantic Web –OWL and OWL-S –An Event Ontology in OWL –Scaling Up Inference Coordinated Probabilistic Relational Model (CPRM)

118 Embodied Construction Grammar Embodied representations –active perceptual and motor schemas (image schemas, x-schemas, frames, etc.) –situational and discourse context Construction Grammar –Linguistic units relate form and meaning/function. –Both constituency and (lexical) dependencies allowed. Constraint-based –based on feature unification (as in LFG, HPSG) –Diverse factors can flexibly interact.

119 Embodied Construction Grammar provides formal tools for linguistic description and analysis motivated largely by cognitive/functional concerns. A shared theory and formalism for different cognitive mechanisms –Constructions, metaphor, mental spaces, etc. Precise specifications of structures/processes involved in language understanding Bridge to detailed simulative inference using embodied representations

120 “Harry walked into the cafe.” Phonology Semantics Pragmatics Morphology Syntax Phonetics

121 “Harry walked into the cafe.” Phonology Semantics Pragmatics Morphology Syntax Phonetics UTTERANCEUTTERANCE

122 ECG Structures Schemas –image schemas, force-dynamic schemas, executing schemas, frames… Constructions –lexical, grammatical, morphological, gestural… Maps –metaphor, metonymy, mental space maps… Spaces –discourse, hypothetical, counterfactual…

123 Image schemas Trajector / Landmark (asymmetric) –The bike is near the house – ? The house is near the bike Boundary / Bounded Region –a bounded region has a closed boundary Topological Relations –Separation, Contact, Overlap, Inclusion, Surround Orientation –Vertical (up/down), Horizontal (left/right, front/back) –Absolute (E, S, W, N) LM TR bounded region boundary

124 schema Container roles interior exterior portal boundary Embodied schemas Interior Exterior Boundary Portal Source Path Goal Trajector These are abstractions over sensorimotor experiences. schema Source-Path-Goal roles source path goal trajector schema name role name

125 Embodied constructions construction H ARRY form : /hEriy/ meaning : Harry construction C AFE form : /k h aefej/ meaning : Cafe Harry CAFE cafe ECG Notation FormMeaning Constructions have form and meaning poles that are subject to type constraints.

126 The meaning pole may evoke schemas (e.g., image schemas) with a local alias. The meaning pole may include constraints on the schemas (e.g., identification constraints  ). construction T O form self f.phon  /t h u w / meaning evokes Trajector-Landmark as tl Source-Path-Goal as spg constraints: tl.trajector  spg.trajector tl.landmark  spg.goal construction T O form self f.phon  /t h u w / meaning evokes Trajector-Landmark as tl Source-Path-Goal as spg constraints: tl.trajector  spg.trajector tl.landmark  spg.goal Representing constructions: T O local alias identification constraint

127 T O vs. I NTO : I NTO adds a Container schema and appropriate bindings. The I NTO construction construction I NTO form self f.phon  / I nt h u w / meaning evokes Trajector-Landmark as tl Source-Path-Goal as spg Container as cont constraints: tl.trajector  spg.trajector tl.landmark  cont cont.interior  spg.goal cont.exterior  spg.source

128 construction S PATIAL- P HRASE constructional constituents sp : Trajector-Landmark lm : Thing form sp f before lm f meaning sp m.landmark  lm m Constructions with constituents : The SPATIAL-PHRASE construction Constructions may also specify constructional constituents and impose form and meaning constraints on them: –order constraints –identification constraints order constraint local alias identification constraint

129 An argument structure construction construction D IRECTED -M OTION subcase of Pred-Expr constructional constituents a : Ref-Exp m : Pred-Exp p : Spatial-Phrase form a f before m f m f before p f meaning evokes Directed-Motion as dm self m.scene  dm dm.agent  a m dm.motion  m m dm.path  p m schema Directed-Motion roles agent : Entity motion : Motion path : SPG

130 Simulation-based language understanding Analysis Process Semantic Specification “Harry walked into the cafe.” Utterance CAFE Simulation Belief State General Knowledge Constructions construction W ALKED form self f.phon  [wakt] meaning : Walk-Action constraints self m.time before Context.speech-time self m..aspect  encapsulated

131 The ECG Analyzer Goes bottom up, generating a semantic interpretation as well as syntactic constituency Uses a chart to store the constituent matches Instead of using FSMs, each construction is turned into a construction recognizer

132 Using a Chart A chart is a data structure that tracks the constituents found during an analysis. –Note that the chart is storing semantic information as well as grammatical. After analysis, the chart is filled with all the semantic chunks (constituents) that were found.

133 Construction Recognizer Each construction is turned into a parameterized procedure. The procedure checks form and semantic constraints simultaneously. Constructions = active knowledge

134 ECG - Clausal Example Construction Active-Caused-Motion constituents subj : RefExp[Agent] verb : Verb[Force-Application] DO : RefExp path : PP[SPG] form subj before verb verb before DO verb before path meaning: caused-motion-scene self m.agent subj m self m.action verb m self m.patient DO m self m.path path m

135 Chunking 0 1 2 3 4 5 6 7 8 9 the woman in the lab coat thought you were sleeping L0 D N P D N N V-tns Pron Aux V-ing L1 ____NP P_______NP VP NP ______VP L2 ____NP _________PP VP NP ______VP L3 ________________________S_____________S After Abney, 1996.

136 Construction Recognizers You want to put a cloth on your hand ? NP Form Meaning “you” [Addressee] Form Meaning D,N [Cloth num:sg] Form Meaning PP$,N [Hand num:sg poss:addr] Like Abney:Unlike Abney: One recognizer per rule Bottom up and level-based Check form and semantics More powerful/slower than FSMs

137 Chunk Chart Interface between chunking and structure merging Each edge is linked to its corresponding semantics. You want to put a cloth on your hand ?

138 Combining Partial Parses Prefer an analysis that spans the input utterance with the minimum number of chunks. When no spanning analysis exists, however, we still have a chart full of semantic chunks. The system tries to build a coherent analysis out of these semantics chunks. This is where structure merging comes in.

139 Structure Merging Closely related to abductive inferential mechanisms like Hobbs’ Unify compatible structures (find fillers for frame roles) Intuition: Unify structures that would have been co-indexed had the missing construction been defined. There are many possible ways to merge structures. In fact, there are an exponential number of ways to merge structures (NP Hard). But using heuristics cuts down the search space.

140 Caused-Motion-Action agent: patient: (1) path: Structure Merging Example Utterance:It is too heavy for you to pour. [Addressee < Anim] It < Entity num:sg … Caused-Motion-Action agent: [Animate] patient: (1) [Entity] path: Before Merging:After Merging: It num:sg … TrajectorLandmark trajector : (1) landmark : [Entity] TrajectorLandmark trajector : (1) landmark : [Addressee ]

141 Semantic Density Semantic density is a simple heuristic to choose between competing analyses. Density of an analysis = (filled roles) / (total roles) The system prefers higher density analyses because a higher density suggests that more frame roles are filled in than in competing analyses. Extremely simple / useful? but it certainly can be improved upon.

142 Measuring Semantic Density The frame on the left has a density of.75 while the frame on the right has a density of 1.

143 Semantic Density At Work [Addressee] Caused-Motion-Action agent : [Animate] patient : [Entity] path:[TrajLM] TrajectorLandmark trajector : landmark : here Motion-Action mover : path:[TrajLM] You move over here or Assume that “You move over here” isn’t recognized and that “move” has two senses.

144 Semantic Density At Work 2 Caused-Motion-Action agent: (1) [Addressee] patient: /**unfilled**/ path: TrajectorLandmark trajector : (1) landmark : here Motion-Action agent: (1) [Addressee] path: TrajectorLandmark trajector : (1) landmark : here There are 2 resulting merged analyses. The Motion-Action frame has a density of 3/3 = 1 and the Caused-Motion-Action frame has a density of 3/4 =.75 because of the missing patient.

145 Language Understanding Process

146 Simulation specification A simulation specification consists of: - schemas evoked by constructions - bindings between schemas

147 Summary: ECG Linguistic constructions are tied to a model of simulated action and perception Embedded in a theory of language processing –Constrains theory to be usable –Frees structures to be just structures, used in processing Precise, computationally usable formalism –Practical computational applications, like MT and NLU –Testing of functionality, e.g. language learning A shared theory and formalism for different cognitive mechanisms –Constructions, metaphor, mental spaces, etc.

148 ECG applications Grammar –Spatial relations/events (Bergen & Chang 1999; Bretones et al. In press) –Verbal morphology (Gurevich 2003, Bergen ms.) –Reference: measure phrases (Dodge and Wright 2002), construal resolution (Porzel & Bryant 2003), reflexive pronouns (Sanders 2003) Semantic representations / inference –Aspectual inference (Narayanan 1997; Chang, Gildea & Narayanan 1998) –Perspective / frames (Chang, Narayanan & Petruck 2002) –Metaphorical inference (Narayanan 1997, 1999) –Simulation semantics (Narayanan 1997, 1999) Language acquisition –Lexical acquisition (Regier 1996, Bailey 1997) –Multi-word constructions (Chang 2004; Chang & Maia 2001)

149 FrameNet for AQUAINT Scenarios FrameNet is annotating the AQUAINT CNS data with frame information –Design and build FN database of Frames and Schemas relavant to the AQUAINT data –Select a set of representative documents and annotate them with FrameNet frames. Gold Standard annotations for the program. All the FrameNet data (DB and annotations) will be released in RDF/OWL in addition to the XML format.

150 Example Document: Country Profile- Libya Frame TypeDescriptionNumber Action Verbs of action often hostile action 50 Resources Resources, assets, capabilities 5 Attitude Different attitude frames 15 Weaponry Different types and descriptions 75 Treaty Different treaties named 12 Organization Organizations named 19

151 FrameNet annotation of CNS data

152 Scaling Up –Scaling Up Language Embodied Construction Grammar FrameNet for Scenarios –Scaling Up Inference Coordinated Probabilistic Relational Model (CPRM) An Application to QA inference –Scaling Up Ontological Knowledge Semantic Web –OWL and OWL-S –An Event Ontology in OWL

153 Structured Probabilistic Inference

154 Probabilistic inference –Filtering P(X_t | o_1…t,X_1…t) Update the state based on the observation sequence and state set –MAP Estimation Argmax h1…hn P(X_t | o_1…t, X_1…t) Return the best assignment of values to the hypothesis variables given the observation and states –Smoothing P(X_t-k | o_1…t, X_1…t) modify assumptions about previous states, given observation sequence and state set –Projection/Prediction/Reachability P(X_t+k | o_1..t, X_1..t)

155 Relational Models Relational models make some ontological commitments: the world consists of objects, and relations over them PRMs are based on a particular “relational logic” borrowed from databases: DatabasesRelational Logic TableClass TupleObject Standard FieldDescriptive Attribute Foreign Key FieldReference Slot

156 PRM Introduction PRMs allow objects to be augmented with a description of relations between instances. The object relational structure can be a relational database or logic program. The PRM model augments the database with probabilities. The model is useful in cases where the configuration of a system (instances) change while the relational schema remains constant.

157 Probabilistic Relational Models A PRM for a Relational Schema S is defined as:  For each Class C and propositional Attribute A(C) in C we have  A set of parents of A (simple, complex or aggregate)  A CPD P(C.A | Pa(C.A))

158 Inference With PRMs SVE inference for a PRM P with q query variables and N attributes is O(Nkb k(m+2) b q ) (Pfeffer 2000)  k is the maximum number of interface variables  q is the number of query variables  m is the maximum tree width for any object in P (related to the markov blanket).

159 Controlling PRM inference The number of interface variables, k, is related to the number of relations that a variable participates in as well as the number of slot chains that the variable participates in –Careful selection of relations (only part-of) can make inference tractable. The tree width m depends on the markov blanket of an attribute. –Control of network topology can reduce this.

160 Adding time to PRM D(T)PRM A two-time-slice PRM (2TPRM) for a relational schema S is defined as follows. For each class C and each propositional attribute A 2 A(C), we have: –A set of parents Pa(C:A) = f{Pa1; Pa2;.. Pal}, where –each Pai has the form C:B or f(C.rho.B), where rho is a slot chain containing the attribute previous at most once, and f() is an aggregation function. –A conditional probability model for P(C.A | Pa(C.A))

161 Adding Time to PRM’s Since time is another relation, doesn’t increase expressive power. –Significant impact of inference tractability since both k and m may become quite large. New Algorithm: Exploit the structure of time using the interface and frontier algorithm (Murphy 2002). –Variables at slice t with links to variables at t+1 form the interface –Interface variables d-separate the past ( t). –Allows for on-line inference algorithms similar to inside-outside algorithm for SCFG’s. –Use SVE to rollover slices. Approximation using Rao-Blackwelized particle filtering (Sanghai, Domingos, and Weld 2003)

162 Structured Probabilistic Inference

163 CPRM inference Combines insights from –the SVE algorithm for PRMs (Pfeffer 2000) –the frontier algorithms for temporal models (Murphy 2002) and the BN SCFG algorithm (Narayanan 99) –Inference algorithms for complex, coordinated events (Narayanan 2002) Expressive Probabilistic Modeling paradigm with relations and branching dynamics. Offers principled methods to bound inferential complexity.

164 Temporal Projection in CPRM

165 Answer Type to Inference Method ANSWER TYPEINFERENCEDESCRIPTION Justify (Proposition)MAPProposition is part of the MAP Ability (Agent, Act)Filtering; Smoothing Past/Current Action enabled given current state Prediction (State)P;R’ MAPPropogate current information and estimate best new state Hypothetical (Condition)S, R_ISmooth intervene and compute state

166 Event Simulation Predicate Extraction Retrieved Documents FrameNet Frames OWL/OWL-S Topic Ontologies Model Parameterization C O N T E X T PRM

167 AnswerBank  AnswerBank is a collection of over a 1200 QA annotations from the AQUAINT CNS corpus.  Questions and answers cover the different domains of the CNS data.  Questions and answers are POS tagged, and syntactically parsed.  Question and Answer predicates are annotated with PropBank arguments and FrameNet (when available) tags.  FrameNet is annotating CNS data with frame information for use by the AQUAINT QA community.  We are planning to add more semantic information including temporal, aspectual information (TIMEML+) and information about event relations and figurative uses.

168 Answer Types for complex questions in AnswerBank ANSWER TYPEEXAMPLENUMBER Justify (Proposition)What is the evidence that IRAQ has WMD? 89 Ability (Agent, Act)How can a Biological Weapons Program be detected? 71 Prediction (State)What were the possible ramifications of India’s launch of the Prithvi missile? 63 Hypothetical (Condition)If Musharraf is removed from power, will Pakistan become a militant Islamic State? 62

169 Building Models Gold Standard: –From the hand-annotated data in the CNS corpus, we manually built CPRM domain models for inference. Semantic Web based: –From FrameNet frames and from semantic web ontologies in OWL (SUMO-based, OpenCYC and others), we built CPRM models (semi-automatic)

170

171 Event Structure Inferences For the annotations we classify complex event structure inferences as –Aspectual Stages of events, viewpoints, temporal relations (such as start(ev1, ev2), interrupt(ev1, ev2)) –Action-Based Resources (produce,consume,lock), preconditions, maintenance conditions, effects. –Metaphoric Event Structure Metaphor (ESM) Events and predications (motion => Action), objects (Motion.Mover => Action.Actor), Parameters(Motion.speed =>Action.rateOfProgress)

172

173 Scaling Up –Scaling Up Language Embodied Construction Grammar FrameNet for Scenarios –Scaling Up Inference Coordinated Probabilistic Relational Model (CPRM) An Application to QA inference –Scaling Up Ontological Knowledge Semantic Web –OWL and OWL-S –An Event Ontology in OWL

174 ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium Answer Structure

175 Content of Inferences ComponentNumberF-Score Manual F-Score OWL Aspectual375.74.65 Action- Feature 459.62.45 Metaphor149.70.62

176 Semantic Web The World Wide Web (WWW) contains a large and expanding information base. HTML is accessible to humans but does not formally describe data in a machine interpretable form. XML remedies this by allowing for the use of tags to describe data (ex. disambiguating crawl) Ontologies are useful to describe objects and their inter-relationships. DAML+OIL (http://www.daml.org) is an markup language based on XML and RDF that is grounded in description logic and is designed to allow for ontology development, transfer, and inference on the web.http://www.daml.org

177 Programmatic Access to the web Web-accessible programs and devices

178 Knowledge Rep’n for the “Semantic Web” XML Schema RDF (Resource Description Framework) RDFS (RDF Schema) OWL/DAML-L (Logic) OWL (Ontology) XML (Extensible Markup Language)

179 Knowledge Rep’n for “Semantic Web Services” XML Schema RDF (Resource Description Framework) RDFS (RDF Schema) DAML-L (Logic) DAML+OIL (Ontology) XML (Extensible Markup Language) DAML-S (Services)

180 The OWL Language OWL REF

181 DAML-S: Semantic Markup for Web Services DAML-S: A DARPA Agent Markup Language for Services DAML+OIL ontology for Web services: well-defined semantics ontologies support reuse, mapping, succinct markup,... Developed by a coalition of researchers from Stanford, SRI, CMU, BBN, and Nokia, Yale, under the auspices of DARPA. DAML-S version 0.6 posted October,2001 http://www.daml.org/services/daml-s [DAML-S Coalition, 2001, 2002] [Narayanan & McIlraith 2003]

182 DAML-S/OWL-S Compositional Primitives process atomic process composite process inputs (conditional) outputs preconditions (conditional) effects control constructs composedBy while sequence If-then-else fork...

183 PROCESS.OWL The OWL-S Process Description

184 Atomic Processes as X-schemas SC Knowledge Preconditions ^ Kref Input Poss k (a,s) Action a Effect......

185 DAML-S Processes as X-schemas …and SC World Preconditions ^ Kref Input Poss k (a,s) Action a Effect...... ^  nn World Poss w (a,s)......

186 Composite Process Constructs Control ConstructDescription Sequence Execute a list of processes in sequence. Concurrent Fork off a bag of processes. Concurrent-Sync Fork off a bag of processes & synchronize/join after completion Choice Choose from a bag of processes. Iterate (condition) Repeat process until (while ). If-Then-Else If then THEN else ELSE.

187 Modeling Composite Process Constructs startfinish ReadyDone Component Control Construct

188 DAML-S Sequence: P1;P2 startfinish Done(P1;P2) Atomic Process P2 Done(P1) Atomic Process P1 Ready

189 DAML-S Sequence: P1;P2 startfinish Done(P1;P2) Atomic Process P2 Done(P1) Atomic Process P1 Ready

190 DAML-S Sequence START FINISH Atomic Process P1 Process P2 DONE(P1) DONE(P1;P2) READY

191 DAML-S Fork: P1|| P2 start finish Done(P1 || P2) Atomic Process P2 Ready(P1) Atomic Process P1 Ready(P2)

192 DAML-S Fork START FINISH P2 DONE(P1|| P2) READY (P1)P1 READY (P2)

193 DAML-S Concurrent-Sync Done(P2) Done(P1) startfinish Atomic Process P2 Ready(P1) Atomic Process P1 Ready(P2)

194 DAML-S Concurrent-Sync START FINISH P2 DONE(P1)READY (P1)P1 READY (P2)DONE(P2)

195 Implementation DAML-S translation to the modeling environment KarmaSIM [Narayanan, 97] (http://www.icsi.berkeley.edu/~snarayan) Basic Program: Input: DAML-S description of Events Output: Network Description of Events in KarmaSIM Procedure: Recursively construct a sub-network for each control construct. Bottom out at atomic event. Construct a net for each atomic event Return network

196

197

198 Example of A WMD Ontology in OWL rdfs:Class rdfs:subClassOfSUMO.owl#Making rdfs:comment Making instances of WeaponOfMassDestruction. rdfs:comment rdfs:Class http://reliant.teknowledge.com/DAML/SUMO.owl

199

200 FrameNet in OWL Program FNtoOWL –Implemented in Java –Uses the JENA API –Given an XML FrameNet database –File in the FN XML format (schema and dtd) The Location (file, URL) of the resulting OWL markup A URL for the FrameNet ontology file –Output An OWL file for the FrameNet database at the specified location. –URL: http://www.icsi.berkeley.edu/~snarayan/FNtoOWL.zip. gz

201 FrameNet in OWL

202 Conclusion Answering complex questions requires semantic representations at multiple levels. –NE and Extraction-based –Predicate Argument Structures –Frame, Topic and Domain Models All these representations should be capable of supporting inference about relational structures, uncertain information, and dynamic context. Both Semantic Extraction techniques and Structured Probabilistic KR and Inference methods have matured to the point that we understand the various algorithms and their properties. Flexible architectures that –embody these KR and inference techniques and –make use of the expanding linguistic and ontological resources (such as on the Semantic Web) Point the way to the future of semantically based QA systems!

203 References (URL) Semantic Resources –FrameNet: http://www.icsi.berkeley.edu/framenet (Papers on FrameNet and Computational Modeling efforts using FrameNet can be found here).http://www.icsi.berkeley.edu/framenet –PropBank: http://www.cis.upenn.edu/~ace/http://www.cis.upenn.edu/~ace/ –Gildea’s Verb Index; http://www.cs.rochester.edu/~gildea/Verbs/ (links FrameNet, PropBank, and VerbNethttp://www.cs.rochester.edu/~gildea/Verbs/ Probabilistic KR (PRM) –http://robotics.stanford.edu/~koller/papers/lprm.ps (Learning PRM)http://robotics.stanford.edu/~koller/papers/lprm.ps –http://www.eecs.harvard.edu/~avi/Papers/thesis.ps.gz (Avi Pfeffer’s PRM Stanford thesis)http://www.eecs.harvard.edu/~avi/Papers/thesis.ps.gz –David Poole IJCAI 2003 –Dan Roth NIPS 2004 (submitted) Dynamic Bayes Nets –http://www.ai.mit.edu/~murphyk/Thesis/thesis.pdf (Kevin Murphy’s Berkeley DBN thesis)http://www.ai.mit.edu/~murphyk/Thesis/thesis.pdf –Weld et all 2003, DPRM Event Structure in Language –http://www.icsi.berkeley.edu/~snarayan/thesis.pdf (Narayanan’s Berkeley PhD thesis on models of metaphor and aspect)http://www.icsi.berkeley.edu/~snarayan/thesis.pdf –ftp://ftp.cis.upenn.edu/pub/steedman/temporality/temporality.ps.gz (Steedman’s article on Temporality with links to previous work on aspect)ftp://ftp.cis.upenn.edu/pub/steedman/temporality/temporality.ps.gz –http://www.icsi.berkeley.edu/NTL (publications on Cognitive Linguistics and computational models of cognitive linguistic phenomena can be found here)http://www.icsi.berkeley.edu/NTL

204 References (URL) Semantic Web –Scientific American Article –http://www.semanticWeb.orghttp://www.semanticWeb.org OWL –http://www.owl.orghttp://www.owl.org OWL-S –http://www.daml.org/serviceshttp://www.daml.org/services

205 Representing Event Frames At the computational level, we use a structured event representation of event frames that formally specify –The frame –Frame Elements and filler types –Constraints and role bindings –Frame-to-Frame relations Subcase Subevent

206 Events and actions schema Event roles before : Phase transition : Phase after : Phase nucleus constraints transition :: nucleus schema Action evokes Event as e roles actor : Entity undergoer : Entity self  e.nucleus beforeaftertransition nucleus undergoer actor

207 The Commercial-Transaction schema schema Commercial-Transaction subcase of Exchange roles customer  participant1 vendor  participant2 money  entity1 : Money goods  entity2 goods-transfer  transfer1 money-transfer  transfer2

208 Implementation DAML-S translation to the modeling environment KarmaSIM [Narayanan, 97] (http://www.icsi.berkeley.edu/~snarayan) Basic Program: Input: DAML-S description of Frame relations + Ontology Output: Network Description of Frames in KarmaSIM Procedure: Recursively construct a sub-network for each control construct. Bottom out at atomic frame. Construct a net for each atomic frame Return network

209

210

211

212

213

214

215 Outline Part V. From Ontologies to Inference –From OWL to CPRM –FrameNet in OWL –FrameNet to CPRM mapping Part VI. A Pilot QA System Implementation –AnswerBank examples –Current results for Inference Type –Current results for Answer Structure

216 AnswerBank  AnswerBank is a collection of over a 1200 QA annotations from the AQUAINT CNS corpus.  Questions and answers cover the different domains of the CNS data.  Questions and answers are POS tagged, and syntactically parsed.  Question and Answer predicates are annotated with PropBank arguments and FrameNet (when available) tags.  FrameNet is annotating CNS data with frame information for use by the AQUAINT QA community.  We are planning to add more semantic information including temporal, aspectual information (TIMEML+) and information about event relations and figurative uses.

217 Event Simulation Predicate Extraction Retrieved Documents FrameNet Frames OWL/OWL-S Topic Ontologies Model Parameterization C O N T E X T PRM

218 Answer Types for complex questions in AnswerBank ANSWER TYPEEXAMPLENUMBER Justify (Proposition)What is the evidence that IRAQ has WMD? 89 Ability (Agent, Act)How can a Biological Weapons Program be detected? 71 Prediction (State)What were the possible ramifications of India’s launch of the Prithvi missile? 63 Hypothetical (Condition)If Musharraf is removed from power, will Pakistan become a militant Islamic State? 62

219 Answer Type to Inference Method ANSWER TYPEINFERENCEDESCRIPTION Justify (Proposition)MAPProposition is part of the MAP Ability (Agent, Act)Filtering; Smoothing Past/Current Action enabled given current state Prediction (State)P;R’ MAPPropogate current information and estimate best new state Hypothetical (Condition)S, R_ISmooth intervene and compute state

220 Conclusion Answering complex questions requires semantic representations at multiple levels. –NE and Extraction-based –Predicate Argument Structures –Frame, Topic and Domain Models All these representations should be capable of supporting inference about relational structures, uncertain information, and dynamic context. Both Semantic Extraction techniques and Structured Probabilistic KR and Inference methods have matured to the point that we understand the various algorithms and their properties. Flexible architectures that –embody these KR and inference techniques and –make use of the expanding linguistic and ontological resources (such as on the Semantic Web) Point the way to the future of semantically based QA systems!

221 References (URL) Semantic Resources –FrameNet: http://www.icsi.berkeley.edu/framenet (Papers on FrameNet and Computational Modeling efforts using FrameNet can be found here).http://www.icsi.berkeley.edu/framenet –PropBank: http://www.cis.upenn.edu/~ace/http://www.cis.upenn.edu/~ace/ –Gildea’s Verb Index; http://www.cs.rochester.edu/~gildea/Verbs/ (links FrameNet, PropBank, and VerbNethttp://www.cs.rochester.edu/~gildea/Verbs/ Probabilistic KR (PRM) –http://robotics.stanford.edu/~koller/papers/lprm.ps (Learning PRM)http://robotics.stanford.edu/~koller/papers/lprm.ps –http://www.eecs.harvard.edu/~avi/Papers/thesis.ps.gz (Avi Pfeffer’s PRM Stanford thesis)http://www.eecs.harvard.edu/~avi/Papers/thesis.ps.gz –David Poole IJCAI 2003 –Dan Roth NIPS 2004 (submitted) Dynamic Bayes Nets –http://www.ai.mit.edu/~murphyk/Thesis/thesis.pdf (Kevin Murphy’s Berkeley DBN thesis)http://www.ai.mit.edu/~murphyk/Thesis/thesis.pdf –Weld et all 2003, DPRM Event Structure in Language –http://www.icsi.berkeley.edu/~snarayan/thesis.pdf (Narayanan’s Berkeley PhD thesis on models of metaphor and aspect)http://www.icsi.berkeley.edu/~snarayan/thesis.pdf –ftp://ftp.cis.upenn.edu/pub/steedman/temporality/temporality.ps.gz (Steedman’s article on Temporality with links to previous work on aspect)ftp://ftp.cis.upenn.edu/pub/steedman/temporality/temporality.ps.gz –http://www.icsi.berkeley.edu/NTL (publications on Cognitive Linguistics and computational models of cognitive linguistic phenomena can be found here)http://www.icsi.berkeley.edu/NTL

222 References (URL) Semantic Web –Scientific American Article –http://www.semanticWeb.orghttp://www.semanticWeb.org OWL –http://www.owl.orghttp://www.owl.org OWL-S –http://www.daml.org/serviceshttp://www.daml.org/services


Download ppt "Structured Probabilistic Inference in an Embodied Construction Grammar and Jerome Feldman International Computer Science Institute U. California at Berkeley."

Similar presentations


Ads by Google