Structured Probabilistic Inference in an Embodied Construction Grammar

Structured Probabilistic Inference in an Embodied Construction Grammar
and Jerome Feldman International Computer Science Institute U. California at Berkeley Berkeley, CA Srini Narayanan International Computer Science Institute Berkeley, CA

Language, Learning and Neural Modeling
Scientific Goal Understand how people learn and use language Practical Goal Build systems that analyze and produce language Approach Embodied linguistic theories with advanced biologically-based computational methods

State of the Art Limited Commercial Speech Applications
transcription, simple response systems Statistical NLP for Restricted Tasks tagging, parsing, information retrieval Template-based Understanding programs expensive, brittle, inflexible, unnatural Essentially no NLU in QA Systems

Hypothesis NLU is essential to large, open domain QA.
One line dismissals (lack of world knowledge) are out of date (by at least 5 yrs.) Substantial Progress in Enabling Technologies Knowledge Representation/Inference Techniques Active Knowledge Dealing With Uncertainty. Simulation Semantics Scaling Up CYC, Wordnet, Term-bases FrameNet, Semantic Web, MetaNet Extraction of Semantic Relations Empirical Linguistic The goal of NLU can be realized

Hypothesis NLU is essential to large, open domain QA.
One line dismissals (lack of world knowledge) are out of date (by at least 5 yrs.) Substantial Progress in Enabling Technologies Knowledge Representation/Inference Techniques Active Knowledge Dealing With Uncertainty. Simulation Semantics Scaling Up CYC, Wordnet, Term-bases FrameNet, Semantic Web, MetaNet Extraction of Semantic Relations Empirical Linguistic The goal of NLU can be realized, perhaps! Anyway, it is time to try again.

Complex questions and semantic information (SH, UTD)
Complex questions are not characterized only by a question class (e.g. manner questions) Example: How can a biological weapons program be detected ? Associated with the pattern “How can X be detected?” And the topic X = “biological weapons program” Processing complex questions is also based on access to the semantics of the question topic The topic is modeled by a set of discriminating relations, e.g. Develop(program); Produce(biological weapons); Acquire(biological weapons) or stockpile(biological weapons) Such relations are extracted from topic-relevant texts

The need for Semantic Inference in QA
Some questions are complex! Example (from UTD CNS QA database): What is the evidence that IRAQ has WMD? Answer: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors.

Answer Structure ANSWER: Evidence-Combined: Pointer to Text Source:
A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. Answer Structure A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium

Answer Structure (continued)
ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: possess(Iraq, delivery systems(type : scud missiles; launchers; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium possess(Iraq, fuel stock(purpose: power launchers)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium hide(Iraq, Seeker: UN Inspectors; Hidden: CBW stockpiles & guided missiles) Justification: DETECTION Schema Inspection status: Past; Likelihood: Medium

Semantic inference for Q/A
The problem of classifying questions E.g. “manner questions”: Example “How did Hitler die?” The problem of recognizing answer types/structures Should “manner of death” by considered an answer type? What other manner of event/action should be considered as answer types? The problem of extracting/justifying/ generating answers to complex questions Should we learn to extract “manner” relations? What other types of relations should we consider? Is relation recognition sufficient for answering complex questions? Is it necessary?

Outline Part I. Introduction: Semantic Inference for QA
Part II. Cognitive Primitives Embodiment in Language Conceptual Relations Cross-linguistic Relations Image Schemas Active Schemas: X-schemas Domain Specific Knowledge Cultural Frames and FrameNet Ontologies

Embodiment Alan Turing (Intelligent Machines,1948)
Of all of these fields, the learning of languages would be the most impressive, since it is the most human of these activities. This field, however, seems to depend rather too much on the sense organs and locomotion to be feasible. Alan Turing (Intelligent Machines,1948)

NTL Manifesto Basic concepts and words derive their meaning from embodied experience. Abstract and Theoretical concepts derive their meaning from metaphorical maps to more basic embodied concepts. Structured Connectionist Models can capture both of these processes nicely.

General and Domain Knowledge
Conceptual Knowledge and Inference Embodied Language and Domain Independent Powerful General Inferences Ubiquitous in Language Domain Specific Frames and Ontologies Framenet Metaphor links domain specific to general E.g., France slipped into recession.

Conceptual schemas Much is known about conceptual schemas
Image Schemas X-schemas Frames Constructions Recently we have developed a scalable formalism for representing conceptual schemas We are testing and refining both the content and formalism based on other languages.

What are Image schemas? Regularities in our perceptual, motor and cognitive systems Structure our experiences and interactions with the world. May be grounded in a specific cognitive system, but are not situation-specific in their application (can apply to many domains of experience) So where do image schemas fit into this? --The idea is that there are regularities in our perceptual, motor and cognitive systems that structure our experiences and interactions in the world. -- For spatial relations, these structural regularities are presumably the basis for semantic primitives. Image schemas describe these primitives, but in addition, primitive schemas can be combined to form more complex image schemas. Image schemas are schematic in at least two ways – though they may be grounded in a specific cognitive or perceptual system, they are not situation-specific in their application (i.e. can apply to many domains of experience, unlike many frames). -- The entities that they apply to are only schematically specified, e.g. they do not apply only to specific shapes or types of objects.

Basis of Image schemas Perceptual systems Motor routines
Social Cognition Image Schema properties depend on Neural circuits Interactions with the world What sort of regularities are we talking about, and what is their basis? Perceptual system: -- Visual system – edge-detecting and orientation-sensitive cells CHECK -- Equilibrium – know orientation of head and body relative to gravity -- Proprioceptic – we are sensitive to the changing tensions of muscles and tendons -- We can sense contact and pressure on our skin Motor routines, with their own structure often referred to as X-schemas [will hear more on this later in course] Neural basis, e.g. generalization of input -- different pathways in the brain – the so-called “what” and “where” pathways in the brain These may use different types of visual information (and for different purposes, e.g. object identification vs. location) Interactions with the world -- affordances of objects, functional purpose of interaction (e.g motor control) --Information from the primary visual cortex (located at the back of the head) is transmitted along two pathways -- the ventral stream to the temporal cortex (the so-called "what" system) and the dorsal stream to the parietal cortex (the "where" system).

Representing image schemas
semantic schema Source-Path-Goal roles: source path goal trajector semantic schema Container roles: interior exterior portal boundary Boundary Interior Trajector Portal Source Goal Path Exterior These are abstractions over sensorimotor experiences.

Schema Formalism SCHEMA <name> SUBCASE OF <schema>
EVOKES <schema> AS <local name> ROLES < self role name>: <role restriction> < self role name> <-> <role name> CONSTRAINTS <role name> <- <value> <role name> <-> <role name> <setting name> :: <role name> <-> <role name> <setting name> :: <predicate> | <predicate>

A Simple Example SCHEMA hypotenuse SUBCASE OF line-segment
SCHEMA hypotenuse SUBCASE OF line-segment EVOKES right-triangle AS rt ROLES Comment inherited from line-segment CONSTRAINTS SELF <-> rt.long-side

Source-Path-Goal SCHEMA: spg ROLES: source: Place path: Directed Curve
SCHEMA: spg ROLES: source: Place path: Directed Curve goal: Place trajector: Entity

Translational Motion SCHEMA translational motion SUBCASE OF motion
SCHEMA translational motion SUBCASE OF motion EVOKES spg AS s ROLES mover <-> s.trajector source <-> s.source goal <-> s.goal CONSTRAINTS before:: mover.location <-> source after:: mover.location <-> goal

Extending Inferential Capabilities
Given the formalization of the conceptual schemas How to use them for inferencing? Earlier pilot systems Used metaphor and Bayesian belief networks Successfully construed certain inferences But don’t scale to large open domains New approach Probabilistic relational models Support an open ontology

Frames Frames are conceptual structures that may be culture specific
Words evoke frames The word “talk” evokes the Communication frame The word buy (sell, pay) evoke the Commercial Transaction (CT) frame. The words journey, set out, schedule, reach etc. evoke the Journey frame. Frames have roles and constraints like schemas. CT has roles vendor, goods, money, customer. Words bind to frames by specifying binding patterns Buyer binds to Customer, Vendor binds to Seller.

The FrameNet Project C Fillmore PI (ICSI)
Co-PI’s: S Narayanan (ICSI, SRI) D Jurafsky (U Colorado) J M Gawron (San Diego State U) Staff: C Baker Project Manager B Cronin Programmer C Wooters Database Designer In addition to the main players, named here, there are at any time six to ten students working with us and one or two visitors from other centers.

Frames and Understanding
Hypothesis: People understand things by performing mental operations on what they already know. Such knowledge is describable in terms of information packets called frames.

FrameNet in the Larger Context
The long-term goal is to reason about the world in a way that humans understand and agree with. Such a system requires a knowledge representation that includes the level of frames. FrameNet can provide such knowledge for a number of domains. FrameNet representations complement ontologies and lexicons.

The core work of FrameNet
characterize frames find words that fit the frames develop descriptive terminology extract sample sentences annotate selected examples derive "valence" descriptions

The Core Data The basic data on which FrameNet descriptions are based take the form of a collection of annotated sentences, each coded for the combinatorial properties of one word in it. The annotation is done manually, but several steps are computer-assisted.

Types of Words / Frames events artifacts, built objects
natural kinds, parts and aggregates terrain features institutions, belief systems, practices space, time, location, motion etc. the vocabulary of a language has many kinds of words; the names of event types are the central objects of our concern in the various applications we’re trying out

FrameNet Product For every target word,
describe the frames or conceptual structures which underlie them, and annotate example sentences that cover the ways in which information from the associated frames are expressed in these sentences. In the lexicon building aspects of our work,…

Complex Frames With Criminal_process we have, for example,
sub-frame relations (one frame is a component of a larger more abstract frame) and temporal relations (one process precedes another)

Frame-to-Frame Relations: Crime_scenario

Frame Relations Frames are linked to each other by various types of relations, three of which are: Inheritance: all FEs of parent frame have corresponding FEs in child (not necessarily with the same name) Using: like inheritance, but not all FEs are bound Subframe: used for sub-events of a complex event; may be temporally ordered (or not) We’ll discuss these three in more detail later.

Applying Frame Structures to QA
Parsing Questions Parsing Answers Result: exact answer= “approximately 7 kg of HEU” Q: What kind of materials were stolen from the Russian navy? FS(Q): What [GOODS: kind of nuclear materials] were [Target-Predicate:stolen] [VICTIM: from the Russian Navy]? A(Q): Russia’s Pacific Fleet has also fallen prey to nuclear theft; in 1/96, approximately 7 kg of HEU was reportedly stolen from a naval base in Sovetskaya Gavan. FS(A(Q)): [VICTIM(P1): Russia’s Pacific Fleet] has also fallen prey to [Goods(P1): nuclear ] [Target-Predicate(P1): theft]; in 1/96, [GOODS(P2): approximately 7 kg of HEU] was reportedly [Target-Predicate (P2): stolen] [VICTIM (P2): from a naval base] [SOURCE(P2): in Sovetskawa Gavan]

Language understanding via simulation
Hypothesis: Linguistic input is converted into a mental simulation based on bodily grounded structures Semantic schemas, including image schemas (Johnson 1987) and executing schemas (Bailey, Narayanan 1997) are abstractions over neurally grounded perceptual and motor representations Linguistic units make reference to these structures in their semantic pole Analysis produces a simulation specification linking these structures and providing parameters for a simulation engine Model requires: schema representations (image, motor, frame, social, etc.) lexical and phrasal construction representations that invoke those schemas and other cultural frames (FrameNet+ frames)

Simulation-based language understanding
“Harry walked to the cafe.” Utterance Constructions Analysis Process General Knowledge Simulation Specification Schema Trajector Goal walk Harry cafe Belief State Cafe Simulation

Embodied Construction Grammar (Bergen & Chang 2002)
Assumptions from Construction Grammar Constructions are form-meaning pairs (Lakoff 1987, Goldberg 1995) Constructions vary in degree of specificity and level of description (morphological, lexical, phrasal, clausal) Constructions evoke and bind semantic schemas Additional influences Cognitive Grammar (Langacker 1987) Frame Semantics (Fillmore 1982) Structured Connectionism (Feldman 1987)

Simulation specification
It should be clear that the simulation specification includes exactly the schematic content of the different elements of the sentence, bound appropriately. As noted earlier, the two representations differ with respect to which image schemas are involved – as reflected by the additional CONTAINER schema in Figure 5b – and in the precise bindings of aspects of the cafe to the SPG schema. Like the image schema representations, the simulation specifications can be viewed as a summary of the much more complex structures that are active when an event is simulated or imagined. Activating these structures – that is, “running” the simulation – can thus provide the much richer basis for inference necessary for accounting for many linguistic phenomena The analysis process produces a simulation specification that includes image-schematic, motor control and conceptual structures provides parameters for a mental simulation

Simulation Semantics BASIC ASSUMPTION: SAME REPRESENTATION FOR PLANNING AND SIMULATIVE INFERENCE Evidence for common mechanisms for recognition and action (mirror neurons) in the F5 area (Rizzolatti et al (1996), Gallese 96, Buccino 2002, Tettamanti 2004) and from motor imagery (Jeannerod 1996) IMPLEMENTATION: x-schemas affect each other by enabling, disabling or modifying execution trajectories. Whenever the CONTROLLER schema makes a transition it may set, get, or modify state leading to triggering or modification of other x-schemas. State is completely distributed (a graph marking) over the network. RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!

Outline Part III. Knowledge representation and inference
KR requirements Dynamic Context and the structure of Events Uncertainty in knowledge sources Representing Metaphors and Maps Active Knowledge and Event Structure X-schemas A computational model of Aspect Bayesian Models of Uncertainty Graphical Models Dynamic Bayes Nets for Modeling State A Computational Model of Metaphor Interpretation DEMO

Answer Structure Temporal Reference/Grounding
ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. Answer Structure A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. Temporal Reference/Grounding A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium

Answer Structure (continued)
ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. Present Progressive Perfect A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: possess(Iraq, delivery systems(type : scud missiles; launchers; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium Present Progressive Continuing possess(Iraq, fuel stock(purpose: power launchers)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium hide(Iraq, Seeker: UN Inspectors; Hidden: CBW stockpiles & guided missiles) Justification: DETECTION Schema Inspection status: Past; Likelihood: Medium

Answer Structure Uncertainty and Belief
ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. Answer Structure A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. Uncertainty and Belief A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium

Answer Structure Uncertainty and Belief
ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. Answer Structure A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. Uncertainty and Belief A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Multiple partly reliable sources Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium

Answer Structure Event Structure Metaphor
ANSWER: Evidence-Combined: Pointer to Text Source: A1: In recent months, Milton Leitenberg, an expert on biological weapons, has been looking at this murkiest and most dangerous corner of Saddam Hussein's armory. A2: He says a series of reports add up to indications that Iraq may be trying to develop a new viral agent, possibly in underground laboratories at a military complex near Baghdad where Iraqis first chased away inspectors six years ago. Answer Structure A3: A new assessment by the United Nations suggests Iraq still has chemical and biological weapons - as well as the rockets to deliver them to targets in other countries. Event Structure Metaphor A4:The UN document says Iraq may have hidden a number of Scud missiles, as well as launchers and stocks of fuel. A5: US intelligence believes Iraq still has stockpiles of chemical and biological weapons and guided missiles, which it hid from the UN inspectors Content: Biological Weapons Program: develop(Iraq, Viral_Agent(instance_of:new)) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Inspection terminated; Status: Attempt ongoing Likelihood: Medium Confirmability: difficult, obtuse, hidden possess(Iraq, Chemical and Biological Weapons) Justification: POSSESSION Schema Previous (Intent and Ability): Prevent(ability, Inspection); Status: Hidden from Inspectors Likelihood: Medium possess(Iraq, delivery systems(type : rockets; target: other countries)) Justification: POSSESSION Schema Previous (Intent and Ability): Hidden from Inspectors; Status: Ongoing Likelihood: Medium

Temporal relations in QA
Results of the workshop are accessible from A set of questions that require the extraction of temporal relations was created (TimeML question corpus) E.g.: “When did the war between Iran and Iraq end?” “Who was Secretary of Defense during the Golf War?” A number of features of these questions were identified and annotated Number of TEMPEX relations in the question Volatility of the question (how often does the answer change) Reference to repetitive events Number of events mentioned in the question

Event Structure for semantically based QA
Reasoning about dynamics Complex event structure Multiple stages, interruptions, resources, framing Evolving events Conditional events, presuppositions. Nested temporal and aspectual references Past, future event references Metaphoric references Use of motion domain to describe complex events. Reasoning with Uncertainty Combining evidence from multiple, unreliable sources Non-monotonic inference Retracting previous assertions Conditioning on partial evidence Linguistic Ambiguity Figurative inference

Relevant Previous Work
Event Structure Aspect (VDT, TimeML), Situation Calculus (Steedman), Frame Semantics (Fillmore), Cognitive Linguistics (Langacker, Talmy, Lakoff, Sweetser), Metaphor and Aspect (Narayanan) Reasoning about Uncertainty Bayes Nets (Pearl), Probabilistic Relational Models (Pfeffer, Koller), Graphical Models (Jordan), First Order Probabilistic Inference (Poole, Braz et al) Reasoning about Dynamics Dynamic Bayes Nets (Murphy, Friedman), Distributed Systems (Alur, Meseguer), Control Theory (Ramadge and Wonham), Causality (Pearl)

Active representations
Many inferences about actions derive from what we know about executing them Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions walker at goal Walking: bound to a specific walker with a direction or goal consumes resources (e.g., energy) may have termination condition (e.g., walker at goal) ongoing, iterative action Certain words are closely associated with biological phenomena. Simple representation based on petri nets. Action/event representation has in common with motor control the need to refer to process states and transitions; and resource consumption/production; parameters. Part of learning/understanding words like “push”, “walk” clearly involves grounded knowledge about how to perform the action, as well as quite complex/concrete inferences based on execution. (e.g....) Note: these representations might be parameterized: “shove”, “walk slowly”, “walk home” and NOTE: this model of action is accurate cross-linguistically, even if some specific conditions on word meaning varies from language to language. This may seem complex, but in fact VERY early children seem to have no problem performing and understanding words like this. And more complicated ones too! energy walker=Harry goal=home

An Active Model of Events
Computationally, actions and events are coded in active representations called x-schemas which are extensions to Stochastic Petri nets. x-schemas are fine-grained action and event representations that can be used for monitoring and control as well as for inference.

Model Review: Stochastic Nets
Basic Mechanism 3 1 2 Resource arc [1] Precondition arc Inhibition arc [1] Firing function -- conjunctive -- logistic -- exponential family

Model Review Firing Semantics 3 1 2

Model Review Result of Firing 1 2

X-Schema Extensions to Petri Nets
Parameterization x-schemas take parameter values (speed, force) Walk(speed = slow, dest = store1) Dynamic Binding X-schemas allow run-time binding to different objects/entities Grasp(cup1), push(cart1) Hierarchical control and durative transitions Walk is composed of steps which are composed of stance and swing phases Stochasticity and Inhibition Uncertainties in world evolution and in action selection

Event Structure in Language
Fine-grained Rich Notion of Contingency Relationships. Phenomena: Aspect, Tense, Force-dynamics, Modals, Counterfactuals Event Structure Metaphor: Phenomena: Abstract Actions are conceptualized in Motion and Manipulation terms. Schematic Inferences are preserved.

Aspect Aspect is the name given to the ways languages describe the structure of events using a variety of lexical and grammatical devices. Viewpoints is walking, walk Phases of events Starting to walk, walking, finish walking Inherent Aspect run vs cough vs. rub Composition with Temporal modifiers, tense.. Noun Phrases (count vs. mass) etc..

Phases, Viewpoints, and Aspects
John is walking to the store. John is about to walk to the store. John walked to the store. John started walking to the store. John is starting to walk to the store. John has walked to the store. John has started to walk to the store. John is about to start walking to the store. John resumed walking to the store. John has been walking to the store. John has finished walking to the store. John almost walked to the store.

Aspectual Information in Questions
The US continues to be concerned about Russian arms sales to Syria. Has the US taken any steps to try to stop the sale of Russian arms to Syria? The US is concerned about Russian arms sales to Syria. Has the US taken any steps to try to detect the sale of Russian arms to Syria? The US is starting to be concerned about Russian arms. Is the US ready to take steps to try to establish the sale of Russian arms to Syria?

Aspectual Types

Aspects of (Climb) Iterate Ready Start Process Finish Done resume
interrupt Suspend Cancel BINDINGS Energy Ready Standing Hold Find hold Pull(self) Stabilize On top

About to + (Climb) (Prospective)
Iterate Ready Start Process Finish Done resume interrupt Suspend Cancel BINDINGS Energy Ready Standing Hold Find hold Pull(self) Stabilize On top

Be + (Climb)-ING (Progressive)
Iterate Ready Start Process Finish Done resume interrupt Suspend Cancel BINDINGS Energy Ready Standing Hold Find hold Pull(self) Stabilize On top

Have + (Climb)-ed (Perfect)
Ready Done Start Process Finish Suspend Cancel interrupt resume Iterate BINDINGS Energy Ready Standing Hold Find hold Pull(self) Stabilize On top

Phasal Aspect Maps to the Controller
Iterative (repeat) Inceptive (start, begin) Iterate Ready Start Process Finish Done interrupt resume Cancel Suspend Completive (finish, end) Resumptive(resume)

Embedding: About to start (X)
Ready Start Process Finish Done resume interrupt Suspend R S P F D r i S C X-Schema for X with bindings

Embedding: Has Started (to X)
Ready Start Process Finish Done resume interrupt Suspend R D S P F C i r X-Schema for X with bindings

Embedding: The end of the beginning
Ready Start Process Finish Done resume interrupt Suspend R S P F D r i S C X-Schema for X with bindings

Embedding: The beginning of the end
Ongoing Finish Done R S P F D r i S C X-Schema for X with bindings

Inherent Aspect Much richer than traditional Linguistic Characterizations (VDT (durative/atomic, telic/atelic)) Action patterns one-shot, repeated, periodic, punctual decomposition: concurrent, alternatives, sequential Goal based schema enabling/disabling Generic control features; interruption, suspension, resumption Resource usage

Other Transitions in the Controller may be coded
Lexical items may code interrupts Stumble is an interrupt to an ongoing walk A combination of grammatical and aktionsart may code of the controller phases Ready to walk : Prospective Resuming his run: Resumptive Has been running: Embedded progressive About to Finish the painting: Embedded Completive. Canceling the meeting vs. Aborting the meeting.

Simulation Semantics BASIC ASSUMPTION: SAME REPRESENTATION FOR PLANNING AND SIMULATIVE INFERENCE Evidence for common mechanisms for recognition and action (mirror neurons) in the F5 area (Rizzolatti et al (1996), Gallese 96, Buccino 2002, Tettamanti 2004) and from motor imagery (Jeannerod 1996) IMPLEMENTATION: x-schemas affect each other by enabling, disabling or modifying execution trajectories. Whenever the CONTROLLER schema makes a transition it may set, get, or modify state leading to triggering or modification of other x-schemas. State is completely distributed (a graph marking) over the network. RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!

A Precise Notion of Contingency Relations
Activation: Executing one schema causes the enabling, start or continued execution of another schema. Concurrent and sequential activation. Inhibition: Inhibitory links prevent execution of the inhibited x-schema by activating an inhibitory arc. The model distinguishes between concurrent and sequential inhibition, mutual inhibition and aperiodicity. Modification: The modifying x-schema results in control transition of the modified xschema. The execution of the modifying x-schema could result in the interruption, termination, resumption of the modified x-schema.

Summary of Aspect Results
Controller mediates between linguistic markings and individual event/verbal x-schemas (Cogsci99, Coling2002) Captures regular event structure; inspired by biological control theory Flexible: specific events may require only a subset of controller; interaction of underlying x-schemas, linguistic markers and hierarchical abstraction/ decomposition of controller accounts for wide range of aspectual phenomena. Important aspectual distinctions, both traditional and novel, can be precisely specified in terms of the interaction of x-schemas with the controller (CogSci97, CogSci 98, AAAI99, IJCAI99, CogSci04): stative/dynamic, durative/punctual: natural in x-schemas telic processes: depletion of resources continuous processes: consumption of resources temporary/effortful states; habituals dynamic interactions with tense, nominals, temporal modifiers incorporation of world knowledge, pragmatics

Event Structure in Language
Fine-grained Rich Notion of Contingency Relationships. Phenomena: Aspect, Tense, Force-dynamics, Modals, Counterfactuals Event Structure Metaphor: Phenomena: Abstract Actions are conceptualized in Motion and Manipulation terms. Schematic Inferences are preserved.

We use metaphors everyday
The council attacked every weak point of his proposal. I don't know how to put my thoughts into words. My summer plans are still up in the air. France slipped into a recession. Something smells fishy, but I can't quite put my finger on it. The economy hasn’t turned the corner, it has just made a U-turn and reversed course (Kerry, 8/10/04) metaphors aren't just in poetry and rhetoric there are a lot of metaphors in daily language

What is the basis for metaphors?
metaphor is understanding one thing in terms of another specifically, we reason about abstract concepts through our sensory-motor experience. that means we have: correlation inference

Metaphors, defined Formally, metaphors are mappings from a source domain to a target domain both the source and target domains are structured by schemas and frames Take a simple example: I've been feeling quite depressed of late. ( Happy is Up; Sad is Down )

SUBCASE OF Orientation
SCHEMA Happiness SUBCASE OF Emotion ROLES Degree SCHEMA Verticality SUBCASE OF Orientation Scale MAP HappyIsUpSadIsDown map-type <- METAPHOR tgt src PAIRS A number of other orientational metaphors: Happy is Up More is Up Control is Up Good is Up Conscious is Up Health is Up Rational is Up

Event Structure Metaphor
Maps motion and manipulation concepts to abstract actions. States are Locations Changes are Movements Causes are Forces Force Dynamic patterns of causation Actions are Self-propelled Movements Speed, step size are parameters that are mapped. Means are Paths crossroads Difficulties are Impediments to Motion Long-term, Purposeful Activities are Journeys Set out, back on track…

Metaphorical Inference
There is a conceptual metaphor, Understanding Is Grasping, according to which one can grasp ideas. One can begin to grasp an idea, but not quite get a hold of it. If you fail to grasp an idea, it can go right by you — or over your head! If you grasp it, you can turn it over in your mind. You can’t hold onto an idea before having grasped it. In short, reasoning patterns about physical grasping can be mapped by conceptual metaphor onto abstract reasoning patterns. Metaphors project the products of simulations to the target (abstract domain)!

Event Structure for semantically based QA
Reasoning about dynamics Complex event structure Multiple stages, interruptions, resources, framing Evolving events Conditional events, presuppositions. Nested temporal and aspectual references Past, future event references Metaphoric references Use of motion domain to describe complex events. Reasoning with Uncertainty Combining evidence from multiple, unreliable sources Non-monotonic inference Retracting previous assertions Conditioning on partial evidence Linguistic Ambiguity Figurative inference

Uncertainty and domain representation
Factorized Representation of Domain uses Dynamic Belief Nets (DBN’s) Probabilistic Semantics Structured Representation

What is a Bayes (belief) net?
Compact representation of joint probability distributions via conditional independence Family of Alarm Earthquake Radio Burglary Alarm Call 0.9 0.1 e b 0.2 0.8 0.01 0.99 B E P(A | E,B) Qualitative part: Directed acyclic graph (DAG) Nodes - random vars. Edges - direct influence Quantitative part: Set of conditional probability distributions Together: Define a unique distribution in a factored form Figure from N. Friedman

What is a Bayes net? A node is conditionally independent of its
ancestors given its parents, e.g. Earthquake Radio Burglary Alarm Call C || R,B,E | A Hence From 25 – 1 = 31 parameters to =10

Why are Bayes nets useful?
- Graph structure supports - Modular representation of knowledge - Local, distributed algorithms for inference and learning - Intuitive (possibly causal) interpretation - Factored representation may have exponentially fewer parameters than full joint P(X1,…,Xn) => lower sample complexity (less data for learning) lower time complexity (less time for inference)

What can Bayes nets be used for?
Posterior probabilities Probability of any event given any evidence Most likely explanation Scenario that explains evidence Rational decision making Maximize expected utility Value of Information Effect of intervention Causal analysis Explaining away effect Earthquake Radio Burglary Alarm Call Radio Call Figure from N. Friedman

Dynamic Bayes Nets Dynamic Bayesian Networks (D(T)BNs) are an extension of Bayesian networks for modeling dynamic systems. state at time t is represented by a set of random variables. The state at time t is dependent on the states at previous time steps. first-order Markovian, and thus we need to represent the transition distribution P(Zt+1 | Zt). This can be done using a two-time-slice Bayesian network fragment (2-TBN) Bt+1, variables from Zt+1 whose parents are variables from Zt and/or Zt+1, and variables from Zt without any parents. Typically, we also assume that the process is stationary,

A Simple DBN for the Economics Domain
Economic State [recession,nogrowth,lowgrowth,higrowth] Policy [Liberalization, Protectionism] Goal [Free Trade, Protection ] Outcome [Success, failure] Difficulty [present, absent] T0 T1

Probabilistic inference
Filtering P(X_t | o_1…t,X_1…t) Update the state based on the observation sequence and state set MAP Estimation Argmaxh1…hnP(X_t | o_1…t, X_1…t) Return the best assignment of values to the hypothesis variables given the observation and states Smoothing P(X_t-k | o_1…t, X_1…t) modify assumptions about previous states, given observation sequence and state set Projection/Prediction/Reachability P(X_t+k | o_1..t, X_1..t)

Answer Type to Inference Method
DESCRIPTION Justify (Proposition) MAP Proposition is part of the MAP Ability (Agent, Act) Filtering; Smoothing Past/Current Action enabled given current state Prediction (State) P;R’ MAP Propogate current information and estimate best new state Hypothetical (Condition) S, R_I Smooth intervene and compute state

Logical Action Theories
Connection to ARD (or other Action Languages): The representation can be used to encode a causal model for a domain description D (in the Syntax of ARD) in that it satisfies all the causal laws in D. Furthermore, a value proposition of the form C after A is entailed by D iff all the terms in C are in Si; the state that results after running the projection algorithm on the action set A. (IJCAI 99) Executing representation, frame axioms are encoded in the topology of the network and transition firing rules respect them. Planning as backward reachability or computing downward closure (IJCAI 99, WWW2002) Links to linear logic. Perhaps a model of stochastic linear logic? (joint work with Jose Meseguer).

Outline Part III. Knowledge representation and inference
KR requirements Dynamic Context and the structure of Events Uncertainty in knowledge sources Representing Metaphors and Maps Active Knowledge and Event Structure X-schemas A computational model of Aspect Bayesian Models of Uncertainty Graphical Models Dynamic Bayes Nets for Modeling State A Computational Model of Metaphor Interpretation DEMO

Task: Interpret simple discourse fragments/ blurbs
France fell into recession. Pulled out by Germany US Economy on the verge of falling back into recession after moving forward on an anemic recovery. Indian Government stumbling in implementing Liberalization plan. Moving forward on all fronts, we are going to be ongoing and relentless as we tighten the net of justice. The Government is taking bold new steps. We are loosening the stranglehold on business, slashing tariffs and removing obstacles to international trade.

Basic Result An implemented computational model that is
Neurally plausible (Feldman and Narayanan 2004) Establishes that motion, manipulation and spatial concepts are used to convey important and subtle information about abstract domains such as International Economics. In 1991, India set out on a path of Liberalization. After making rapid strides in the first few years, the Government policy hit a first of a series of roadblocks in By 1998, the new BJP Government had reoriented the Government’s policy ..

I/O as Feature Structures
Indian Government stumbling in implementing liberalization plan

Basic Primitives An fine-grained executing model of action and events (X-schemas). A factorized representation of probabilistic domain knowledge (DBN’s) A controller X-schema that models inter-event relations and forms the basis of inference through simulation. A model of metaphor maps that project bindings from embodied to application domains.

KARMA DEMO SOURCE DOMAINS: MOTION, HEALTH
TARGET DOMAINS: INTERNATONAL ECONOMICS METAPHOR MAPS: EVENT STRUCTURE METAPHOR

Results Model was implemented and tested on discourse fragments from a database of 50 newspaper stories in international economics from standard sources such as WSJ, NYT, and the Economist. Results show that motion terms are often the most effective method to provide the following types of information about abstract plans and actions. Information about uncertain events and dynamic changes in goals and resources. (sluggish, fall, off-track, no steam) Information about evaluations of policies and economic actors and communicative intent (strangle-hold, bleed). Communicating complex, context-sensitive and dynamic economic scenarios (stumble, slide, slippery slope). Commincating complex event structure and aspectual information (on the verge of, sidestep, giant leap, small steps, ready, set out, back on track). ALL THESE BINDINGS RESULT FROM REFLEX, AUTOMATIC INFERENCES PROVIDED BY X-SCHEMA BASED INFERENCES.

Scaling Up Scaling Up Language Scaling Up Inference
Embodied Construction Grammar FrameNet for Scenarios Scaling Up Inference Coordinated Probabilistic Relational Model (CPRM) An Application to QA inference Scaling Up Ontological Knowledge Semantic Web OWL and OWL-S An Event Ontology in OWL

Outline Part IV. Scaling Up Scaling Up Language
Embodied Construction Grammar FrameNet for Scenarios Scaling Up Ontological Knowledge Semantic Web OWL and OWL-S An Event Ontology in OWL Scaling Up Inference Coordinated Probabilistic Relational Model (CPRM)

Embodied Construction Grammar
Embodied representations active perceptual and motor schemas (image schemas, x-schemas, frames, etc.) situational and discourse context Construction Grammar Linguistic units relate form and meaning/function. Both constituency and (lexical) dependencies allowed. Constraint-based based on feature unification (as in LFG, HPSG) Diverse factors can flexibly interact. common schema representations for concepts and constructional meaning dynamic, cognitively motivated representations (image schemas, x-schemas, frames, etc.)

A shared theory and formalism for different cognitive mechanisms
Embodied Construction Grammar provides formal tools for linguistic description and analysis motivated largely by cognitive/functional concerns. A shared theory and formalism for different cognitive mechanisms Constructions, metaphor, mental spaces, etc. Precise specifications of structures/processes involved in language understanding Bridge to detailed simulative inference using embodied representations

Pragmatics Semantics Syntax Morphology Phonology Phonetics
“Harry walked into the cafe.” Phonology Semantics Pragmatics Morphology Syntax Phonetics

UTTERANCE Pragmatics Semantics Syntax Morphology Phonology Phonetics
“Harry walked into the cafe.” UTTERANCE Phonology Semantics Pragmatics Morphology Syntax Phonetics

ECG Structures Schemas Constructions Maps Spaces
image schemas, force-dynamic schemas, executing schemas, frames… Constructions lexical, grammatical, morphological, gestural… Maps metaphor, metonymy, mental space maps… Spaces discourse, hypothetical, counterfactual…

Image schemas Trajector / Landmark (asymmetric)
LM Trajector / Landmark (asymmetric) The bike is near the house ? The house is near the bike Boundary / Bounded Region a bounded region has a closed boundary Topological Relations Separation, Contact, Overlap, Inclusion, Surround Orientation Vertical (up/down), Horizontal (left/right, front/back) Absolute (E, S, W, N) TR bounded region boundary

Embodied schemas These are abstractions over sensorimotor experiences.
schema name schema Source-Path-Goal roles source path goal trajector schema Container roles interior exterior portal boundary role name Boundary Interior Trajector Portal Source Goal Path Exterior These are abstractions over sensorimotor experiences.

Embodied constructions
ECG Notation Form Meaning construction HARRY form : /hEriy/ meaning : Harry Harry CAFE construction CAFE form : /khaefej/ meaning : Cafe cafe Constructions have form and meaning poles that are subject to type constraints.

Representing constructions: TO
construction TO form selff.phon  /thuw/ meaning evokes Trajector-Landmark as tl Source-Path-Goal as spg constraints: tl.trajector « spg.trajector tl.landmark « spg.goal local alias identification constraint The meaning pole may evoke schemas (e.g., image schemas) with a local alias. The meaning pole may include constraints on the schemas (e.g., identification constraints «).

The INTO construction construction INTO form meaning evokes
selff.phon  /Inthuw/ meaning evokes Trajector-Landmark as tl Source-Path-Goal as spg Container as cont constraints: tl.trajector « spg.trajector tl.landmark « cont cont.interior « spg.goal cont.exterior « spg.source TO vs. INTO: INTO adds a Container schema and appropriate bindings.

Constructions with constituents: The SPATIAL-PHRASE construction
construction SPATIAL-PHRASE constructional constituents sp : Trajector-Landmark lm : Thing form spf before lmf meaning spm.landmark « lmm local alias order constraint identification constraint Constructions may also specify constructional constituents and impose form and meaning constraints on them: order constraints identification constraints

An argument structure construction
construction DIRECTED-MOTION subcase of Pred-Expr constructional constituents a : Ref-Exp m: Pred-Exp p : Spatial-Phrase form af before mf mf before pf meaning evokes Directed-Motion as dm selfm.scene « dm dm.agent « am dm.motion « mm dm.path « pm schema Directed-Motion roles agent : Entity motion : Motion path : SPG You saw Directed-Motion last time (“Harry walked into the café”). This one is a basic transitive construction. OLD: I won’t go over this, but you can just take my word that this conveys the same information in a more textual form.

Simulation-based language understanding
Belief State General Knowledge Constructions construction WALKED form selff.phon  [wakt] meaning : Walk-Action constraints selfm.time before Context.speech-time selfm..aspect  encapsulated “Harry walked into the cafe.” Utterance Analysis Process Semantic Specification Similar to what we’re doing today (not coincidentally). Utterances evoke a complex network of conceptual schemas that are simulated in context to produce a rich set of inferences. BUT: entire process draws on many different kinds of data: constructions (lexical and grammatical, etc.), ontology, context. (We’re focusing on just a part of it today.) CAFE Simulation

The ECG Analyzer Goes bottom up, generating a semantic interpretation as well as syntactic constituency Uses a chart to store the constituent matches Instead of using FSMs, each construction is turned into a construction recognizer

Using a Chart A chart is a data structure that tracks the constituents found during an analysis. Note that the chart is storing semantic information as well as grammatical. After analysis, the chart is filled with all the semantic chunks (constituents) that were found.

Construction Recognizer
Each construction is turned into a parameterized procedure. The procedure checks form and semantic constraints simultaneously. Constructions = active knowledge

ECG - Clausal Example Construction Active-Caused-Motion constituents
subj : RefExp[Agent] verb : Verb[Force-Application] DO : RefExp path : PP[SPG] form subj before verb verb before DO verb before path meaning:caused-motion-scene selfm.agent <-> subjm selfm.action <-> verbm selfm.patient <-> DOm selfm.path <-> pathm

Chunking L3 ________________________S_____________S
L2 ____NP _________PP VP NP ______VP L1 ____NP P_______NP VP NP ______VP L0 D N P D N N V-tns Pron Aux V-ing the woman in the lab coat thought you were sleeping After Abney, 1996.

Construction Recognizers
Form Meaning D,N <-> [Cloth num:sg] Form Meaning PP$,N <-> [Hand num:sg poss:addr] Form Meaning “you”<->[Addressee] NP NP NP NP NP You want to put a cloth on your hand ? Like Abney: Unlike Abney: One recognizer per rule Bottom up and level-based Check form and semantics More powerful/slower than FSMs

Chunk Chart Interface between chunking and structure merging
Each edge is linked to its corresponding semantics. You want to put a cloth on your hand ?

Combining Partial Parses
Prefer an analysis that spans the input utterance with the minimum number of chunks. When no spanning analysis exists, however, we still have a chart full of semantic chunks. The system tries to build a coherent analysis out of these semantics chunks. This is where structure merging comes in.

Structure Merging Closely related to abductive inferential mechanisms like Hobbs’ Unify compatible structures (find fillers for frame roles) Intuition: Unify structures that would have been co-indexed had the missing construction been defined. There are many possible ways to merge structures. In fact, there are an exponential number of ways to merge structures (NP Hard). But using heuristics cuts down the search space.

Structure Merging Example
Utterance:It is too heavy for you to pour. Before Merging: It < Entity num:sg … After Merging: Caused-Motion-Action agent: patient: (1) path: [Addressee < Anim] [Addressee ] It num:sg … Caused-Motion-Action agent: [Animate] patient: (1) [Entity] path: TrajectorLandmark trajector : (1) landmark : TrajectorLandmark trajector : (1) landmark : [Entity]

Semantic Density Semantic density is a simple heuristic to choose between competing analyses. Density of an analysis = (filled roles) / (total roles) The system prefers higher density analyses because a higher density suggests that more frame roles are filled in than in competing analyses. Extremely simple / useful? but it certainly can be improved upon.

Measuring Semantic Density
The frame on the left has a density of .75 while the frame on the right has a density of 1.

Semantic Density At Work
You [Addressee] Caused-Motion-Action agent : [Animate] patient : [Entity] path:[TrajLM] Motion-Action mover : path:[TrajLM] or move TrajectorLandmark trajector : landmark : here Assume that “You move over here” isn’t recognized and that “move” has two senses. over here

Semantic Density At Work 2
Motion-Action agent: (1) [Addressee] path: Caused-Motion-Action agent: (1) [Addressee] patient: /**unfilled**/ path: TrajectorLandmark trajector : (1) landmark : here TrajectorLandmark trajector : (1) landmark : here There are 2 resulting merged analyses. The Motion-Action frame has a density of 3/3 = 1 and the Caused-Motion-Action frame has a density of 3/4 = .75 because of the missing patient.

Language Understanding Process

Simulation specification
It should be clear that the simulation specification includes exactly the schematic content of the different elements of the sentence, bound appropriately. As noted earlier, the two representations differ with respect to which image schemas are involved – as reflected by the additional CONTAINER schema in Figure 5b – and in the precise bindings of aspects of the cafe to the SPG schema. Like the image schema representations, the simulation specifications can be viewed as a summary of the much more complex structures that are active when an event is simulated or imagined. Activating these structures – that is, “running” the simulation – can thus provide the much richer basis for inference necessary for accounting for many linguistic phenomena A simulation specification consists of: schemas evoked by constructions bindings between schemas

Summary: ECG Linguistic constructions are tied to a model of simulated action and perception Embedded in a theory of language processing Constrains theory to be usable Frees structures to be just structures, used in processing Precise, computationally usable formalism Practical computational applications, like MT and NLU Testing of functionality, e.g. language learning A shared theory and formalism for different cognitive mechanisms Constructions, metaphor, mental spaces, etc.

ECG applications Grammar Semantic representations / inference
Spatial relations/events (Bergen & Chang 1999; Bretones et al. In press) Verbal morphology (Gurevich 2003, Bergen ms.) Reference: measure phrases (Dodge and Wright 2002), construal resolution (Porzel & Bryant 2003), reflexive pronouns (Sanders 2003) Semantic representations / inference Aspectual inference (Narayanan 1997; Chang, Gildea & Narayanan 1998) Perspective / frames (Chang, Narayanan & Petruck 2002) Metaphorical inference (Narayanan 1997, 1999) Simulation semantics (Narayanan 1997, 1999) Language acquisition Lexical acquisition (Regier 1996, Bailey 1997) Multi-word constructions (Chang 2004; Chang & Maia 2001)

FrameNet for AQUAINT Scenarios
FrameNet is annotating the AQUAINT CNS data with frame information Design and build FN database of Frames and Schemas relavant to the AQUAINT data Select a set of representative documents and annotate them with FrameNet frames. Gold Standard annotations for the program. All the FrameNet data (DB and annotations) will be released in RDF/OWL in addition to the XML format.

Example Document: Country Profile- Libya
Frame Type Description Number Action Verbs of action often hostile action 50 Resources Resources, assets, capabilities 5 Attitude Different attitude frames 15 Weaponry Different types and descriptions 75 Treaty Different treaties named 12 Organization Organizations named 19

FrameNet annotation of CNS data

Structured Probabilistic Inference

Probabilistic inference
Filtering P(X_t | o_1…t,X_1…t) Update the state based on the observation sequence and state set MAP Estimation Argmaxh1…hnP(X_t | o_1…t, X_1…t) Return the best assignment of values to the hypothesis variables given the observation and states Smoothing P(X_t-k | o_1…t, X_1…t) modify assumptions about previous states, given observation sequence and state set Projection/Prediction/Reachability P(X_t+k | o_1..t, X_1..t)

Relational Models Relational models make some ontological commitments: the world consists of objects, and relations over them PRMs are based on a particular “relational logic” borrowed from databases: Databases Relational Logic Table Class Tuple Object Standard Field Descriptive Attribute Foreign Key Field Reference Slot

PRM Introduction PRMs allow objects to be augmented with a description of relations between instances. The object relational structure can be a relational database or logic program. The PRM model augments the database with probabilities. The model is useful in cases where the configuration of a system (instances) change while the relational schema remains constant.

Probabilistic Relational Models
A PRM for a Relational Schema S is defined as: For each Class C and propositional Attribute A(C) in C we have A set of parents of A (simple, complex or aggregate) A CPD P(C.A | Pa(C.A))

Inference With PRMs O(Nkbk(m+2)bq) (Pfeffer 2000)
SVE inference for a PRM P with q query variables and N attributes is O(Nkbk(m+2)bq) (Pfeffer 2000) k is the maximum number of interface variables q is the number of query variables m is the maximum tree width for any object in P (related to the markov blanket).

Controlling PRM inference
The number of interface variables, k, is related to the number of relations that a variable participates in as well as the number of slot chains that the variable participates in Careful selection of relations (only part-of) can make inference tractable. The tree width m depends on the markov blanket of an attribute. Control of network topology can reduce this.

Adding time to PRM D(T)PRM
A two-time-slice PRM (2TPRM) for a relational schema S is defined as follows. For each class C and each propositional attribute A 2 A(C), we have: A set of parents Pa(C:A) = f{Pa1; Pa2; .. Pal}, where each Pai has the form C:B or f(C.rho.B), where rho is a slot chain containing the attribute previous at most once, and f() is an aggregation function. A conditional probability model for P(C.A | Pa(C.A))

Adding Time to PRM’s Since time is another relation, doesn’t increase expressive power. Significant impact of inference tractability since both k and m may become quite large. New Algorithm: Exploit the structure of time using the interface and frontier algorithm (Murphy 2002). Variables at slice t with links to variables at t+1 form the interface Interface variables d-separate the past (< t) from the future slices (> t). Allows for on-line inference algorithms similar to inside-outside algorithm for SCFG’s. Use SVE to rollover slices. Approximation using Rao-Blackwelized particle filtering (Sanghai, Domingos, and Weld 2003)

Structured Probabilistic Inference

CPRM inference Combines insights from
the SVE algorithm for PRMs (Pfeffer 2000) the frontier algorithms for temporal models (Murphy 2002) and the BN SCFG algorithm (Narayanan 99) Inference algorithms for complex, coordinated events (Narayanan 2002) Expressive Probabilistic Modeling paradigm with relations and branching dynamics. Offers principled methods to bound inferential complexity.

Temporal Projection in CPRM

Model Parameterization
Retrieved Documents Predicate Extraction C O N T E X PRM <Pred(args), Topic Model, Answer Type> Model Parameterization FrameNet Frames <Simulation Triggering > Event Simulation OWL/OWL-S Topic Ontologies < PRM Update>

AnswerBank AnswerBank is a collection of over a 1200 QA annotations from the AQUAINT CNS corpus. Questions and answers cover the different domains of the CNS data. Questions and answers are POS tagged, and syntactically parsed. Question and Answer predicates are annotated with PropBank arguments and FrameNet (when available) tags. FrameNet is annotating CNS data with frame information for use by the AQUAINT QA community. We are planning to add more semantic information including temporal, aspectual information (TIMEML+) and information about event relations and figurative uses.

Answer Types for complex questions in AnswerBank
EXAMPLE NUMBER Justify (Proposition) What is the evidence that IRAQ has WMD? 89 Ability (Agent, Act) How can a Biological Weapons Program be detected? 71 Prediction (State) What were the possible ramifications of India’s launch of the Prithvi missile? 63 Hypothetical (Condition) If Musharraf is removed from power, will Pakistan become a militant Islamic State? 62

Building Models Gold Standard: Semantic Web based:
From the hand-annotated data in the CNS corpus, we manually built CPRM domain models for inference. Semantic Web based: From FrameNet frames and from semantic web ontologies in OWL (SUMO-based, OpenCYC and others), we built CPRM models (semi-automatic)

Event Structure Inferences
For the annotations we classify complex event structure inferences as Aspectual Stages of events, viewpoints, temporal relations (such as start(ev1, ev2), interrupt(ev1, ev2)) Action-Based Resources (produce,consume,lock), preconditions, maintenance conditions, effects. Metaphoric Event Structure Metaphor (ESM) Events and predications (motion => Action), objects (Motion.Mover => Action.Actor), Parameters(Motion.speed =>Action.rateOfProgress)

Content of Inferences Component Number F-Score Manual OWL Aspectual
375 .74 .65 Action-Feature 459 .62 .45 Metaphor 149 .70

Semantic Web The World Wide Web (WWW) contains a large and expanding information base. HTML is accessible to humans but does not formally describe data in a machine interpretable form. XML remedies this by allowing for the use of tags to describe data (ex. disambiguating crawl) Ontologies are useful to describe objects and their inter-relationships. DAML+OIL ( is an markup language based on XML and RDF that is grounded in description logic and is designed to allow for ontology development, transfer, and inference on the web.

Programmatic Access to the web
Web-accessible programs and devices

Knowledge Rep’n for the “Semantic Web”
OWL/DAML-L (Logic) OWL (Ontology) XML Schema RDF (Resource Description Framework) RDFS (RDF Schema) XML (Extensible Markup Language)

Knowledge Rep’n for “Semantic Web Services”
DAML-L (Logic) DAML+OIL (Ontology) DAML-S (Services) XML Schema RDF (Resource Description Framework) RDFS (RDF Schema) XML (Extensible Markup Language)

The OWL Language OWL REF

DAML-S: Semantic Markup for Web Services
DAML-S: A DARPA Agent Markup Language for Services DAML+OIL ontology for Web services: well-defined semantics ontologies support reuse, mapping, succinct markup, ... Developed by a coalition of researchers from Stanford, SRI, CMU, BBN, and Nokia, Yale, under the auspices of DARPA. DAML-S version 0.6 posted October,2001 [DAML-S Coalition, 2001, 2002] [Narayanan & McIlraith 2003]

DAML-S/OWL-S Compositional Primitives
inputs (conditional) outputs preconditions (conditional) effects process atomic process composite process control constructs composedBy while sequence If-then-else fork ...

The OWL-S Process Description
PROCESS.OWL

Atomic Processes as X-schemas
SC Knowledge Preconditions ^ Kref Input Possk(a,s) Action a Effect .

DAML-S Processes as X-schemas
…and SC World Preconditions ^ Kref Input Possk(a,s) Action a Effect . p1 p2 pn World Possw(a,s)

Composite Process Constructs
Control Construct Description Sequence Execute a list of processes in sequence. Concurrent Fork off a bag of processes. Concurrent-Sync Fork off a bag of processes & synchronize/join after completion Choice Choose from a bag of processes. Iterate (condition) Repeat process until <cond> (while <cond>). If-Then-Else If <cond> then THEN else ELSE.

Modeling Composite Process Constructs
start finish Component Control Construct Ready Done

DAML-S Sequence: P1;P2 start finish Done(P1;P2) Atomic Process P2
Ready

DAML-S Sequence START FINISH Atomic Process P1 Process P2 DONE(P1)
DONE(P1;P2) READY

DAML-S Fork: P1|| P2 start finish Done(P1 || P2) Atomic Process P1
Ready(P1) Ready(P2) Atomic Process P2

DAML-S Fork START FINISH P2 DONE(P1|| P2) READY (P1) P1 READY (P2)

DAML-S Concurrent-Sync
Done(P2) Done(P1) start finish Atomic Process P2 Ready(P1) P1 Ready(P2)

DAML-S Concurrent-Sync
FINISH START READY (P1) P1 DONE(P1) READY (P2) DONE(P2) P2

Implementation Input: DAML-S description of Events
DAML-S translation to the modeling environment KarmaSIM [Narayanan, 97] ( Basic Program: Input: DAML-S description of Events Output: Network Description of Events in KarmaSIM Procedure: Recursively construct a sub-network for each control construct. Bottom out at atomic event. Construct a net for each atomic event Return network

Example of A WMD Ontology in OWL
<rdfs:Class rdf:ID="DevelopingWeaponOfMassDestruction"> <rdfs:subClassOf rdf:resource= SUMO.owl#Making"/> <rdfs:comment> Making instances of WeaponOfMassDestruction. </rdfs:comment> </rdfs:Class>

FrameNet in OWL Program FNtoOWL Implemented in Java Uses the JENA API
Given an XML FrameNet database File in the FN XML format (schema and dtd) The Location (file, URL) of the resulting OWL markup A URL for the FrameNet ontology file Output An OWL file for the FrameNet database at the specified location. URL:

FrameNet in OWL

Conclusion Answering complex questions requires semantic representations at multiple levels. NE and Extraction-based Predicate Argument Structures Frame, Topic and Domain Models All these representations should be capable of supporting inference about relational structures, uncertain information, and dynamic context. Both Semantic Extraction techniques and Structured Probabilistic KR and Inference methods have matured to the point that we understand the various algorithms and their properties. Flexible architectures that embody these KR and inference techniques and make use of the expanding linguistic and ontological resources (such as on the Semantic Web) Point the way to the future of semantically based QA systems!

References (URL) Semantic Resources Probabilistic KR (PRM)
FrameNet: (Papers on FrameNet and Computational Modeling efforts using FrameNet can be found here). PropBank: Gildea’s Verb Index; (links FrameNet, PropBank, and VerbNet Probabilistic KR (PRM) (Learning PRM) (Avi Pfeffer’s PRM Stanford thesis) David Poole IJCAI 2003 Dan Roth NIPS 2004 (submitted) Dynamic Bayes Nets (Kevin Murphy’s Berkeley DBN thesis) Weld et all 2003, DPRM Event Structure in Language (Narayanan’s Berkeley PhD thesis on models of metaphor and aspect) ftp://ftp.cis.upenn.edu/pub/steedman/temporality/temporality.ps.gz (Steedman’s article on Temporality with links to previous work on aspect) (publications on Cognitive Linguistics and computational models of cognitive linguistic phenomena can be found here)

References (URL) Semantic Web OWL OWL-S Scientific American Article
OWL OWL-S

Representing Event Frames
At the computational level, we use a structured event representation of event frames that formally specify The frame Frame Elements and filler types Constraints and role bindings Frame-to-Frame relations Subcase Subevent

Events and actions schema Event roles before : Phase
transition : Phase after : Phase nucleus constraints transition :: nucleus before after transition nucleus schema Action evokes Event as e roles actor : Entity undergoer : Entity self « e.nucleus undergoer actor

The Commercial-Transaction schema
schema Commercial-Transaction subcase of Exchange roles customer « participant1 vendor « participant2 money « entity1 : Money goods « entity2 goods-transfer « transfer1 money-transfer « transfer2

Implementation Input: DAML-S description of Frame relations + Ontology
DAML-S translation to the modeling environment KarmaSIM [Narayanan, 97] ( Basic Program: Input: DAML-S description of Frame relations + Ontology Output: Network Description of Frames in KarmaSIM Procedure: Recursively construct a sub-network for each control construct. Bottom out at atomic frame. Construct a net for each atomic frame Return network

Outline Part V. From Ontologies to Inference
From OWL to CPRM FrameNet in OWL FrameNet to CPRM mapping Part VI. A Pilot QA System Implementation AnswerBank examples Current results for Inference Type Current results for Answer Structure

AnswerBank AnswerBank is a collection of over a 1200 QA annotations from the AQUAINT CNS corpus. Questions and answers cover the different domains of the CNS data. Questions and answers are POS tagged, and syntactically parsed. Question and Answer predicates are annotated with PropBank arguments and FrameNet (when available) tags. FrameNet is annotating CNS data with frame information for use by the AQUAINT QA community. We are planning to add more semantic information including temporal, aspectual information (TIMEML+) and information about event relations and figurative uses.

Model Parameterization
Retrieved Documents Predicate Extraction C O N T E X PRM <Pred(args), Topic Model, Answer Type> Model Parameterization FrameNet Frames <Simulation Triggering > Event Simulation OWL/OWL-S Topic Ontologies < PRM Update>

Answer Types for complex questions in AnswerBank
EXAMPLE NUMBER Justify (Proposition) What is the evidence that IRAQ has WMD? 89 Ability (Agent, Act) How can a Biological Weapons Program be detected? 71 Prediction (State) What were the possible ramifications of India’s launch of the Prithvi missile? 63 Hypothetical (Condition) If Musharraf is removed from power, will Pakistan become a militant Islamic State? 62

Conclusion Answering complex questions requires semantic representations at multiple levels. NE and Extraction-based Predicate Argument Structures Frame, Topic and Domain Models All these representations should be capable of supporting inference about relational structures, uncertain information, and dynamic context. Both Semantic Extraction techniques and Structured Probabilistic KR and Inference methods have matured to the point that we understand the various algorithms and their properties. Flexible architectures that embody these KR and inference techniques and make use of the expanding linguistic and ontological resources (such as on the Semantic Web) Point the way to the future of semantically based QA systems!

References (URL) Semantic Resources Probabilistic KR (PRM)
FrameNet: (Papers on FrameNet and Computational Modeling efforts using FrameNet can be found here). PropBank: Gildea’s Verb Index; (links FrameNet, PropBank, and VerbNet Probabilistic KR (PRM) (Learning PRM) (Avi Pfeffer’s PRM Stanford thesis) David Poole IJCAI 2003 Dan Roth NIPS 2004 (submitted) Dynamic Bayes Nets (Kevin Murphy’s Berkeley DBN thesis) Weld et all 2003, DPRM Event Structure in Language (Narayanan’s Berkeley PhD thesis on models of metaphor and aspect) ftp://ftp.cis.upenn.edu/pub/steedman/temporality/temporality.ps.gz (Steedman’s article on Temporality with links to previous work on aspect) (publications on Cognitive Linguistics and computational models of cognitive linguistic phenomena can be found here)

References (URL) Semantic Web OWL OWL-S Scientific American Article
OWL OWL-S

Structured Probabilistic Inference in an Embodied Construction Grammar

Similar presentations

Presentation on theme: "Structured Probabilistic Inference in an Embodied Construction Grammar"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Structured Probabilistic Inference in an Embodied Construction Grammar

Similar presentations

Presentation on theme: "Structured Probabilistic Inference in an Embodied Construction Grammar"— Presentation transcript:

Similar presentations

About project

Feedback