1. Bailey Model Showed how simple hand action verbs may be acquired based on motor control schemas and parameterization. Used Model Merging which allowed for –One-shot learning (Maps to recruitment learning) Could label and perform actions (given a command, world state pair) Uses parameters over motor-control schemas as inductive bias Limitations –Inference Connections between events –Abstract uses Event Structure Frames Metaphor

2. Event Structure-1 Srini Narayanan CS182/CogSci110/Ling109 Spring 2006 snarayan@icsi.berkeley.edu

3. Active representations Many inferences about actions derive from what we know about executing them Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions Walking: bound to a specific walker with a direction or goal consumes resources (e.g., energy) may have termination condition (e.g., walker at goal ) ongoing, iterative action walker =Harry goal =home energy walker at goal

4. X-Schema Extensions to Petri Nets Parameterization –x-schemas take parameter values (speed, force) Walk(speed = slow, dest = store1) Dynamic Binding –X-schemas allow run-time binding to different objects/entities Grasp(cup1), push(cart1) Hierarchical control and durative transitions –Walk is composed of steps which are composed of stance and swing phases Stochasticity and Inhibition –Uncertainties in world evolution and in action selection

5. Event Structure in Language Commonplace discourse fragments/blurbs –Low inflation is starting to pull France out of recession. –E3 continue to push Iran to uphold IAEA obligations. –US Economy on the verge of falling back into recession after moving forward on an anemic recovery. –Indian Government stumbling in implementing Liberalization plan. –Moving forward on all fronts, we are going to be ongoing and relentless as we tighten the net of justice. –The Government is taking bold new steps. We are loosening the stranglehold on business, slashing tariffs and removing obstacles to international trade.

6. Event Structure in Language Fine-grained Rich Notion of Contingency Relationships. –Phenomena: Aspect, Tense, Force-dynamics, Modals, Counterfactuals Event Structure Metaphor: –Phenomena: Abstract Actions are conceptualized in Motion and Manipulation terms. –Schematic Inferences are preserved.

7. Aspect Aspect is the name given to the ways languages describe the structure of events using a variety of lexical and grammatical devices. –Viewpoints is walking, walk –Phases of events Starting to walk, walking, finish walking –Inherent Aspect run vs cough vs. rub –Composition with Temporal modifiers, tense.. Noun Phrases (count vs. mass) etc..

8. Grammatical Aspect Languages have grammatical constructions that indicate the type of situation described. Progressive: She was running home. Perfect: I’ve had a wonderful evening. Inceptive: She started knitting. Prospective: She’s about to leave. Resumptive: Peace talks resume. Iterative: They ran twice around the track.

9. Phases, Viewpoints, and Aspects John is walking to the store. John is about to walk to the store. John walked to the store. John started walking to the store. John is starting to walk to the store. John has walked to the store. John has started to walk to the store. John is about to start walking to the store. John resumed walking to the store. John has been walking to the store. John has finished walking to the store. John almost walked to the store.

10. A Walk X-schema

11. A Climb X-schema

12. Common Patterns Posture = Up Energy Available Ready Dest = top(obj) Loop BEGIN Execute (subschema) END At Dest Done Posture = Up Ground ok Ready Loop BEGIN Execute(subschema) END At Dest Done STARTFINISH

13. Pre-motor Versus Motor Cortex Whenever we perform a complex motor movement, such as picking up a glass and taking a drink, at least two distinct parts of the brain are activated: The motor cortex, where there are neural ensembles that control “motor synergies” — relatively simple actions like opening or closing the hand, flexing or extending the elbow, turning the wrist, and so on. Complex motor schemas, however, are carried out by neural circuitry in the pre-motor cortex, circuitry connected via neural bindings to the appropriate synergies in the motor cortex. In picking up a glass and taking a drink, both pre-motor cortex and motor cortex are activated, as are binding connections between them.

14. The Controller X-Schema In modeling complex premotor action schemas, we make the following hypothesis All complex premotor schemas are compositions of a single type of structure. The same neural computational structure, when disengaged from the motor cortex, can characterize aspect (that is, event structure) in the world’s languages. When dynamically active, this structure can compute the logic of aspect. We call this structure the “Controller X-schema.”

15. The Structure of the Controller X-Schema Initial State Starting Phase Transition Precentral State Central Phase Transition (either instantaneous, prolonged, or ongoing) Postcentral State* Ending Phase Transition Final State Postcentral Options: *A check to see if a goal state has been achieved *An option to stop/resume *An option to iterate or continue the main process -Narayanan, 1997

17. A Schema Controller An active controller that sends signals to the embedded schema and transitions based on signals from the embedded schema. Useful for higher level monitoring and coordination of actions. Ready DoneStartProcessFinish Suspend Cancel interrupt resume iterate

18. A Generic Process Schema Part of Conceptual Structure. Generalizes over actions and events. Has internal state and models evolution of processes. Ready DoneStartProcessFinish Suspend Cancel interrupt resume iterate

19. Aspects of (Climb) Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate Energy Ready Standing On top Hold Find hold Pull(self) Stabilize BINDINGS

20. About to + (Climb) (Prospective) Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate Energy Ready Standing On top Hold Find hold Pull(self) Stabilize BINDINGS

21. Cancel + (Climb) Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate Energy Ready Standing On top Hold Find hold Pull(self) Stabilize BINDINGS

22. Start + (Climb)-ING Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate Energy Ready Standing On top Hold Find hold Pull(self) Stabilize BINDINGS

23. Be + (Climb)-ING (Progressive) Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate Energy Ready Standing On top Hold Find hold Pull(self) Stabilize BINDINGS

24. Suspend (Climb)-ING Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate Energy Ready Standing On top Hold Find hold Pull(self) Stabilize BINDINGS

25. Resumed + (Climb)-ING (Resumptive) Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate Energy Ready Standing On top Hold Find hold Pull(self) Stabilize BINDINGS

26. Finish (End) + (Climb)-ING Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate Energy Ready Standing On top Hold Find hold Pull(self) Stabilize BINDINGS

27. Have + (Climb)-ed (Perfect) Energy Ready Standing On top Hold Find hold Pull(self) Stabilize Ready DoneStartProcessFinish Suspend Cancel interrupt resume Iterate BINDINGS

28. Embedding: Has Started (to X) Ready DoneStartProcessFinish Suspend interrupt resume R DSPF S C i r X-Schema for X with bindings

29. Phasal Aspect Maps to the Controller Ready DoneStartProcessFinish Suspend Cancel interruptresume Iterate Inceptive (start, begin) Iterative (repeat) Completive (finish, end) Resumptive(resume)

30. Embedding: About to start (X) Ready DoneStartProcessFinish Suspend interrupt resume R DSPF S C i r X-Schema for X with bindings

31. Embedding: Has Started (to X) Ready DoneStartProcessFinish Suspend interrupt resume R DSPF S C i r X-Schema for X with bindings

32. Begins and Ends “This is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning." –Speech given at the Lord Mayor's Luncheon, Mansion House, London, November 10, 1942. Winston Churchill

33. Embedding: It’s not this (the end) Ongoing FinishDone X-Schema for X with bindings

34. Embedding: It’s not this (beginning of the end) Ongoing FinishDone R DPF S C i r X-Schema for X with bindings S

35. Embedding: But this (The end of the beginning) Ready DoneStartProcessFinish Suspend interrupt resume R DSPF S C i r X-Schema for X with bindings

36. Inherent Aspect (Aksionsart) Vendler-Dowty-Taylor (VDT) classification –Events and States –Events can be Punctual or Durative Atelic or Telic –States satisfy the downward entailment property If a state holds in some interval, it holds in all sub- intervals of that interval.

37. Inherent Aspect Much richer than traditional Linguistic Characterizations (VDT (durative/atomic, telic/atelic)) Action patterns –one-shot, repeated, periodic, punctual –decomposition: concurrent, alternatives, sequential Goal based schema enabling/disabling Generic control features; –interruption, suspension, resumption Resource usage

39. Basic Event X-schemas State Event Transition Simple Event Simple Action Complex Event/Process Complex State

40. Aspectual Types

45. Other Transitions in the Controller may be coded Lexical items may code interrupts –Stumble is an interrupt to an ongoing walk A combination of grammatical and aktionsart may code of the controller phases –Ready to walk : Prospective –Resuming his run: Resumptive –Has been running: Embedded progressive –About to Finish the painting: Embedded Completive. –Canceling the meeting vs. Aborting the meeting.

46. Interaction of Aspect with Tense Reichenbach’s system uses three pointers –Speech Time (S) –Reference Time (R) –Event Time (E) Tense is a partial ordering relation between the pointers –Simple Past E < R, E < S –Perfect E < R < S

48. The Present Tense The Present tense is a state selector. It therefore selects a rest from an input event, resulting in various coercion effects: –Habitual and generic readings of iterated-event sentences, e.g., She smokes, Oil floats on water –‘Progressive-style’ readings of event sentences in languages other than English, e.g., French: Eh bien, à present, je me sens mieux. Le morale revient. ‘Now I’m feeling better. My morale is coming back.’ (Binet, Bidochon 8: 42) –‘Perfect-style’ readings of state-phase sentences in languages other than English, e.g., Ca fait dix minutes qu’elle nous parle de la moquette! ‘She’s been telling us about the carpet for 10 minutes.’ (Binet, Bidochon 10:17)

49. The Present Triumvirate R SPF S C i r R S P F S C i r R DS P F S C i r JAN IS RUNNING R S P F S C i r R DSPF S C i r JAN HAS BEEN RUNNING JAN RUNS

50. Other Present Tense Affordances Of course, we can extend through embedding ANY of the available states in the CONTROLLER. –John is starting his run. –John starts his run (every morning). –John stops his run after 3 miles. (He never achieves his goal of running 5). –John has been canceling his run. –John cancels his run (twice a week). –We have been restarting this Harley for the last 5 mins. –The meeting is about to resume. –My morale is returning (Michaelis 02). Question: Do (which) languages have constructions for these states?

51. Viewpoint Aspect (Perfective/Imperfective)

52. Perfective/Imperfective Perfective Imperfective

53. Simulation and Reference Interval Perfective Imperfective

54. Levels of Granularity Events can be construed at different levels of granularity based on various contextual factors. –In 1991, McEnroe injured his knee while playing tennis. –This morning, I injured my knee while playing tennis. –He is coughing (normal time scale vs. slow- motion film time scale).

55. Summary of Aspect Results Controller mediates between linguistic markings and individual event/verbal x- schemas (Cogsci99, Coling2002) Captures regular event structure; inspired by biological control theory Flexible: specific events may require only a subset of controller; interaction of underlying x- schemas, linguistic markers and hierarchical abstraction/ decomposition of controller accounts for wide range of aspectual phenomena. Important aspectual distinctions, both traditional and novel, can be precisely specified in terms of the interaction of x-schemas with the controller (CogSci97, CogSci98, AAAI99, IJCAI99, CogSci04, CogLing2005): stative/dynamic, durative/punctual: natural in x-schemas telic processes: depletion of resources continuous processes: consumption of resources temporary/effortful states; habituals dynamic interactions with tense, nominals, temporal modifiers incorporation of world knowledge, pragmatics Ongoing Work: Simulation Semantics and Tense-Aspect (with Laura Michaelis)

56. Simulation hypothesis We understand utterances by mentally simulating their content. –Simulation exploits some of the same neural structures activated during performance, perception, imagining, memory… –Linguistic structure parameterizes the simulation. Language gives us enough information to simulate

57. Simulation Semantics BASIC ASSUMPTION: SAME REPRESENTATION FOR PLANNING AND SIMULATIVE INFERENCE – Evidence for common mechanisms for recognition and action (mirror neurons) in the F5 area (Rizzolatti et al (1996), Gallese 96, Buccino 2002, Tettamanti 2004) and from motor imagery (Jeannerod 1996) IMPLEMENTATION: –x-schemas affect each other by enabling, disabling or modifying execution trajectories. Whenever the CONTROLLER schema makes a transition it may set, get, or modify state leading to triggering or modification of other x-schemas. State is completely distributed (a graph marking) over the network. RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!

64. A Precise Notion of Contingency Relations Activation: Executing one schema causes the enabling, start or continued execution of another schema. Concurrent and sequential activation. Inhibition: Inhibitory links prevent execution of the inhibited x-schema by activating an inhibitory arc. The model distinguishes between concurrent and sequential inhibition, mutual inhibition and aperiodicity. Modification: The modifying x-schema results in control transition of the modified xschema. The execution of the modifying x-schema could result in the interruption, termination, resumption of the modified x-schema.

65. General and Domain Knowledge Conceptual Knowledge and Inference –Embodied –Language and Domain Independent –Powerful General Inferences –Ubiquitous in Language Domain Specific Frames and Ontologies –FrameNet, OWL ontologies Metaphor links domain specific to general –E.g., France slipped into recession.

66. Frames Frames are conceptual structures that may be culture specific Words evoke frames –The word “talk” evokes the Communication frame –The word buy (sell, pay) evoke the Commercial Transaction (CT) frame. –The words journey, set out, schedule, reach etc. evoke the Journey frame. Frames have roles and constraints like schemas. –CT has roles vendor, goods, money, customer. Words bind to frames by specifying binding patterns –Buyer binds to Customer, Vendor binds to Seller.

67. Event Frames Event frames have temporal structure that comprises of the controller event structure and generally have constraints on what precedes them, what happens during them, and what state the world is in once the event has been completed.

68. Sample Event Frame: Commercial Transaction Initial state: Vendor has Goods, wants Money Customer wants Goods, has Money Transition: Vendor transmits Goods to Customer Customer transmits Money to Vendor Final state: Vendor has Money Customer has Goods

69. Sample Event Frame: Commercial Transaction Initial state: Vendor has Goods, wants Money Customer wants Goods, has Money Transition: Vendor transmits Goods to Customer Customer transmits Money to Vendor Final state: Vendor has Money Customer has Goods (It’s a bit more complicated than that.)

70. Partial Wordlist for Commercial Transactions Verbs: pay, spend, cost, buy, sell, charge Nouns: cost, price, payment Adjectives: expensive, cheap

71. Meaning and Syntax  The various words that evoke this frame introduce the elements of the frame in different ways.  The identities of the buyer, seller, goods and money  Information expressed in sentences containing these words occurs in different places in the sentence depending on the word.

72. Language understanding: analysis & simulation “Harry walked into the cafe.” Analysis Process Semantic Specification Utterance Constructions Lexicon General Knowledge Belief State CAFE Simulation construction W ALKED form self f.phon  [wakt] meaning : Walk-Action constraints self m.time before Context.speech-time self m..aspect  encapsulated