1. Event Extraction Error Analysis and Event-Event Relations Heng Ji jih@rpi.edu

2. 2 Challenges of Event Extraction 2pm-2:30pm Event-event Relations 2:30pm-2:50pm Student Presentations [70mins] 2:50pm-4pm Causal Relation Exercise 4pm-4:50pm Outline

3. 3 State-of-the-art Information Extraction and Knowledge Discovery Techniques are Entity-Centric Event-centric Knowledge Discovery from unstructured data to organize knowledge in a network structure Construct High-quality Event Networks – Node: an event (a cluster of coreferential event mentions) Event Mention Extraction based on Joint Modeling with Structured Prediction Event Coreference Resolution based on semantic attributes – Edge: relation between events Some are horizontal and some are vertical, forming an event cone From Entity-Centric to Event-Centric

4. John Boehner (attendee) [Title: House Speaker Member of: Republican Residence: Ohio] Obamacare (act) Barack Obama (approver) [Title: President Employer: US] 2013/10/01 Doc2 US government (organization) CLOSE-ORG US congress (agent) 2013/10/02 Doc3 2013/10/03 Doc5 Healthcare reform (cause) 2010/03/03 Doc1 SIGN Obama ( attendee ) MEET Lockheed Martin (Employer) 3000 Workers (Employee) LEAVE COREFERENCE SUB-EVENT CAUSAL 2013/10/02 Doc4 NSF (organization) DARPA (organization) CLOSE-ORG From Entity-Centric to Event-Centric 4

5. 5 – An Event is a specific occurrence involving participants. – An Event is something that happens. – An Event can frequently be described as a change of state. Chart from (Dölling, 2011) Most of current NLP work focused on this What is an event?

6. 6 Topic Detection and Tracking (TDT) Automatic Content Extraction (ACE) & Entities, Relations and Events (ERE) Propbank, Nombank Timebank, Discourse Treebank Temporal order of coarse-grained groups of events (“topics”) Defined 33 types of events, each event mention includes a trigger word and arguments with roles Each predicate is an event type, no arguments Some nominals and adjectives allowed Rich semantic roles on concepts, no typing on predicates Abstract Meaning Representation (AMR) Chart adapted from (Strassel, 2012)

7. 7 Coarse-grained predicate-argument representations defined in Propbank (Palmer et al., 2005, Xue and Palmer, 2009) and FrameNet (Baker et al., 1998) – No attempt to cluster the predicates – Assign coarse-grained roles to arguments Fine-grained event trigger and argument representations defined in ACE and ERE – Only covers 33 subtypes; e.g., Some typical types of emergent events such as “donation” and “evacuation” in response to a natural disaster are missing Both representations are missing: – Event hierarchies, Relations among events – Measurement to quantify uncertainty, Implicit argument roles – The narrator’s role, profile and behavior intent and temporal effects – Hard to construct a timeline of events across documents – Efforts are being made by the DEFT event working group (NAACL13/ACL14 Event Workshops and this DEFT PI meeting Event Workshop) Limitations of Previous Event Representations

8. 8 Coarse-grained predicate-argument representations defined in Propbank (Palmer et al., 2005, Xue and Palmer, 2009) and FrameNet (Baker et al., 1998) – No attempt to cluster the predicates – Assign coarse-grained roles to arguments Fine-grained event trigger and argument representations defined in ACE and ERE – Only covers 33 subtypes; e.g., Some typical types of emergent events such as “donation” and “evacuation” in response to a natural disaster are missing Both representations are missing: – Event hierarchies, Relations among events – Measurement to quantify uncertainty, Implicit argument roles – The narrator’s role, profile and behavior intent and temporal effects – Hard to construct a timeline of events across documents – Efforts are being made by the DEFT event working group (NAACL13/ACL14 Event Workshops and this DEFT PI meeting Event Workshop) Limitations of Previous Event Representations

9. A New Representation for Events 9 Current NLP’s Focus

10. Typical Event Mention Extraction Features Trigger Labeling  Lexical Tokens and POS tags of candidate trigger and context words  Dictionaries Trigger list, synonym gazetteers  Syntactic the depth of the trigger in the parse tree the path from the node of the trigger to the root in the parse tree the phrase structure expanded by the parent node of the trigger the phrase type of the trigger  Entity the entity type of the syntactically nearest entity to the trigger in the parse tree the entity type of the physically nearest entity to the trigger in the sentence Argument Labeling  Event type and trigger Trigger tokens Event type and subtype  Entity Entity type and subtype Head word of the entity mention  Context Context words of the argument candidate  Syntactic the phrase structure expanding the parent of the trigger the relative position of the entity regarding to the trigger (before or after) the minimal path from the entity to the trigger the shortest length from the entity to the trigger in the parse tree (Chen and Ji, 2009)

11. 11 OOV this morning in Michigan, a second straight night and into this morning, hundreds of people have been rioting [attack] in Benton harbor. Scene understanding This was the Italian ship that was taken -- that was captured [transfer-ownership] by Palestinian terrorists back in 1985 and some may remember the story of Leon clinghover, he was in a cheal chair and the terrorists shot him and pushed him over the side of the ship into the Mediterranean where he obviously, died. The gloved one claims the label has been releasing new albums and Jackson five merchandise without giving [transfer-money] him "a single dollar." Airlines are getting [transport] flyers to destinations ontime more often. Three Young Boys,ages 2, 5 and 10 survived and are in critical [injure] condition after spending in 18 hours in the cold. Event Trigger Identification Missing Errors

12. 12 Lara hijacking in 1995 has been captured [arrest-jail] by U.S. forces in or near Baghdad. I want to take [transport] this opportunity to stand behind the Mimi and proclaim my solidarity. But when we moved [transport] in closer and started to scan the area, then I saw the 2-year-old waving from next to the airplane. He's left [transport] a lot on the table. Barbara, Bill Bennet's Glam gambling loss [die] changed my opinion. Stewart has found the road to fortune wherever she has traveled [transport]. And it's hard to win back that sort of brand equity that she's lost [end-position]. Police measure the spot where ten months old Miana Williams landed after she was thrown [transport] from a seventh floor window at this apartment building. The baby fell [die] 80 feet. These tree branches cushioned her fall [die] and saved her life. Event Trigger Identification Spurious Errors

13. 13 He -- After landing and taking a brief tour of part of the ship, he also went [transport] out on to the flight deck and observed some of the flight operations of some of the f-18 aircraft that are now leaving these aircraft… still hurts [attack] me to read this. – Metaphor features: semantic categories (noun.animal, noun.cognition) or degree of abstractness Any farm kid knows that certain animals cannot see for a few days after they're born [born], including puppies. We happen to be at a very nice spot by the beach where this is a chance for people to get [transport] away from CNN coverage, everything, and kind of relax. He bought the machinery, moved [transport] to a new factory, rehired some of the old workers and started heritage programs. Under the reported plans, Blackstone Group would buy Vivendi's theme park division, including Universal Studios Hollywood, Universal Orlando in Florida and Universal's ownership interests in parks in Spain and Japan, a source close to the negotiations [meet] told the paper. Event Trigger Identification Spurious Errors (Con’t)

14. 14 a man in New York is facing attempted murder charges after allegedly throwing [transport  attack] his baby seven stories to the ground below. He's also being fined [transfer-money  fine] $3 million and was ordered to pay $1.2 million in restitution to the New York State Tax Commission. ``They admitted to having nuclear capability and weapons at this moment,'' said Rep. Curt Weldon, who headed a delegation of U.S. lawmakers that visited [meet  transport] Pyongyang for three days ending Sunday. Blasphemy is punishable by death [die  execute] under the Pakistan Penal Code. We discussed [phone-write  meet] the Middle East peace process. Event Trigger Classification Errors

15. 15 Scene understanding: This was the Italian ship that was taken -- that was captured by Palestinian terrorists back in 1985 [die_time] and some may remember the story of Leon clinghover, he was in a cheal chair and the terrorists shot him and pushed him over the side of the ship into the Mediterranean [die_place] where he [die_victim] obviously, died. She [die_victim] was going to fight to the death. Then police say the baby's mother pulled out a kitchen knife [die_instrument] opinion on the 911 tape you can hear Williams tape say "go ahead kill me." Coreference: another story out of Belgrade [attack_place], violence at the highest form. Unusual syntactic structure: Two f-14 Tomcats struck the targets, the same area was a site [attack_place] of heavy bombing yesterday. The EU foreign ministers met hours after U.S. President George W. Bush gave Saddam [attack_target] 48 hours to leave Iraq or face invasion. Event Argument Identification Missing Errors

16. 16 OOV: We've seen in the past in Bosnia for example, you held elections and all of the old ethnic thugs [election_person] get into power because they have organization and they have money and they stop the process of genuine building of democracy. There are charges, U.S. charges which have expired but could, I am told, possibly be re -- restarted for piracy, hostage taking and conspiracy [charge_crime]. Stewart and former imclone CEO Sam Waksal shared a broker who now also faces charges of obstruction of justice and perjury [charge_crime]. I was born four years too late to have slept with JFK [born_time]? He called the case and I'm quoting now, a judge's worst nightmare, but he noted that Maryland Parole Boards and the facility where Michael Serious was held, the institution made what the judge called the final decision on whether to release serious [release-parole_person]. Last week Williamson, a mother of four, was found stabbed to death at a condominium [die_place] in Greenbelt, Maryland. A source tell US Enron is considering suing its own investment bankers for giving it bad financial advice [Sue_crime]. Event Argument Identification Missing Errors (Cont’)

17. 17 He's been in Libya and he's been living under the protection of Saddam Hussein in Baghdad [die_place], but he is wanted for murder in Italy Italy. The EU foreign ministers met hours after U.S. President George W. Bush [meet_entity] gave Saddam 48 hours to leave Iraq or face invasion. Event Argument Identification Spurious Errors

18. 18 His private plane [vehicle  artifact] arrived at Heathrow Airport. a cruise ship [instrument  target] is being searched off the Hawaii coast after two notes threatening terrorist attack were found on board. Event Argument Role Classification Errors

19. Event-Event Relation Taxonomy Preliminary version, under refinement with guidance from Ed Hovy Focus: subevent, confirmation, causality and temporality Inheritance Expansion Comparison Contingency Temporality Reemergence Subevent Variation Conjunction Disjunction Confirmation Competition Negation Opposite Concession Causality Conditionality Synchronism Reference/Target Ancestor/Descendant Semina/Variance Generation/Neighborship Generation/Instantiation Superior/Inferior/parallelism Initiator/Negator N/A Prerequisite/Opposite to Expectation Cause/Result Condition/Emergence Priority/Posteriority N/A Main Type Subtype Role Asynchronism 19

20. Event-Event Relation: Inheritance Inheritance Reemergence Subevent Variation Reference/Target Ancestor/Descendant Semina/Variance Eruption of Mount Vesuvius in 1944. Eruption of Mount Vesuvius in 1906. Inheritance. Reemergence Inheritance. Variation Mount Vesuvius keeps silent recently. Inheritance.Subevent The eruption covered most of north area with ash. Definition: if X clones or succeeds Y, or serves as a revolutionary change of Y Role=ReferenceRole=DescendantRole=Variance 20

21. Event-Event Relation: Expansion Definition: if X and Y jointly constitute a larger event or share the same subevent, or X serves as the fine-grain instance or evidence of Y, Expansion Conjunction Disjunction Confirmation Generation/Neighborship Generation/Instantiation Second Explosion. First Explosion. Boston Marathon Bombing. Expansion.Conjunction Expansion.Disjunction Arrest the suspect Try the suspect Dzhokhar was suspected Made bombs at home. Appear on the scene. Expansion.Confirmation X and Y share the same subevent X and Y jointly constitute a main event X serves as the fine-grain instance or evidence of Y 21

22. Event-Event Relation: Contingency Definition: if X creates a condition for the emergence of Y, or results in Y The operation to raise the Kursk submarine was scheduled to begin in about two weeks. The special hoisting equipment had been properly installed in the massive pick-up boat. Contingency.Conditionality X creates a condition for the emergence of Y Contingency Causality Conditionality Cause/Result Condition/Emergence Role=ConditionRole=Emergence A violent storm would hit Barents sea. Salvage plan was postponed indefinitely. Contingency.Causality X results in the emergence of Y Role=Cause Role=Result 22 Primacy Effect

23. Relations: BEFORE, AFTER, INCLUDES, INCLUDED _ IN, SIMULTANEOUS, VAGUE Annotators must label all pairs within one sentence window Annotation guidelines address resulting difficult cases Evaluate and develop systems to decide whether a pair can be related 23 There were four or five people inside, and they just started firing Ms. Sanders was hit several times and was pronounced dead at the scene. The other customers fled, and the police said it did not appear that anyone else was injured. There were four or five people inside, and they just started firing Ms. Sanders was hit several times and was pronounced dead at the scene. The other customers fled, and the police said it did not appear that anyone else was injured. Original TimeBank** TimeBank-Dense (Cassidy et al., 2014); ** (Pustejovsky et al., 2003) From TimeBank to TimeBank-Dense New relation Event-Event Relation: Temporality

24. 24 An Example: Input [Oct.25 News] WNYC: The Office of Emergency Management situation room is now open, and Mayor Michael Bloomberg said they are monitoring the progress of the storm. "We saw that hurricanes like Irene can really do damage and we have to take them seriously, but we don't expect based on current forecasts to have anything like that," Bloomberg said. "But we're going to make sure we're prepared." He advised residents in flood-prone areas be prepared to evacuate. [Oct.29 Tweets] OMG!!! NYC IS GETTING THE EYE OF HURRICANE SANDY NEXT WEEK....SCARY!!! I HAVE TO EVACUATE MY NEIGHBORHOOD!

25. 25 Representing Event-Event Relations in KB Entity KB ID = E1Name = Office of Emergency Management ID = E2Name = Michael BloombergTitle = MayorEmployer = E5 ID = E3Name = Residents in flood-phone areas ID = E4Name = WNYC Property = News ID = E5Name = New York City ID = E6Name = Tammi Delancy #Follower s= 211 #Friends = 770 Role = Participant &Commenter City_of_reside nce = E5

26. 26 Representing Event-Event Relations in KB Event KB ID = EV1 PropertiesType= Warning Novelty= 1.0 Modality = Asserted Tense = Past Polarity = Positive Genericity = Specific Triggeradvised ArgumentsAdvisor = E1, E2 Advisees = E3 Advice = Evacuation Narrator = E4Time = 10/25/2012Place = E5 Result EventEV2 Children EventEV ID = EV2 PropertiesType= Evacuation Novelty= 1.0 Modality = Asserted Tense= Future Polarity = Positive Genericity = Specific Triggerevacuate ArgumentsNarrator = E4Time = 10/29/2012Place = E5 Cause EventEV1 Parent EventEV4

27. 27 We have a reasonable understanding on how to detect temporal relations (Ji et al., 2013; Cassidy and Ji, 2014) but not others Main Challenges – Clustering coreferential events - what is event coreference anyway (Hovy et al., 2013)? – Event-event relations are often logic relations without explicit linguistic indicators – Fine-grained distinction / Implicit logic relations (Hong et al., 2012) Deep background and commonsense knowledge acquisition – FrameNet/Google ngram / Concept Net /Background news – Retrospective Historical Event Detection  precursor prediction (Hong and Ji, 2014submission) – Take advantage of rich knowledge representation such as AMR ( Banarescu et al., 2013 ) Pilot System