1. Temporal Ordering of Events in the News Domain Preethi Raghavan

2. Motivation Users have temporal information needs Query: “Prime Minister United Kingdom 2000” Query : “Prime Minister United Kingdom immediately before 2000” Problem Traditional information retrieval systems do not exploit the temporal content in documents Possibilities Integrate temporal dimension into an information retreival framework Question Answering Relative order of events in multi-document summarization

3. TimeBank Corpus Characteristics News reports annotated using the TimeML specification 186 documents, with a total of 68.5K words. 10% of the corpus is held out as test data TimeML annotations  EVENT: typically verbs  TIMEX3: temporal expressions  TLINK: relates events using temporal relations modeled after Allen’s Interval Algebra + James F. Allen: Maintaining knowledge about temporal intervals. In: Communications of the ACM., 1983

4. Example Unordered Events in a Document New evidence is suggesting that a series of bombings in Atlanta and last month’s explosion at an Alabama women's clinic might be related In 1996, a bomb blast shocks the Olympic games One person is killed

5. Simplified Sample TimeML Annotation A bomb blast shocks the Olympic games. <TLINK relType= "BEFORE" eventID="e138" relatedToEvent="e11"/>

6. Methodology Infer partial order by learning the relation between event pairs in a document ◦ Collapsed labels used:  BEFORE = {IBEFORE, BEFORE}  AFTER = {IAFTER, AFTER}  OVERLAPS = {SIMULTANEOUS, INCLUDES, INCLUDED_BY, DURING, BEGINS, ENDS, ENDED_BY, BEGUN_BY, IDENTITY} ◦ For instance, in document d1  e2 BEFORE e3  e2 AFTER e1  e3 OVERLAPS e4 Infer global temporal order using the proposed approaches ◦ d1: e1, e2, e3

7. Event Pairs Classification: Feature Set Training data: 3000 event pairs Testing data: 481 event pairs Features: ◦ Event Class: Occurrence (bombing, discovered), Reporting (say) ◦ Tense: Present, Past etc. ◦ Aspect: Progressive, Perfective etc. ◦ Polarity: Positive, Negative ◦ Event Phrase ◦ Temporal Expression occurring in the same sentence as the event ◦ Same aspect, Same tense

8. Event Pair Classification Results Event-Event Relation using 13 Labels Event-Event Relation using 3 Labels ClassifierPrecision (%)Recall(%)Accuracy (%) Naïve Bayes4.281.641.45 SVM4.561019.8 MaxEnt14.9814.833.89 ClassifierPrecision (%)Recall(%)Accuracy (%) Naïve Bayes43.640.439 SVM17.533.352.4 MaxEnt47.143.156.1

9. Event Pair Classification Results MaxEnt, Overall Accuracy 56.1% ◦ (Majority Classifier 52.4%) Other Experiments ◦ Experiments in Mani et. al use 6 disjunctive labels. Overall accuracy 62.5% ◦ Collapsing BEFORE and AFTER into the same category will increase accuracy ClassifierPrecision (%)Recall(%)F1 BEFORE43.119.126.5 AFTER38.121.127.1 OVERLAPS6188.972.4

10. Event Pair Classification Results TimeBank + Aquaint Corpus (6234 Event-Event pairs) 6 labels ◦ (BEGINS, SIMULTANEOUS,BEFORE, IBEFORE,ENDS, INCLUDES) ◦ MaxEnt Overall accuracy 62.179 2 labels ◦ (BEFORE, OVERLAPS) ◦ MaxEnt Overall accuracy 69.711 ClassifierPrecision (%)Recall(%)F1 BEFORE74.477273.3 OVERLAPS63.466.564.9

11. Inferring Global Temporal Order Ordering of events as a Temporal Directed Acyclic Graph (TDAG) Nodes: Events Edges: Temporal relation between events Cycles are prohibited ◦ Since the graph encodes order Coarse annotation scheme ◦ Does not capture overlap ◦ Only captures precedence relations

12. Problem Given a partial ordering of event pairs, how do we generate a TDAG to establish global ordering?

13. Greedy Approach Greedy Algorithm (1) Sort edges according to scores. (2) Start with an empty graph. (3) Add the current largest edge into the graph. (4) Apply transitive closure and constraints. (5) Repeat (3) and (4) until all edges are considered.

14. Integer Linear Programming For a document with N event pairs, each pair (i, j) can be related in the graph as ◦ i BEFORE j ◦ i AFTER j ◦ i not connected to j Given the probability scores for the relation assigned to each event pair Objective: ◦ Optimize the score of a TDAG by maximizing the sum of the scores of all edges in the graph

15. ILP Constraints No cycles Enforce transitivity Connectivity constraint

16. Reference TDAG

17. Inferring Global Temporal Order TDAG generated using ILP

18. Observations ILP generates some feasible solution, but not necessarily optimal In certain cases, it recognized the presence of a link, but is not able to accurately predict its direction A single wrongly inferred relation may lead to generation of multiple wrong inferences For the reference TDAG, ◦ ILP gives us 80% accuracy ◦ Greedy gives 60% accuracy

19. Conclusions Accuracy for 6 disjunctive labels matches the baseline by Mani et al. for event pair relation classification Global ordering helps infer new relations between events This could also be used to increase the size of training data and learn on an increased corpus.

20. References 1. Philip Bramsen, Pawan Deshpande, Yoong Keok, Lee, Regina Barzilay, Inducing Temporal Graphs. EMNLP (2006) 2. Inderjeet Mani, Marc Verhagen, Ben Wellner, Chong Min Lee and James Pustejovsky, Machine Learning of Temporal Relations. ACL (2006) 3. J. Pustejovsky, J. Castano, R. Ingria, R. Sauri, R. Gauzauskas, A. Setzer, G. Katz, TimeML: Robust Specification of Event and Temporal Expression in Text. IWCS (2003) 4. J. F. Allen. Towards a general theory of action and time. Artificial Intelligence, July 1984 5. www.timeml.org/site/timebank/timebank.html www.timeml.org/site/timebank/timebank.html 6. Mixed Integer Programming Solver: CPLEX 7. Modeling tool: AMPL