1. Exploiting Timelines to Enhance Multi-document Summarization Jun-Ping Ng, Yan Chen, Min-Yen Kan and Zhoujun Li National University of Singapore Beihang University

2. Cyclone Sidr 2007, JTWC designation: 06B Cyclone Sidr 2007, JTWC designation: 06B Image Courtesy: Univ. Wisconsin- Madison “A fierce cyclone packing extreme winds and torrential rain smashed into Bangladesh’s southwestern coast Thursday, …” 24 Jun 2014ACL 2014 - Timelines in Summarization 2

3. 24 Jun 2014ACL 2014 - Timelines in Summarization 3 Image Courtesy: US Navy / Wikipedia “… wiping out homes and trees in what officials described as the worst storm in years.”

4. 24 Jun 2014ACL 2014 - Timelines in Summarization 4 Image Courtesy: US State Department / Wikipedia “More than 100,000 coastal villagers have been evacuated before the cyclone made landfall.”

5. Image Courtesy: US Navy / Wikipedia 1991 Bangladesh Cyclone “The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.” 24 Jun 2014ACL 2014 - Timelines in Summarization 5

6. 24 Jun 2014ACL 2014 - Timelines in Summarization 6 [3] “The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.” [2] “More than 100,000 coastal villagers have been evacuated before the cyclone made landfall.” [1] “A fierce cyclone packing extreme winds and torrential rain smashed into Bangladesh’s southwestern coast Thursday, wiping out homes and trees in what officials described as the worst storm in years.”

7. 24 Jun 2014ACL 2014 - Timelines in Summarization 7 [3] “The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.” [2] “More than 100,000 coastal villagers have been evacuated before the cyclone made landfall.” [1] “A fierce cyclone packing extreme winds and torrential rain smashed into Bangladesh’s southwestern coast Thursday, wiping out homes and trees in what officials described as the worst storm in years.”

8. Timelines from Text 24 Jun 2014ACL 2014 - Timelines in Summarization 8 [3] “The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.” [2] “More than 100,000 coastal villagers have been evacuated before the cyclone made landfall.” [1] “A fierce cyclone packing extreme winds and torrential rain smashed into Bangladesh’s southwestern coast Thursday, wiping out homes and trees in what officials described as the worst storm in years.”

9. Key time spans are summary worthy 24 Jun 2014ACL 2014 - Timelines in Summarization 9 [3] “The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.” [2] “More than 100,000 coastal villagers have been evacuated before the cyclone made landfall.” [1] “A fierce cyclone packing extreme winds and torrential rain smashed into Bangladesh’s southwestern coast Thursday, wiping out homes and trees in what officials described as the worst storm in years.”

10. Timelines + Summarization Timelines (per input document) Summarization System Summary Lexical and positional features Timeline- derived features 10 24 Jun 2014ACL 2014 - Timelines in Summarization Timelines + Summarization

11. Outline Goal and Motivation Timeline Generation Integrating Timelines –In Scoring: (Contextual) Importance, Density –In Re-ordering: TimeMMR Experiments Discussion 11 24 Jun 2014ACL 2014 - Timelines in Summarization

12. Timeline Generation 12 24 Jun 2014ACL 2014 - Timelines in Summarization

13. 1. Event-Event Temporal Classification 13 24 Jun 2014ACL 2014 - Timelines in Summarization (Ng et al., 2013; EMNLP)

14. 2. Event-Timex Temporal Classification 14 24 Jun 2014ACL 2014 - Timelines in Summarization (Ng and Kan, 2012; COLING)

15. 3. Timex Normalization 15 “Today”  June 6, 2014 24 Jun 2014ACL 2014 - Timelines in Summarization (HeidelTime; Strötgen and Gertz, 2013)

16. Timeline Construction 1.Map normalized timexes to timeline 2.Place events which OVERLAP with timexes onto timeline 3.Place events which OVERLAP with other events onto the timeline 4.Insert rest of events based on BEFORE/AFTER ordering 16 1999 24 Jun 2014ACL 2014 - Timelines in Summarization

17. Integrating Timelines into SWING 17 Time Span Importance Contextual Time Span Importance Sentence Temporal Coverage Density Time MMR 24 Jun 2014ACL 2014 - Timelines in Summarization Temporal Processing Summarization Pipeline SWING (Ng et al., COLING 2012, TAC 2011) State-of-the-art open-source extractive summarizer https://github.com/WI NG-NUS/SWING https://github.com/WI NG-NUS/SWING Basic, k of n sentence summaries

18. 1. Time Span Importance (TSI) Time spans which contain many events are more salient Sentences which references events in these time spans are thus better candidates for a summary 24 Jun 2014ACL 2014 - Timelines in Summarization 18

19. 2. Contextual Time Span Importance (CTSI) Time spans near to important time spans are important Search left and right for local peaks 24 Jun 2014ACL 2014 - Timelines in Summarization 19, where

20. 3. Sentence Temporal Coverage Density (TCD) Favour sentences which –contain more events –covering a wide variety of time spans 24 Jun 2014ACL 2014 - Timelines in Summarization 20

21. Identifying Redundancies SWING makes use of the Maximal Marginal Relevance (MMR) algorithm to identify redundancies in selected sentences MMR is based largely on surface lexical similarities Idea: Let’s use time as a basis to penalize the selection of sentences from redundant time periods. 21 24 Jun 2014ACL 2014 - Timelines in Summarization

22. TimeMMR Beyond lexical similarities, identify sentences which contain substantial time span overlap. Candidate sentences which share many time spans with selected sentences are penalized. 22 (1)An official in Barisal, 120 kilometres south of Dhaka, spoke of severe destruction as the 500 kilometre-wide mass of cloud passed overhead. (2)“Many trees have been uprooted and houses and schools blown away,” Mostofa Kamal, a district relief and rehabilitation officer, told AFP by telephone. (3)“Mud huts have been damaged and the roofs of several houses blown off,” said the state’s relief minister, Mortaza Hossain. Lexically dissimilar but redundant 24 Jun 2014ACL 2014 - Timelines in Summarization Proportion of overlap

23. Experiments Data –TAC 2010 dataset for training –TAC 2011 dataset for testing Temporal Processing Systems –HeidelTime (Strötgen and Gertz, 2013) –E-T temporal classification (Ng and Kan, 2012) –E-E temporal classification (Ng et al., 2013) Summarization baseline –SWING (Ng et al., 2012) 23 24 Jun 2014ACL 2014 - Timelines in Summarization

24. Results 24 #ConfigurationR-2 RSWING0.1339 B1CLASSY0.1278 1SWING + Timeline Features0.1394* 2SWING + Timeline Features + TimeMMR0.1389 24 Jun 2014ACL 2014 - Timelines in Summarization Doesn’t seem very effective! * = p < 0.1, ** = p < 0.05, against R row

25. Analysis: Timelines contain errors Errors from underlying temporal processing systems Simplifying assumptions made in timeline construction Lack of consistency checking and validation For effective use, we must identify good timelines Identify timelines which potentially contain more errors Exclude these when performing summarization 25 24 Jun 2014ACL 2014 - Timelines in Summarization

26. Reliability Filtering Short timelines can result when the system fails to extract or relate events and timexes Features derived from short timelines are prone to have extreme values Use the length of a timeline as a gauge of its accuracy Don’t use timelines shorter than average (as computed over the whole collection) 26 24 Jun 2014ACL 2014 - Timelines in Summarization

27. With Reliability Filtering 27 #ConfigurationR-2 RSWING0.1339 B1CLASSY0.1278 1SWING + Timeline Features0.1394* 2SWING + Timeline Features + TimeMMR0.1389 3SWING + Timeline Features [Filtered]0.1418** 4SWING + Timeline Features + TimeMMR [Filtered]0.1402** 24 Jun 2014ACL 2014 - Timelines in Summarization TimeMMR doesn’t seem effective! Why? * = p < 0.1, ** = p < 0.05, against R row

28. Does TimeMMR actually help? 28 L1An Iraqi reporter threw his shoes at visiting U.S. President George W. Bush and called him a ”dog” in Arabic during a news conference with Iraqi Prime Minister Nuri al-Maliki in Baghdad R1 L2”All I can report is it is a size 10,.R2 L3Muntadhar al-Zaidi, reporter of Baghdadiya television jumped and threw his two shoes one by one at the president, who ducked and thus narrowly missed being struck, raising chaos in the hall in Baghdad’s heavily fortified green Zone. The incident occurred as Bush was appearing with Iraqi Prime Minister Nouri al-Maliki. R3 L4The president lowered his head and the first shoe hit the American and Iraqi flags behind the two leaders. Muntadhar al-Zaidi, reporter of Baghdadiya television jumped and threw his two shoes one by one at the president, who ducked and thus narrowly missed being struck, raising chaos in the hall in Baghdad’s heavily fortified green Zone. R4 L5TheThe president lowered his head and theR5 R-2: 0.2643, worse by R-2R-2: 0.2772, better by R-2 24 Jun 2014ACL 2014 - Timelines in Summarization Possibly Redundant? = Could an (automated) evaluation metric cater for time?

29. Conclusion Use of automatic timeline generation Integration of timelines into summarization –Sentence scoring via timeline features –Sentence re-ordering via TimeMMR –Length based timeline filtering helps to ameliorate errors 29 24 Jun 2014ACL 2014 - Timelines in Summarization For details on temporal processing, see: Jun Ping’s work at COLING 2012, EMNLP 2013 and his doctoral thesis (2014) Questions? If not, ask for more detailed analysis!

30. ADDITIONAL SLIDES 30 24 Jun 2014ACL 2014 - Timelines in Summarization

31. Related Work For Sentence Reordering –Barzilay et al., 1999 Recency as an indicator of salience –Goldstein et al., 2000;Wan, 2007; Demartini et al., 2010 –Liu et al., 2009 (“Temporal Graph”) –Wu, 2008 (“Largest Cluster”) TREC Temporal Summarization Track –Not as relevant; about monitoring an event over time 24 Jun 2014ACL 2014 - Timelines in Summarization 31 Close to our TSI

32. 32 24 Jun 2014ACL 2014 - Timelines in Summarization Baseline; worse With time features; better

33. TSI: A crane accident With TSI, the cause of the accident in this summary is included; the alternative R1 sentence is background information and does not occur at any key time span. 24 Jun 2014ACL 2014 - Timelines in Summarization 33 With TSI; better Without TSI; worse

34. 34 24 Jun 2014ACL 2014 - Timelines in Summarization With CTSI; better Without CTSI; worse With CTSI, the “warn” and “disappear” events were promoted in importance due to their proximity with peak P CTSI: Coral Reef Preservation

35. Timeline Caveats Some events span a long period of time (i.e., “1999”) Events are ordered based on the start of the duration Timeline captures relative order Construction algorithm does not attempt to reconcile contradictions 24 Jun 2014ACL 2014 - Timelines in Summarization 35

36. Timex Normalization Source:Bethard, 2013 24 Jun 2014ACL 2014 - Timelines in Summarization 36

37. References Jun-Ping Ng, Interpreting Text with Time, Doctoral Thesis, National University of Singapore, 2014 Jun-Ping Ng, Min-Yen Kan, Ziheng Lin, Wei Feng, Bin Chen, Jian Su, Chew-Lim Tan, Exploiting Discourse Analysis for Article-Wide Temporal Classification, EMNLP 2013 Jun-Ping Ng, Praveen Bysani, Ziheng Lin, Min-Yen Kan, Chew-Lim Tan, Exploiting Category-Specific Information for Multi-Document Summarization, COLING 2012 Jun-Ping Ng, Min-Yen Kan, Improved Temporal Relation Classification using Dependency Parses and Selective Crowdsourced Annotations, COLING 2012 24 Jun 2014ACL 2014 - Timelines in Summarization 37