Exploiting Timelines to Enhance Multi-document Summarization Jun-Ping Ng, Yan Chen, Min-Yen Kan and Zhoujun Li National University of Singapore Beihang University

Cyclone Sidr 2007, JTWC designation: 06B Cyclone Sidr 2007, JTWC designation: 06B Image Courtesy: Univ. Wisconsin- Madison “A fierce cyclone packing extreme winds and torrential rain smashed into Bangladesh’s southwestern coast Thursday, …” 24 Jun 2014ACL Timelines in Summarization 2

24 Jun 2014ACL Timelines in Summarization 3 Image Courtesy: US Navy / Wikipedia “… wiping out homes and trees in what officials described as the worst storm in years.”

24 Jun 2014ACL Timelines in Summarization 4 Image Courtesy: US State Department / Wikipedia “More than 100,000 coastal villagers have been evacuated before the cyclone made landfall.”

Image Courtesy: US Navy / Wikipedia 1991 Bangladesh Cyclone “The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.” 24 Jun 2014ACL Timelines in Summarization 5

24 Jun 2014ACL Timelines in Summarization 6 [3] “The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.” [2] “More than 100,000 coastal villagers have been evacuated before the cyclone made landfall.” [1] “A fierce cyclone packing extreme winds and torrential rain smashed into Bangladesh’s southwestern coast Thursday, wiping out homes and trees in what officials described as the worst storm in years.”

24 Jun 2014ACL Timelines in Summarization 7 [3] “The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.” [2] “More than 100,000 coastal villagers have been evacuated before the cyclone made landfall.” [1] “A fierce cyclone packing extreme winds and torrential rain smashed into Bangladesh’s southwestern coast Thursday, wiping out homes and trees in what officials described as the worst storm in years.”

Timelines from Text 24 Jun 2014ACL Timelines in Summarization 8 [3] “The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.” [2] “More than 100,000 coastal villagers have been evacuated before the cyclone made landfall.” [1] “A fierce cyclone packing extreme winds and torrential rain smashed into Bangladesh’s southwestern coast Thursday, wiping out homes and trees in what officials described as the worst storm in years.”

Key time spans are summary worthy 24 Jun 2014ACL Timelines in Summarization 9 [3] “The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.” [2] “More than 100,000 coastal villagers have been evacuated before the cyclone made landfall.” [1] “A fierce cyclone packing extreme winds and torrential rain smashed into Bangladesh’s southwestern coast Thursday, wiping out homes and trees in what officials described as the worst storm in years.”

Timelines + Summarization Timelines (per input document) Summarization System Summary Lexical and positional features Timeline- derived features Jun 2014ACL Timelines in Summarization Timelines + Summarization

Outline Goal and Motivation Timeline Generation Integrating Timelines –In Scoring: (Contextual) Importance, Density –In Re-ordering: TimeMMR Experiments Discussion Jun 2014ACL Timelines in Summarization

Timeline Generation Jun 2014ACL Timelines in Summarization

1. Event-Event Temporal Classification Jun 2014ACL Timelines in Summarization (Ng et al., 2013; EMNLP)

2. Event-Timex Temporal Classification Jun 2014ACL Timelines in Summarization (Ng and Kan, 2012; COLING)

3. Timex Normalization 15 “Today”  June 6, Jun 2014ACL Timelines in Summarization (HeidelTime; Strötgen and Gertz, 2013)

Timeline Construction 1.Map normalized timexes to timeline 2.Place events which OVERLAP with timexes onto timeline 3.Place events which OVERLAP with other events onto the timeline 4.Insert rest of events based on BEFORE/AFTER ordering Jun 2014ACL Timelines in Summarization

Integrating Timelines into SWING 17 Time Span Importance Contextual Time Span Importance Sentence Temporal Coverage Density Time MMR 24 Jun 2014ACL Timelines in Summarization Temporal Processing Summarization Pipeline SWING (Ng et al., COLING 2012, TAC 2011) State-of-the-art open-source extractive summarizer NG-NUS/SWING NG-NUS/SWING Basic, k of n sentence summaries

1. Time Span Importance (TSI) Time spans which contain many events are more salient Sentences which references events in these time spans are thus better candidates for a summary 24 Jun 2014ACL Timelines in Summarization 18

2. Contextual Time Span Importance (CTSI) Time spans near to important time spans are important Search left and right for local peaks 24 Jun 2014ACL Timelines in Summarization 19, where

3. Sentence Temporal Coverage Density (TCD) Favour sentences which –contain more events –covering a wide variety of time spans 24 Jun 2014ACL Timelines in Summarization 20

Identifying Redundancies SWING makes use of the Maximal Marginal Relevance (MMR) algorithm to identify redundancies in selected sentences MMR is based largely on surface lexical similarities Idea: Let’s use time as a basis to penalize the selection of sentences from redundant time periods Jun 2014ACL Timelines in Summarization

TimeMMR Beyond lexical similarities, identify sentences which contain substantial time span overlap. Candidate sentences which share many time spans with selected sentences are penalized. 22 (1)An official in Barisal, 120 kilometres south of Dhaka, spoke of severe destruction as the 500 kilometre-wide mass of cloud passed overhead. (2)“Many trees have been uprooted and houses and schools blown away,” Mostofa Kamal, a district relief and rehabilitation officer, told AFP by telephone. (3)“Mud huts have been damaged and the roofs of several houses blown off,” said the state’s relief minister, Mortaza Hossain. Lexically dissimilar but redundant 24 Jun 2014ACL Timelines in Summarization Proportion of overlap

Experiments Data –TAC 2010 dataset for training –TAC 2011 dataset for testing Temporal Processing Systems –HeidelTime (Strötgen and Gertz, 2013) –E-T temporal classification (Ng and Kan, 2012) –E-E temporal classification (Ng et al., 2013) Summarization baseline –SWING (Ng et al., 2012) Jun 2014ACL Timelines in Summarization

Results 24 #ConfigurationR-2 RSWING B1CLASSY SWING + Timeline Features0.1394* 2SWING + Timeline Features + TimeMMR Jun 2014ACL Timelines in Summarization Doesn’t seem very effective! * = p < 0.1, ** = p < 0.05, against R row

Analysis: Timelines contain errors Errors from underlying temporal processing systems Simplifying assumptions made in timeline construction Lack of consistency checking and validation For effective use, we must identify good timelines Identify timelines which potentially contain more errors Exclude these when performing summarization Jun 2014ACL Timelines in Summarization

Reliability Filtering Short timelines can result when the system fails to extract or relate events and timexes Features derived from short timelines are prone to have extreme values Use the length of a timeline as a gauge of its accuracy Don’t use timelines shorter than average (as computed over the whole collection) Jun 2014ACL Timelines in Summarization

With Reliability Filtering 27 #ConfigurationR-2 RSWING B1CLASSY SWING + Timeline Features0.1394* 2SWING + Timeline Features + TimeMMR SWING + Timeline Features [Filtered]0.1418** 4SWING + Timeline Features + TimeMMR [Filtered]0.1402** 24 Jun 2014ACL Timelines in Summarization TimeMMR doesn’t seem effective! Why? * = p < 0.1, ** = p < 0.05, against R row

Does TimeMMR actually help? 28 L1An Iraqi reporter threw his shoes at visiting U.S. President George W. Bush and called him a ”dog” in Arabic during a news conference with Iraqi Prime Minister Nuri al-Maliki in Baghdad R1 L2”All I can report is it is a size 10,.R2 L3Muntadhar al-Zaidi, reporter of Baghdadiya television jumped and threw his two shoes one by one at the president, who ducked and thus narrowly missed being struck, raising chaos in the hall in Baghdad’s heavily fortified green Zone. The incident occurred as Bush was appearing with Iraqi Prime Minister Nouri al-Maliki. R3 L4The president lowered his head and the first shoe hit the American and Iraqi flags behind the two leaders. Muntadhar al-Zaidi, reporter of Baghdadiya television jumped and threw his two shoes one by one at the president, who ducked and thus narrowly missed being struck, raising chaos in the hall in Baghdad’s heavily fortified green Zone. R4 L5TheThe president lowered his head and theR5 R-2: , worse by R-2R-2: , better by R-2 24 Jun 2014ACL Timelines in Summarization Possibly Redundant? = Could an (automated) evaluation metric cater for time?

Conclusion Use of automatic timeline generation Integration of timelines into summarization –Sentence scoring via timeline features –Sentence re-ordering via TimeMMR –Length based timeline filtering helps to ameliorate errors Jun 2014ACL Timelines in Summarization For details on temporal processing, see: Jun Ping’s work at COLING 2012, EMNLP 2013 and his doctoral thesis (2014) Questions? If not, ask for more detailed analysis!

ADDITIONAL SLIDES Jun 2014ACL Timelines in Summarization

Related Work For Sentence Reordering –Barzilay et al., 1999 Recency as an indicator of salience –Goldstein et al., 2000;Wan, 2007; Demartini et al., 2010 –Liu et al., 2009 (“Temporal Graph”) –Wu, 2008 (“Largest Cluster”) TREC Temporal Summarization Track –Not as relevant; about monitoring an event over time 24 Jun 2014ACL Timelines in Summarization 31 Close to our TSI

32 24 Jun 2014ACL Timelines in Summarization Baseline; worse With time features; better

TSI: A crane accident With TSI, the cause of the accident in this summary is included; the alternative R1 sentence is background information and does not occur at any key time span. 24 Jun 2014ACL Timelines in Summarization 33 With TSI; better Without TSI; worse

34 24 Jun 2014ACL Timelines in Summarization With CTSI; better Without CTSI; worse With CTSI, the “warn” and “disappear” events were promoted in importance due to their proximity with peak P CTSI: Coral Reef Preservation

Timeline Caveats Some events span a long period of time (i.e., “1999”) Events are ordered based on the start of the duration Timeline captures relative order Construction algorithm does not attempt to reconcile contradictions 24 Jun 2014ACL Timelines in Summarization 35

Timex Normalization Source:Bethard, Jun 2014ACL Timelines in Summarization 36

References Jun-Ping Ng, Interpreting Text with Time, Doctoral Thesis, National University of Singapore, 2014 Jun-Ping Ng, Min-Yen Kan, Ziheng Lin, Wei Feng, Bin Chen, Jian Su, Chew-Lim Tan, Exploiting Discourse Analysis for Article-Wide Temporal Classification, EMNLP 2013 Jun-Ping Ng, Praveen Bysani, Ziheng Lin, Min-Yen Kan, Chew-Lim Tan, Exploiting Category-Specific Information for Multi-Document Summarization, COLING 2012 Jun-Ping Ng, Min-Yen Kan, Improved Temporal Relation Classification using Dependency Parses and Selective Crowdsourced Annotations, COLING Jun 2014ACL Timelines in Summarization 37