1. Statistical NLP Spring 2010 Lecture 22: Summarization Dan Klein – UC Berkeley Includes slides from Aria Haghighi, Dan Gillick

4. Selection Maximum Marginal Relevance mid-‘90s present Maximize similarity to the query Minimize redundancy [Carbonell and Goldstein, 1998] s1s1s1s1 s3s3s3s3 s2s2s2s2 s4s4s4s4 Q Greedy search over sentences

5. Maximum Marginal Relevance Graph algorithms [Mihalcea 05++] mid-‘90s present Selection

6. Maximum Marginal Relevance Graph algorithms mid-‘90s present s1s1s1s1 s3s3s3s3 s2s2s2s2 s4s4s4s4 Nodes are sentences Selection

7. Maximum Marginal Relevance Graph algorithms mid-‘90s present s1s1s1s1 s3s3s3s3 s2s2s2s2 s4s4s4s4 Nodes are sentences Edges are similarities Selection

8. Maximum Marginal Relevance Graph algorithms mid-‘90s present s1s1s1s1 s3s3s3s3 s2s2s2s2 s4s4s4s4 Nodes are sentences Edges are similarities Stationary distribution represents node centrality Selection

9. Maximum Marginal Relevance Graph algorithms Word distribution models mid-‘90s present Input document distributionSummary distribution ~w P A (w) Obama? speech? health? Montana?w P D (w) Obama0.017 speech0.024 health0.009 Montana0.002 Selection

10. Maximum Marginal Relevance Graph algorithms Word distribution models mid-‘90s present SumBasic [Nenkova and Vanderwende, 2005] Value(w i ) = P D (w i ) Value(s i ) = sum of its word values Choose s i with largest value Adjust P D (w) Repeat until length constraint Selection

11. Maximum Marginal Relevance Graph algorithms Word distribution models Regression models mid-‘90s present s1s1s1s1 s2s2s2s2 s3s3s3s3 word values positionlength 12124 4214 6318 s2s2s2s2 s3s3s3s3 s1s1s1s1 F(x) frequency is just one of many features Selection

12. Maximum Marginal Relevance Graph algorithms Word distribution models Regression models Topic model-based [Haghighi and Vanderwende, 2009] mid-‘90s present Selection

20. H & V 09PYTHY

23. Maximum Marginal Relevance Graph algorithms Word distribution models Regression models Topic models Globally optimal search mid-‘90s present [McDonald, 2007] s1s1s1s1 s3s3s3s3 s2s2s2s2 s4s4s4s4 Q Optimal search using MMR Integer Linear Program Selection

24. [Gillick and Favre, 2008] Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration. s1s1s1s1 s2s2s2s2 s3s3s3s3 s4s4s4s4 conceptvalue Selection

25. [Gillick and Favre, 2008] Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration. conceptvalue obama3 s1s1s1s1 s2s2s2s2 s3s3s3s3 s4s4s4s4 Selection

26. [Gillick and Favre, 2008] Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration. conceptvalue obama3 health2 s1s1s1s1 s2s2s2s2 s3s3s3s3 s4s4s4s4 Selection

27. [Gillick and Favre, 2008] Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration. conceptvalue obama3 health2 house1 s1s1s1s1 s2s2s2s2 s3s3s3s3 s4s4s4s4 Selection

28. [Gillick and Favre, 2008] Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. conceptvalue obama3 health2 house1 s1s1s1s1 s2s2s2s2 s3s3s3s3 s4s4s4s4 The health care bill is a major test for the Obama administration.summarylengthvalue {s 1, s 3 }175 {s 2, s 3, s 4 }176 Length limit: 18 words greedy optimal Selection

29. [Gillick, Riedhammer, Favre, Hakkani-Tur, 2008] total concept value summary length limit maintain consistency between selected sentences and concepts Integer Linear Program for the maximum coverage model Selection

30. [Gillick and Favre, 2009] This ILP is tractable for reasonable problems Selection

31. How to include sentence position? First sentences are unique Selection

32. How to include sentence position? Only allow first sentences in the summary Up-weight concepts appearing in first sentences Identify more sentences that look like first sentences surprisingly strong baseline included in TAC 2009 system first sentence classifier is not reliable enough yet Selection

33. Some interesting work on sentence ordering [Barzilay et. al., 1997; 2002] But choosing independent sentences is easier First sentences usually stand alone well Sentences without unresolved pronouns Classifier trained on OntoNotes: <10% error rate Baseline ordering module (chronological) is not obviously worse than anything fancier Selection

34. 52 submissions 27 teams 44 topics 10 input docs 100 word summaries Gillick & Favre Rating scale: 1-10 Humans in [8.3, 9.3] Rating scale: 1-10 Humans in [8.5, 9.3] Rating scale: 0-1 Humans in [0.62, 0.77] Rating scale: 0-1 Humans in [0.11, 0.15] Results [G & F, 2009]

35. [Gillick and Favre, 2008] Error Breakdown?

36. Sentence extraction is limiting... and boring! But abstractive summaries are much harder to generate… in 25 words? Beyond Extraction?

37. http://www.rinkworks.com/bookaminute/

38. Brown first saw the new planet Jan. 8 using the 48-inch Samuel Oschin Telescope Sample Errors:... news that the U.K. Panel on Takeover and Mergers... outside the U.S. Amoco already has shed such operations Preprocessing

39. Sentence Boundary Detection (SBD) is non-trivial because periods are ambiguous (sentence boundaries, abbreviations, or both). Wall Street Journal Corpus

40. Preprocessing Local context (one word on each side of the period) is sufficient. 80% of sentence-ending abbrs. in the WSJ test set Key innovations: Using an abbreviation list is too coarse.Abbr. Ends sentence TotalRatio Inc.1096830.16 Co.805660.14 Corp.676990.10 U.S.458000.06 Calif.24860.28 Ltd.231120.21