1. Pastra and Saggion, EACL 2003 Colouring Summaries BLEU Katerina Pastra and Horacio Saggion Department of Computer Science, Natural Language Processing Group, University of Sheffield, U.K.

2. Pastra and Saggion, EACL 2003 Machine Translation vs. Summarization MT: accurate and fluent translation of source doc Auto Sum: informative, reduced version of source We will focus on:  Automatically generated extracts  Single-document Summarization - Sentence level compression  Automatic content-based evaluation  Reuse of evaluation metrics across NLP areas

3. Pastra and Saggion, EACL 2003 The challenge MT: demanding content evaluation Extracts: is their evaluation trivial by definition ??? Idiosyncrasies of the extract evaluation task:  Compression level and rate  High human disagreement on extract adequacy  Could an MT evaluation metric be ported to Automatic Summarization (extract) evaluation ?  If so, which testing parameters should be considered?

4. Pastra and Saggion, EACL 2003 BLEU Developed for MT evaluation (Papineni et al. ’01) => achieves high correlation with human judgement => is reliable even when run >> on different documents >> against different number of model references i.e. reliability is not affected by the use of either multiple references or just a single one

5. Pastra and Saggion, EACL 2003 Using BLEU in NLP >> 0.66 correlation in single-document summaries at 100 words compression rate against a single- reference summary >> 0.82 correlation when multiple-judged document units (sort of multiple references) used NLG (Zajic and Dorr, 2002) Summarization (Lin and Hovy, 2002) Lin-Hovy conclude: The use of a single reference affects reliability

6. Pastra and Saggion, EACL 2003 Evaluation Experiments set up Variables: compression rate, text cluster, gold standard HKNews Corpus (English - Chinese) 18K documents in English 40 thematic clusters = 400 documents each sentence in the cluster assessed by 3 judges with utility values (0-10) encoded in XML

7. Pastra and Saggion, EACL 2003 Evaluation Software Features: position, similarity with document, similarity with query, term distribution, NE scores, etc. (all normalised) Features are linearly combined to obtain sentence scores and sentence extracts Gate & Summarization classes Semantic tagging and Statistical Analysis Software

8. Pastra and Saggion, EACL 2003 Gold standards and summarisers QB = Query-sentence similarity summary Simple 1 = Doc-sentence similarity summary Simple 2 = Lead-based summary Simple 3 = End-of-document summary Reference n = utility based extract based on the utility given by judge n (n = 1,2,3) Reference all = utility based extract based on the sum of utilities given by the n judges

9. Pastra and Saggion, EACL 2003

10. Experiment 1 2 references compared against the third in 5 different compression rates in two text clusters (all available combinations) Are the results BLEU gives on inter-annotator agreement consistent ? => Inconsistency both across text clusters and within clusters at different compression rates (the latter more consistent than the former) => Reliability of BLEU in Sum seems to depend on values of the variables used. If so, how could one identify the appropriate values?

11. Pastra and Saggion, EACL 2003 Experiment 1 2 references compared against the third in 5 different compression rates in two text clusters (all available combinations) Ref 2 - 119710%20%30%40%50% Reference 10.50 - 10.67 - 10.73 - 1 0.79 - 1 Reference 30.34 - 20.51 - 20.52 - 20.63 - 20.69 - 2 Ref 2 - 12510%20%30%40%50% Reference 10.36 - 10.41 - 10.59 - 20.67 - 20.78 - 1 Reference 30.20 - 20.46 - 20.66 - 10.73 - 10.73 - 2

12. Pastra and Saggion, EACL 2003 Experiment 2 For reference X within cluster Y across compression rates the ranking of the systems is not consistent Reference 310%20%30%40%50% Query-Based0.44 20.50 10.58 10.66 10.71 1 Simple 10.10 30.23 30.48 30.57 30.64 3 Simple 20.52 10.45 20.53 20.62 20.68 2 Simple 30.03 40.07 40.08 40.11 4

13. Pastra and Saggion, EACL 2003 Experiment 3 For reference X at compression Y across clusters the ranking of the systems is not consistent Reference 1 – 30%1197125241 Query-Based0.53 10.52 20.46 1 Simple 10.32 30.54 10.32 2 Simple 20.49 20.38 30.29 3 Simple 30.05 40.07 40.08 4

14. Pastra and Saggion, EACL 2003 Experiment 4 For reference ALL across clusters at multiple compression rates the ranking of the systems is (more) consistent Ref-ALL 119710 %20%30%40%50% Query-Based0.55 10.47 10.49 10.62 10.63 2 Simple 10.3184 20.32 30.40 30.49 30.62 3 Simple 20.3134 30.39 20.44 20.56 20.67 1 Simple 30.02 40.03 40.07 40.11 40.13 4 Ref-ALL 12510 %20%30%40%50% Query-Based0.44 10.43 10.57 10.72 10.7641 2 Simple 10.18 30.3684 20.54 20.60 30.68 3 Simple 20.32 20.3673 30.44 30.66 20.7691 1 Simple 30.03 40.06 40.07 40.10 40.14 4

15. Pastra and Saggion, EACL 2003 Experiment 4 (cont.) Is there a way to use BLEU with a single reference summary and still get reliable results back? 10%20%30%40%50%Average Rank Ref1-12513241234213413241234 * 1234 Ref2-1251324 23141324 Ref3-1252314 1324 2314* 2314 Ref1-1197132423141324 23141324 Ref2-11971324 23141324 Ref3-11971324 23141324

16. Pastra and Saggion, EACL 2003 Fails to capture semantic equivalences between n-grams in both their various lexical and syntactical manifestations Notes on BLEU Examples: “Of the 9,928 drug abusers reported in first half of the year, 1,445 or 14.6% were aged under 21.” vs. “...number of reported abusers” “This represents a decrease of 17% over the 1,740 young drug abusers in the first half of 1998.”

17. Pastra and Saggion, EACL 2003 Use of multiple reference summaries needed when using BLEU in Summarization Lack of such resources could probably be overcome using the average rank aggregation technique Conclusions Future work: Scaling up of the experiments Correlation of BLEU with other content-based metrics used in Summarization