1. June 2004 D ARPA TIDES MT Workshop Measuring Confidence Intervals for MT Evaluation Metrics Ying Zhang Stephan Vogel Language Technologies Institute Carnegie Mellon University

2. June 2004 D ARPA TIDES MT Workshop MT Evaluation Metrics Human Evaluations (LDC) –Fluency and Adequacy Automatic Evaluation Metrics –mWER: edit distance between the hypothesis and the closest reference translation –mPER: position independent error rate –BLEU: –Modified BLEU: –NIST:

3. June 2004 D ARPA TIDES MT Workshop Measuring the Confidence Intervals One score per test set How accurate is this score? To measure the confidence interval a population is required Building a test set with multiple human reference translations is expensive Bootstrapping (Efron 1986) –Introduced in 1979 as a computer-based method for estimating the standard errors of a statistical estimation –Resampling: creating an artificial population by sampling with replacement –Proposed by Franz Och (2003) to measure the confidence intervals for automatic MT evaluation metrics

4. June 2004 D ARPA TIDES MT Workshop A Schematic of the Bootstrapping Process Score 0

5. June 2004 D ARPA TIDES MT Workshop An Efficient Implementation Translate and evaluate on 2,000 test sets? –No Way! Resample the n-gram precision information for the sentences –Most MT systems are context independent at the sentence level; –MT evaluation metrics are based on information collected for each testing sentences –E.g. for BLEU and NIST RefLen: 61 52 56 59 ClosestRefLen 56 1-gram: 56 46 428.41 –Similar for human judgment and other MT metrics Approximation for NIST information gain Scripts available at: http://projectile.is.cs.cmu.edu/research/public/tools/bootStrap/tu torial.htm http://projectile.is.cs.cmu.edu/research/public/tools/bootStrap/tu torial.htm

6. June 2004 D ARPA TIDES MT Workshop Confidence Intervals 7 MT systems from June 2002 evaluation Observations: –Relative confidence interval: NIST<M-Bleu<Bleu –I.e. NIST scores have more discriminative powers than BLEU

7. June 2004 D ARPA TIDES MT Workshop Are Two MT Systems Different? Comparing two MT systems’ performance –Using the similar method as for single system –E.g. Diff(Sys1-Sys2):Median=-1.7355 [-1.5453,-1.9056] –If the confidence intervals overlap with 0, two systems are not significantly different –M-Bleu and NIST have more discriminative power than Bleu –Automatic metrics have pretty high correlations with the human ranking –Human judges like system E (Syntactic system) more than B (Statistical system), but automatic metrics do not

8. June 2004 D ARPA TIDES MT Workshop How much testing data is needed

9. June 2004 D ARPA TIDES MT Workshop How much testing data is needed NIST scores increase steadily with the growing test set size The distance between the scores of the different systems remains stable when using 40% or more of the test set The confidence intervals become narrower for larger test set * System A, (Bootstrap Size B=2000)

10. June 2004 D ARPA TIDES MT Workshop How many reference translations are sufficient? Confidence intervals become narrower with more reference translations [100%](1-ref)~[80~90%](2-ref)~[70~80%](3-ref)~[60%~70%](4-ref) One additional reference translation compensates for 10~15% of testing data * System A, (Bootstrap Size B=2000)

11. June 2004 D ARPA TIDES MT Workshop Bootstrap-t interval vs. normal/t interval Normal distribution / t-distribution Student’s t-interval (when n is small) Bootstrap-t interval –For each bootstrap sample, calculate –The alpha-th percentile is estimated by the value, such that –Bootstrap-t interval is – e.g. if B=1000, the 50 th largest value and the 950 th largest value gives the bootstrap-t interval Assuming that

12. June 2004 D ARPA TIDES MT Workshop Bootstrap-t interval vs. Normal/t interval (Cont.) Bootstrap-t intervals assumes no distribution, but –It can give erratic results –It can be heavily influenced by a few outlying data points When B is large, the bootstrap sample scores are pretty close to normal distribution Assume normal distribution gives more reliable intervals, e.g. for BLEU relative confidence interval (B=500) –STDEV=0.27 for bootstrap-t interval –STDEV=0.14 for normal/student-t interval

13. June 2004 D ARPA TIDES MT Workshop The Number of Bootstrap Replications B Ideal bootstrap estimate of the confidence interval takes Computational time increases linearly with B The greater the B, the smaller of the standard deviation of the estimated confidence intervals. E.g. for BLEU’s relative confidence interval –STDEV = 0.60 when B=100 –STDEV = 0.27 when B=500 Two rules of thumb: –Even a small B, say B=100 is usually informative –B>1000 gives quite satisfactory results

14. June 2004 D ARPA TIDES MT Workshop References Efron, B. and R. Tibshirani : 1986, Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy, Statistical Science 1, p. 54-77. F. Och. 2003. Minimum Error Rate Training in Statistical Machine Translation. In Proc. Of ACL, Sapporo, Japan. M. Bisani and H. Ney : 2004, 'Bootstrap Estimates for Confidence Intervals in ASR Performance Evaluation', In Proc. of ICASP, Montreal, Canada, Vol. 1, pp. 409-412. G. Leusch, N. Ueffing, H. Ney : 2003, 'A Novel String-to-String Distance Measure with Applications to Machine Translation Evaluation', In Proc. 9th MT Summit, New Orleans, LO. I Dan Melamed, Ryan Green and Joseph P. Turian : 2003, 'Precision and Recall of Machine Translation', In Proc. of NAACL/HLT 2003, Edmonton, Canada. King M., Popescu-Belis A. & Hovy E. : 2003, 'FEMTI: creating and using a framework for MT evaluation', In Proc. of 9th Machine Translation Summit, New Orleans, LO, USA. S. Nießen, F.J. Och, G. Leusch, H. Ney : 2000, 'An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research', In Proc. LREC 2000, Athens, Greece. NIST Report : 2002, Automatic Evaluation of Machine Translation Quality Using N-gram Co- Occurrence Statistics, http://www.nist.gov/speech/tests/mt/doc/ngram-study.pdf Papineni, Kishore & Roukos, Salim et al. : 2002, 'BLEU: A Method for Automatic Evaluation of Machine Translation', In Proc. of the 20th ACL. Ying Zhang, Stephan Vogel, Alex Waibel : 2004, 'Interpreting BLEU/NIST scores: How much improvement do we need to have a better system?,' In: Proc. of LREC 2004, Lisbon, Portugal.