1. Evaluation in NLP Zdeněk Žabokrtský

2. Intro The goal of NLP evaluation is to measure one or more qualities of an algorithm or a system Definition of proper evaluation criteria is one way to specify precisely an NLP problem Need for evaluation metrics and evaluation data (language data resources).

3. Automatic vs. manual evaluation automatic –comparing the system's output with the gold standard output –the cost of producing the gold standard data... –... but then easily repeatable without additional cost manual –for many NLP problems, the definition of a gold standard can prove impossible (e.g., when inter-annotator agreement is insufficient) –manual evaluation is performed by human judges, which are instructed to estimate the quality of a system, based on a number of criteria

4. Intrinsic vs. extrinsic evaluation Intrinsic evaluation – considers an isolated NLP system and characterizes its performance mainly with respect to a gold standard result. Extrinsic evaluation –considers the NLP system as a component in a more complex setting.

5. Lower and upper bounds Naturally, the performance of our system is expected to be inside the interval given by –lower bound - result of a baseline solution (less complex or even trivial system the performance of which is supposed to be easily surpassed) –upper bound - inter-annotator agreement

6. Evaluation metrics the simplest case: –if the number of task instances is known –if the system gives exactly one answer for each instance –if there is exactly one possible clearly correct answer for each instance –if all errors are equally wrong But what if not?

7. Precision and recall But –we are in 2D now –new issue: precision-recall tradeoff

8. F-measure one way how to get back to 1D weighted harmonic mean of P and R usually evenly weighted

9. Evaluation in classification tasks confusion matrix

10. Evaluation in phrase-structure parsing nontrivial, because –number added nonterminals is not known in advance –it is not clear what should be treated as the (atomic) task instance GEIG metric –Grammar Evaluation Interest Group; used in Parseval –counts the proportion of bracketings which group the same sequences of words in both trees LA metric –leaf-ancestor metric - similarity in sequences of node labels along the paths from terminal nodes to tree root.

11. Evaluation in dependency parsing straighforward delimitatation of problem instance - - no nonterminal nodes are added unlabeled accuracy –proportion of correctly attached nodes (nodes with correctly predicted parent) labeled accuracy –proportion of nodes which are correctly attached and whose labels (dependency relation) are correct too

12. Evaluation in automatic speech recognition not only the recognized words might be incorrect, but the number of recognized words might be different too WER - Word Error Rate

13. Evaluation in machine translation obviously highly non-trivial –there are always more translations possible –there are always more criteria to be judged (translation fidelity, grammatical/pragmatic/stylistic correctness...?) –the essence of translation -- transfering the same meaning from one natural language to annother -- cannot be evaluated by the contemporary machines at all !!! current approaches –either to use human judges –or to use reference (human-made) translations and string- wise comparison metrics which are hoped to correlate with human judgement: BLEU, NIST, METEOR...

14. BLEU zkratka? N - maximal considered n-gram length (usually 4) p n - precision on n-gram using (a set of) reference translation(s) w n - positive weight (typically 1/N) BP - brevity penalty (to compensate easier n-gram precision on shorter candidate sentences): r - length of reference translation c - length of candidate translation

15. Interannotator agreement IAA - measure to tell how good human experts perform when given a specific task (to measure the reliability of manual annotations) e.g. F-measure on data from two annotators (one of them virtually treated as gold standard, symmetric if F1) But: nonzero value is obtained even if annotators' decisions are uncorrelated example –two annotators making classifications into two classes –1st annotator: 80% A, 20% B –2nd annotator 85% A, 15% B –probability of agreement by chance: 0.8*0.85 + 0.2*0.15 = 71% desired measure: 1 if they agree in all decisions, 0 if their agreement is equal to agreement by chance

16. Cohen's Kappa takes into account the agreement occurring by chance P a - relative observed agreement between annotators P e - probability of agreement by chance but kappa -- as a means for quantifying actual level of agreement -- is still a source of much controversy

17. Evaluation rounding Number of significant digits is linked to experiment setting and reflects its result uncertainty. Writing more digits in an answer than justified by the number of digits in the data is bad. Do not say that error rate of your system is 42.8571% if it has made 3 errors in 7 task instances (superfluous precision). Basic rules for rounding: –Multiplication/division - the number of significant digits in an answer should equal the least number of significant digits in any one of the numbers being multiplied/divided. –Addition/subtraction - the number of decimal places (not significant digits) in the answer should be the same as the least number of decimal places in any of the numbers being added or subtracted. But: the number of significant digits in a value provides only a very rough indication of its precision -- better to use confidence interval (e.g. 3.28±.05) at certain probability level (typically at 95%).

18. Towards more robust evaluation K-fold cross validation (usually K=10): –1) partition the data into K roughly equally sized subsamples –2) perform cyclically K iterations use K-1 subsamples for training use 1 subsample for testing –3) average the iterations' results more reliable results, especially if you have only small or in some sense non-uniform data

19. "Shared Tasks" in NLP contests in implementing systems for a specified task some of them quite popular in the NLP community (e.g. CoNLL) conditioned by existence of training and evaluation data and of evaluation metrics Examples: –Message Understanding Conferences (MUCs) 1980-1990's, information retrieval, named entity recognition –Parseval 1991, phrase-structure parsing –Senseval, Semeval word sense disambiguation –WMT Shared Task ACL Workshop in machine translation, 2006-2008 –CoNLLL Shared Task Conference on Computational Natural Language Learning named entity resolution (2003) semantic role labeling (2004,2005) multilingual dependency parsing (2006, 2007) Joint Parsing of Syntactic and Semantic Dependencies - 2008