Fall 2005 Lecture Notes #8 EECS 595 / LING 541 / SI 661 Natural Language Processing

Evaluation of NLP systems

The classical pipeline (for supervised learning) Training set/dev set/test set Dumb baseline Intelligent baseline Your algorithm Human ceiling Accuracy/precision/recall Multiple references Statistical significance

Special cases Document retrieval systems Part of speech tagging Parsing –Labeled recall –Labeled precision –Crossing brackets

Word classes and part-of-speech tagging

Part of speech tagging Problems: transport, object, discount, address More problems: content French: est, président, fils “Book that flight” – what is the part of speech associated with “book”? POS tagging: assigning parts of speech to words in a text. Three main techniques: rule-based tagging, stochastic tagging, transformation-based tagging

Rule-based POS tagging Use dictionary or FST to find all possible parts of speech Use disambiguation rules (e.g., ART+V) Typically hundreds of constraints can be designed manually

Example in French ^ beginning of sentence La rf b nms u article teneur nfs nms noun feminine singular Moyenne jfs nfs v1s v2s v3s adjective feminine singular en p a b preposition uranium nms noun masculine singular des p r preposition rivi`eres nfp noun feminine plural, x punctuation bien_que cs subordinating conjunction délicate jfs adjective feminine singular À p preposition calculer v verb

Sample rules BS3 BI1: A BS3 (3rd person subject personal pronoun) cannot be followed by a BI1 (1st person indirect personal pronoun). In the example: ``il nous faut'' ({\it we need}) - ``il'' has the tag BS3MS and ``nous'' has the tags [BD1P BI1P BJ1P BR1P BS1P]. The negative constraint ``BS3 BI1'' rules out ``BI1P'', and thus leaves only 4 alternatives for the word ``nous''. N K: The tag N (noun) cannot be followed by a tag K (interrogative pronoun); an example in the test corpus would be: ``... fleuve qui...'' (...river, that...). Since ``qui'' can be tagged both as an ``E'' (relative pronoun) and a ``K'' (interrogative pronoun), the ``E'' will be chosen by the tagger since an interrogative pronoun cannot follow a noun (``N''). R V:A word tagged with R (article) cannot be followed by a word tagged with V (verb): for example ``l' appelle'' (calls him/her). The word ``appelle'' can only be a verb, but ``l''' can be either an article or a personal pronoun. Thus, the rule will eliminate the article tag, giving preference to the pronoun.

Confusion matrix INJJNNNNPRBVBDVBN IN-.2.7 JJ NN NNP RB VBD VBN Most confusing: NN vs. NNP vs. JJ, VBD vs. VBN vs. JJ

HMM Tagging T = argmax P(T|W), where T=t 1,t 2,…,t n By Bayes’s theorem: P(T|W) = P(T)P(W|T)/P(W) Thus we are attempting to choose the sequence of tags that maximizes the rhs of the equation P(W) can be ignored P(T)P(W|T) = ? P(T) is called the prior, P(W|T) is called the likelihood.

HMM tagging (cont’d) P(T)P(W|T) = P(w i |w 1 t 1 …w i-1 t i-1 t i )P(t i |t 1 …t i-2 t i-1 ) Simplification 1: P(W|T) = P(w i |t i ) Simplification 2: P(T)= P(t i |t i-1 ) T = argmax P(T|W) = argmax P(w i |t i ) P(t i |t i-1 )    

Estimates P(NN|DT) = C(DT,NN)/C(DT)=56509/ =.49 P(is|VBZ = C(VBZ,is)/C(VBZ)=10073/21627=.47

Example Secretariat/NNP is/VBZ expected/VBN to/TO race/VB tomorrow/NR People/NNS continue/VBP to/TO inquire/VB the/AT reason/NN for/IN the/AT race/NN for/IN outer/JJ space/NN TO: to+VB (to sleep), to+NN (to school)

Example NNPVBZVBNTO VB NR Secretariatisexpectedracetomorrowto NNPVBZVBNTO NN NR Secretariatisexpectedracetomorrowto

Example (cont’d) P(NN|TO) = P(VB|TO) =.83 P(race|NN) = P(race|VB) = P(NR|VB) =.0027 P(NR|NN) =.0012 P(VB|TO)P(NR|VB)P(race|VB) = P(NN|TO)P(NR|NN)P(race|NN) =

Decoding Finding what sequence of states is the source of a sequence of observations Viterbi decoding (dynamic programming) – finding the optimal sequence of tags Input: HMM and sequence of words, output: sequence of states

Transformation-based learning P(NN|race) =.98 P(VB|race) =.02 Change NN to VB when the previous tag is TO Types of rules: –The preceding (following) word is tagged z –The word two before (after) is tagged z –One of the two preceding (following) words is tagged z –One of the three preceding (following) words is tagged z –The preceding word is tagged z and the following word is tagged w