1. N-gram model limitations Important question was asked in class: what do we do about N-grams which were not in our training corpus? Answer given: we distribute some probability mass from seen N-grams to this new N- gram. This leads to another question: how do we do this?

2. Unsmoothed bigrams Recall that we use unigram and bigram counts to compute bigram probabilities: – P(w n |w n-1 ) = C(w n-1 w n ) / C(w n-1 )

3. Recall exercise from last class Suppose text had N words, how many bigrams (tokens) does it contain? At most N: we assume appearing before first word to get a bigram probability for the word in the initial position. Example (5 words): – words: w1 w2 w3 w4 w5 – bigrams: w1, w1 w2, w2 w3, w3 w4, w4 w5

4. How many possible bigrams are there? With a vocabulary of N words, there are N 2 possible bigrams.

5. Example description Berkeley Restaurant Project corpus – approximately 10,000 sentences – 1616 word types – tables will show counts or probabilities for 7 word types, carefully chosen so that the 7 by 7 matrix is not too sparse – notice that many counts in first table are zero (25 zeros of 49 entries)

6. Unsmoothed N-grams IwanttoeatChinesefoodlunch I81087013000 want307860686 to30108603012 eat002019252 Chinese200001201 food190170000 lunch4000010 Bigram counts (figure 6.4 from text)

7. Computing probabilities Recall formula (we normalize by unigram counts): – P(w n |w n-1 ) = C(w n-1 w n ) / C(w n-1 ) Unigram counts are: WORDIwanttoeatChinesefoodlunch UNIGRAM COUNT 3437121532569382131506459 p( eat | to ) = c( to eat ) / c( to ) = 860 / 3256 =.26 p( to | eat ) = c( eat to ) / c(eat) = 2 / 938 =.0021

8. Unsmoothed N-grams w n-1 w n IwanttoeatChinesefoodlunch I.0023.320.0038000 want.00250.640.0049.0066.0049 to.000920.0031.26.000920.0037 eat00.00210.02.0021.055 Chinese.00940000.56.0047 food.0130.0110000 lunch.00870000.00220 Bigram probabilities (figure 6.5 from text): p( w n | w n-1 )

9. What do zeros mean? Just because a bigram has a zero count or a zero probability does not mean that it cannot occur – it just means it didn’t occur in the training corpus. So we arrive back at our question: what do we do with bigrams that have zero counts when we encounter them?

10. Let’s rephrase the question How can we ensure that none of the possible bigrams have zero counts/probabilities? Process of spreading the probability mass around to all possible bigrams are called smoothing. We start with a very simple model: add-one smoothing.

11. Add-one smoothing counts New counts are gotten by adding one to original counts across the board. This ensures that there are no zero counts, but typically adds to much probability mass to non-occurring bigrams.

12. Add-one smoothing probabilities Unadjusted probabilities: – P(w n |w n-1 ) = C(w n-1 w n ) / C(w n-1 ) Adjusted probabilities: – P*(w n |w n-1 ) = [ C(w n-1 w n ) + 1 ] / [ C(w n-1 ) + V ] V is total number of word types in vocabulary In numerator we add one to the count of each bigram – as with the plain counts. In denominator we add V, since we are adding one more bigram token of the form w n-1 w, for each w in our vocabulary

13. A simple approach to smoothing: Add-one smoothing IwanttoeatChinesefoodlunch I91088114111 want417871797 to41118614113 eat113120353 Chinese311111212 food201181111 lunch5111121 Add-one smoothed bigram counts (figure 6.6 from text)

14. Calculating the probabilities Recall the formula for the adjusted probabilities: – P*(w n |w n-1 ) = [ C(w n-1 w n ) + 1 ] / [ C(w n-1 ) + V ] Unigram counts (adjusted by adding V=1616): WORDIwanttoeatChinesefoodlunch UNIGRAM COUNT 5053293148722554182931222075 p( eat | to ) = c( to eat ) / c( to ) = 861 / 4872 =.18 (was.26) p( to | eat ) = c( eat to ) / c( eat ) = 3 / 2554 =.0012 (was.0021) p( eat | lunch ) = c( lunch eat ) / c( lunch ) = 1 / 2075 =.00048 (was 0) p( eat | want ) = c( want eat ) / c( want ) = 1 / 2931 =.00034 (was 0)

15. A simple approach to smoothing: Add-one smoothing IwanttoeatChinesefoodlunch I.0018.22.0002.0028.0002 want.0014.00035.28.00035.0025.0032.0025 to.00082.00021.0023.18.00082.00021.0027 eat.00039.0012.00039.0078.0012.021 Chinese.0016.00055.066.0011 food.0064.00032.0058.00032 lunch.0024.00048.00096.00048 Add-one smoothed bigram probabilities (figure 6.7 from text)

16. Discounting We can define the discount to be the ratio of new and old counts (in our case smoothed and unsmoothed counts). Discounts for add-one smoothing for this example: WORDIwanttoeatChinesefoodlunch ADD-ONE DISCOUNT.68.42.69.37.12.48.22

17. Witten-Bell discounting Another approach to smoothing Basic idea: “Use the count of things you’ve seen once to help estimate the count of things you’ve never seen.” [p. 211] Total probability mass assigned to all (as yet) unseen bigrams is T / [ T + N ], where – T is the total number of observed types – N is the number of tokens “We can think of our training corpus as a series of events; one event for each token and one event for each new type.” [p. 211] Formula above estimates “the probability of a new type event occurring.” [p. 211]

18. Distribution of probability mass This probability mass is distributed evenly amongst the unseen bigrams. Z = number of zero-count bigrams. p i * = T / [ Z*(N + T) ]

19. Discounting This probability mass has to come from somewhere! p i * = c i / (N + T) if c i > 0 Smoothed counts are c i * = T/Z * N/(N+T) if c i = 0 (work back from probability formula) c i * = c i * N/(N+T) if c i > 0

20. Witten-Bell discounting IwanttoeatChinesefoodlunch I81060.06213.062 want3.046740.046686 to3.085108273.08512 eat.075 2 17246 Chinese2.012 1091 food18.05916.059 lunch4.026 1 Witten-Bell smoothed (discounted) bigram counts (figure 6.9 from text)

21. Discounting comparison Table shows discounts for add-one and Witten-Bell smoothing for this example: WORDIwanttoeatChinesefoodlunch ADD-ONE DISCOUNT.68.42.69.37.12.48.22 WITTEN- BELL DISCOUNT.97.94.96.88.91.94.91

22. Training sets and test sets Corpus divided into training set and test set Need test items to not be in training set, else they will have artificially high probability Can use this to evaluate different systems: – train two different systems on the same training set – compare performance of systems on the same test set