1. N-Gram Language Models
ChengXiang Zhai
Department of Computer Science
University of Illinois, Urbana-Champaign

2. Outline General questions to ask about a language model
N-gram language models
Special case: Unigram language models
Smoothing Methods

3. Central Questions to Ask about a LM: “ADMI”
Application: Why do you need a LM? For what purpose?
Data: What kind of data do you want to model?
Model: How do you define the model?
Inference: How do you infer/estimate the parameters?
Evaluation metric for a LM
Speech recognition
Perplexity
Speech text data
Limited memory
N-gram LM
Smoothing methods
Data set for estimation & evaluation
Assumptions to be made
Inference/Estimation algorithm

4. Central Question in LM: p(w1w2…wm|C)=?
What is C? We usually ignore C (=“context”) since it depends on the application, but it’s important to consider it when applying a LM
Refinement 1: p(w1w2…wm|C)  p(w1w2…wm)
What random variables are involved? What is the event space?
What event does “w1w2…wm” represent? What is the sample space?
p(w1w2…wm) =p(X=w1w2…wm) vs. p(X1=w1,X2=w2,…Xm=wm)?
X=?
X1=?
X2=?
Xm=?

5. Central Question in LM: p(w1w2…wm|C)=?
Refinement 2: p(w1w2…wm) = p(X1=w1,X2=w2,…Xm=wm)
What assumption have we made here?
Chaining Rule: p(w1w2…wm)= p(w1)p(w2|w1)...p(wm|w1w2…wm-1)
What about p(w1w2…wm)= p(wm)p(wm-1|wm)...p(w1|w2…wm)?
Refinement 3: Assume limited dependence (only depends on the n previous words)  N-gram LM
p(X1=w1,X2=w2,…Xm=wm)p(X1=w1) p(X2=w2|X1=w1)…p(Xn=wn|X1=w1,…,Xn-1=wn-1) …
p(w1w2…wm)  p(w1) p(w2|w1)…p(wn|w1,…,wn-1)… p(wm|wm-n+1,…,wm-1)

6. Key Assumption in N-gram LM:
Ignored
p(wm|w1,…, wm-n+1…,wm-1)=p(wm|wm-n+1,…,wm-1)
Does this assumption hold?

7. Estimation of N-Gram LMs
Text Data: D
Question: p(wm|wm-n+1,…,wm-1)=?
Boils down to estimate p(w1,w2,…,wm), ML estimate is:
P(X|Y)=p(X,Y)/p(Y)
p(wm-n+1,…,wm-1,wm)
p(wm|wm-n+1,…,wm-1) =
p(wm-n+1,…,wm-1)
c(w1w2…wm,D)
Count of word sequence “w1w2…wm”
p(w1,w2,…,wm) =
𝒖 𝒊 ∈𝑽 𝒄( 𝒖 𝟏 𝒖 𝟐 … 𝒖 𝒎 ,𝑫)
Total counts of all word sequences of length m

8. ML Estimate of N-Gram LM
c(wm-n+1…wm-1wm, D)
p(wm|wm-n+1,…,wm-1) =
𝒖∈𝑽 𝒄( 𝒘 𝒎−𝒏+𝟏 … 𝒘 𝒎−𝟏 𝒖,𝑫)
Count of long word sequences may be zero!
Not accurate
Cause problems when computing the conditional probability p(wm|wm-n+1,…,wm-1)
Solution: smoothing
Key idea: backoff to shorter N-grams, eventually to unigrams
Treat shorter N-gram models as prior in Bayesian estimation

9. Special Case of N-Gram LM: Unigram LM
Generate text by generating each word INDEPENDENTLY
p(wm|w1,…, wm-n+1…,wm-1)=p(wm): History didn’t matter!
How to estimate a unigram LM?
Text data: d
Maximum Likelihood estimator:
Count of word w in d
pML(w|d) = 𝑐(𝑤,𝑑) 𝑤′∈𝑉 𝑐( 𝑤 ′ ,𝑑)
Total Counts of all words in d

10. Unigram Language Model Smoothing (Illustration)
p(w|d)
Max. Likelihood Estimate
Smoothed LM w
Word

11. How to Smooth? All smoothing methods try to
discount the probability of words seen in a text data set
re-allocate the extra counts so that unseen words will have a non-zero count
Method 1: Additive smoothing: Add a constant  to the counts of each word, e.g., “add 1”
Count of w in d
“Add one”, Laplace
Vocabulary size
Length of d (total counts)

12. Improve Additive Smoothing
Should all unseen words get equal probabilities?
We can use a reference model to discriminate unseen words
Discounted ML estimate
Reference language model
Normalizer
Prob. Mass for unseen words

13. However, how do we define p(w|REF)?
p(w|REF): Reference Language Model
What do we know about those unseen words?
Why are there unseen words?
Zipf’s law: most words occur infrequently in text (e.g., just once)
Unseen words are non-relevant to a topic
Unseen words are relevant, but the text data sample isn’t large enough to include them
The context variable C in p(w1w2…wm|C) can provide a basis for defining p(w|REF)
E.g., in retrieval, p(w|Collection) can serve as p(w|REF) for estimating a language model for an individual document p(w|d)

14. Interpolation vs. Backoff
Interpolation: view p(w|REF) as a prior and the actual counts as observed evidence
Backoff (Katz-Backoff): if the count is sufficiently high (sufficient evidence), we’d trust the ML estimate, otherwise, we simply ignore the ML estimate and go for p(w|REF)
𝑝 𝑤 𝑑 = 𝛽 𝑐(𝑤,𝑑) |𝑑| 𝜆𝑝(𝑤|𝑅𝐸𝐹)
𝑖𝑓 𝑐 𝑤,𝑑 >𝑘
otherwise

15. Smoothing Methods based on Interpolation
Method 2: Absolute discounting (Kneser-Ney Smoothing): Subtract a constant  from the count of each word
Method 3: Linear interpolation (Jelinek-Mercer smoothing): “Shrink” uniformly toward p(w|REF)
# unique words
parameter
ML estimate

16. Smoothing Methods based on Interpolation (cont.)
Method 4 Dirichlet Prior/Bayesian (McKay): Assume pseudo counts p(w|REF)
parameter
What would happen if we increase/decrease ? What if |d|  +

17. Linear Interpolation (Jelinek-Mercer) Smoothing
Document d
the 0.1 a ..
computer 0.02
database 0.01 ……
text 0.001
network 0.001
mining …
Collection LM
P(w|C)
Unigram LM p(w|)=?
Total #words=100 …
text ?
mining ?
association ?
database ?
query ?
network?
text 10
mining 5
association 3
database 3
algorithm 2 …
query 1
efficient 1
10/100
5/100
3/100
1/100
0/100
𝜆∈[0,1]

18. Dirichlet Prior (Bayesian) Smoothing
the 0.1 a ..
computer 0.02
database 0.01 ……
text 0.001
network 0.001
mining …
Collection LM
P(w|C)
Document d
Unigram LM p(w|)=?
Total #words=100 …
text ?
mining ?
association ?
database ?
query ?
network?
text 10
mining 5
association 3
database 3
algorithm 2 …
query 1
efficient 1
10/100
5/100
3/100
1/100
0/100
 ∈[0,+)

19. Dirichlet Prior Smoothing (Bayesian Smoothing)
Bayesian estimator of multinomial distribution (unigram LM)
First consider posterior of parameters: p(|d) p(d| )p()
Then, consider the mean or mode of the posterior distribution
Sampling distribution(of data): p(d|)
Prior (on model parameters): P()=p(1 ,…, N), where 1 is probability of the i-th word in the vocabulary
Conjugate Prior: intuitive & mathematically convenient
“encode” the prior as “extra pseudo counts,” which can be conveniently combined with the observed actual counts
p(d| ) and p() have the same functional form

20. Dirichlet Prior Smoothing
Dirichlet distribution is a conjugate prior for multinomial sampling distribution
“pseudo” word counts i= p(wi|REF) X

21. Dirichlet Prior Smoothing (cont.)
Posterior distribution of parameters:
The predictive distribution is the same as the mean:
Dirichlet prior smoothing

22. Good Turing Smoothing
Key Idea: Assume total # unseen events to be n1 (# of singletons), and adjust all the seen events in the same way
Adjusted count
Sum of counts of all terms that occurred c(w,d)+1 times
Share the counts among all the words that occurred c(w,d) times
Heuristics are needed

23. Smoothing of p(wm|w1,…, wm-n+1…,wm-1)
c(wm,wm-n+1,…,wm-1; D)
What if
this is zero?
p(wm|wm-n+1,…,wm-1) =
𝒖∈𝑽 𝒄(𝒖, 𝒘 𝒎−𝒏+𝟏 ,…, 𝒘 𝒎−𝟏 ;𝑫)
How should we define p(w|REF)?
In general, p(w|REF) can be defined based on any “clues” from the history h=(wm-n+1,…,wm-1)
Most natural: p(w|REF)=p(wm|wm-n+2,…,wm-1), ignore wm-n+1 ; can be done recursively to rely on shorter and shorter history
In general, relax the condition to make it less specific so as to increase the counts we can collect (e.g., shorten the history, cluster the history)

24. What You Should Know
What is an N-gram language model? What assumptions are made in an N-gram language model? What are the events involved?
How to compute ML estimate of an N-gram language model?
Why do we need to do smoothing in general?
Know the major smoothing methods and how they work: additive smoothing, absolute discount, linear interpolation (fixed coefficient), Dirichlet prior, Good Turing
Know the basic idea of deriving Dirichlet prior smoothing