1. CS590I: Information Retrieval
Retrieval Models: Language models
Luo Si
Department of Computer Science
Purdue University

2. Retrieval Model: Language Model
Introduction to language model
Unigram language model
Document language model estimation
Maximum Likelihood estimation
Maximum a posterior estimation
Jelinek Mercer Smoothing
Model-based feedback

3. Language Models: Motivation
Vector space model for information retrieval
Documents and queries are vectors in the term space
Relevance is measure by the similarity between document vectors and query vector
Problems for vector space model
Ad-hoc term weighting schemes
Ad-hoc similarity measurement
No justification of relationship between relevance and similarity
We need more principled retrieval models…

4. Introduction to Language Models:
Language model can be created for any language sample
A document
A collection of documents
Sentence, paragraph, chapter, query…
The size of language sample affects the quality of language model
Long documents have more accurate model
Short documents have less accurate model
Model for sentence, paragraph or query may not be reliable

5. Introduction to Language Models:
A document language model defines a probability distribution over indexed terms
E.g., the probability of generating a term
Sum of the probabilities is 1
A query can be seen as observed data from unknown models
Query also defines a language model (more on this later)
How might the models be used for IR?
Rank documents by Pr( | )
Rank documents by language models of and based on kullback-Leibler (KL) divergence between the models (come later)

6. Language Model for IR: Example
Generate retrieval results
sport, basketball
Estimate the generation probability of Pr( | )
Language Model for
Language Model for
Language Model for
Estimating language model for each document
sport, basketball, ticket, sport
stock, finance, finance, stock
basketball, ticket, finance, ticket, sport

7. Language Models Three basic problems for language models
What type of probabilistic distribution can be used to construct language models?
How to estimate the parameters of the distribution of the language models?
How to compute the likelihood of generating queries given the language modes of documents?

8. Multinomial/Unigram Language Models
Language model built by multinomial distribution on single terms (i.e., unigram) in the vocabulary
Examples:
Five words in vocabulary (sport, basketball, ticket, finance, stock)
For a document , its language mode is:
{Pi(“sport”), Pi(“basketball”), Pi(“ticket”), Pi(“finance”), Pi(“stock”)}
Formally:
The language model is: {Pi(w) for any word w in vocabulary V}

9. Multinomial/Unigram Language Models
Multinomial Model for
Multinomial Model for
Multinomial Model for
Estimating language model for each document
sport, basketball, ticket, sport
basketball, ticket, finance, ticket, sport
stock, finance, finance, stock

10. Maximum Likelihood Estimation (MLE)
Find model parameters that make generation likelihood reach maximum:
M*=argmaxMPr(D|M)
There are K words in vocabulary, w1...wK (e.g., 5)
Data: one document with counts tfi(w1), …, tfi(wK), and length | |
Model: multinomial M with parameters {pi(wk)}
Likelihood: Pr( | M)
M*=argmaxMPr( |M)

11. Maximum Likelihood Estimation (MLE)
Use Lagrange multiplier approach Set partial derivatives to zero Get maximum likelihood estimate

12. Maximum Likelihood Estimation (MLE)
(psp, pb, pt, pf, pst) =
(0.5,0.25,0.25,0,0)
(psp, pb, pt, pf, pst) =
(0.2,0.2,0.4,0.2,0)
(psp, pb, pt, pf, pst) =
(0,0,0,0.5,0.5)
Estimating language model for each document
sport, basketball, ticket, sport
basketball, ticket, finance, ticket, sport
stock, finance, finance, stock

13. Maximum Likelihood Estimation (MLE)
Assign zero probabilities to unseen words in small sample
A specific example:
Only two words in vocabulary (w1=sport, w2=business) like (head, tail) for a coin; A document generates sequence of two words or draw a coin for many times
Only observe two words (flip the coin twice) and MLE estimators are:
“business sport” Pi(w1)=0.5
“sport sport” Pi(w1)=1 ?
“business business” Pi(w1)=0 ?

14. Maximum Likelihood Estimation (MLE)
A specific example:
Only observe two words (flip the coin twice) and MLE estimators are:
“business sport” Pi(w1)*=0.5
“sport sport” Pi(w1)*=1 ?
“business business” Pi(w1)*=0 ?
Data sparseness problem

15. Solution to Sparse Data Problems
Maximum a posterior (MAP) estimation
Shrinkage
Bayesian ensemble approach

16. Maximum A Posterior (MAP) Estimation
Maximum A Posterior Estimation:
Select a model that maximizes the probability of model given observed data
M*=argmaxMPr(M|D)=argmaxMPr(D|M)Pr(M)
Pr(M): Prior belief/knowledge
Use prior Pr(M) to avoid zero probabilities
A specific examples:
Only two words in vocabulary (sport, business)
For a document :
Prior Distribution

17. Maximum A Posterior (MAP) Estimation
Maximum A Posterior Estimation:
Introduce prior on the multinomial distribution
Use prior Pr(M) to avoid zero probabilities, most of coins are more or less unbiased
Use Dirichlet prior on p(w)
Hyper-parameters
Constant for pK
(x) is gamma function

18. Maximum A Posterior (MAP) Estimation
For the two word example:
a Dirichlet prior
P(w1)2 (1-P(w1)2)

19. Maximum A Posterior (MAP) Estimation
M*=argmaxMPr(M|D)=argmaxMPr(D|M)Pr(M)
Pseudo Counts

20. Maximum A Posterior (MAP) Estimation
A specific example:
Only observe two words (flip a coin twice):
“sport sport” Pi(w1)*=1 ?
P(w1)2 (1-P(w1)2)
times

21. Maximum A Posterior (MAP) Estimation
A specific example:
Only observe two words (flip a coin twice):
“sport sport” Pi(w1)*=1 ?

22. MAP Estimation Unigram Language Model
Maximum A Posterior Estimation:
Use Dirichlet prior for multinomial distribution
How to set the parameters for Dirichlet prior

23. MAP Estimation Unigram Language Model
Maximum A Posterior Estimation:
Use Dirichlet prior for multinomial distribution
There are K terms in the vocabulary:
Hyper-parameters
Constant for pK

24. MAP Estimation Unigram Language Model
MAP Estimation for unigram language model:
Use Lagrange Multiplier; Set derivative to 0
Pseudo counts set by hyper-parameters

25. MAP Estimation Unigram Language Model
MAP Estimation for unigram language model:
Use Lagrange Multiplier; Set derivative to 0
How to determine the appropriate value for hyper-parameters?
When nothing observed from a document
What is most likely pi(wk) without looking at the content of the document?

26. MAP Estimation Unigram Language Model
MAP Estimation for unigram language model:
What is most likely pi(wk) without looking at the content of the document?
The most likely pi(wk) without looking into the content of the document d is the unigram probability of the collection:
{p(w1|c), p(w2|c),…, p(wK|c)}
Without any information, guess the behavior of one member on the behavior of whole population
Constant

27. MAP Estimation Unigram Language Model
MAP Estimation for unigram language model:
Use Lagrange Multiplier; Set derivative to 0
Pseudo counts
Pseudo document length

28. Maximum A Posterior (MAP) Estimation
Dirichlet MAP Estimation for unigram language model:
Step 0: compute the probability on whole collection based collection unigram language model
Step 1: for each document , compute its smoothed unigram language model (Dirichlet smoothing) as

29. Maximum A Posterior (MAP) Estimation
Dirichlet MAP Estimation for unigram language model:
Step 2: For a given query ={tfq(w1),…, tfq(wk)}
For each document , compute likelihood
The larger the likelihood, the more relevant the document is to the query

30. Dirichlet Smoothing & TF-IDF?
TF-IDF Weighting:

31. Dirichlet Smoothing & TF-IDF
TF-IDF Weighting:

32. Dirichlet Smoothing & TF-IDF

33. Dirichlet Smoothing & TF-IDF
Irrelevant part
TF-IDF Weighting:

34. Dirichlet Smoothing & TF-IDF
Look at the tf.idf part

35. Dirichlet Smoothing Hyper-Parameter
When is very small, approach MLE estimator
When is very large, approach probability on whole collection
How to set appropriate ?

36. Dirichlet Smoothing Hyper-Parameter
Leave One out Validation:
... wj w1
Leave w1 out
...
Leave wj out
...

37. Dirichlet Smoothing Hyper-Parameter
Leave One out Validation:
Leave all words out one by one for a document
... wj w1
Do the procedure for all documents in a collection
Find appropriate

38. Dirichlet Smoothing Hyper-Parameter
What type of document/collection would get large ?
Most documents use similar vocabulary and wording pattern as the whole collection
What type of document/collection would get small ?
Most documents use different vocabulary and wording pattern than the whole collection

39. Shrinkage
Maximum Likelihood (MLE) builds model purely on document data and generates query word
Model may not be accurate when document is short (many unseen words)
Shrinkage estimator builds more reliable model by consulting more general models (e.g., collection language model)
Example: Estimate P(Lung_Cancer|Smoke)
West Lafayette
Indiana
U.S.

40. Shrinkage Jelinek Mercer Smoothing
Assume for each word, with probability , it is generated from document language model (MLE), with probability , it is generated from collection language model (MLE)
Linear interpolation between document language model and collection language model
JM Smoothing:

41. Shrinkage Relationship between JM Smoothing and Dirichlet Smoothing

42. Model Based Feedback
Equivalence of retrieval based on query generation likelihood and Kullback-Leibler (KL) Divergence between query and document language models
Kullback-Leibler (KL) Divergence between two probabilistic distributions
It is the distance between two probabilistic distributions
It is always larger than zero
How to prove it ?

43. Model Based Feedback
Equivalence of retrieval based on query generation likelihood and Kullback-Leibler (KL) Divergence between query and document language models
Loglikelihood of query generation probability
Document independent constant
Generalize query representation to be a distribution (fractional term weighting)

44. Estimating document language model Estimating language model
Model Based Feedback
Calculate KL Divergence
Retrieval results
Estimating query language model
Language Model for
Estimating document language model
Estimating language model
Language Model for
Estimate the generation probability of Pr( | )
Retrieval results

45. Estimating document language model
Model Based Feedback
Feedback Documents from initial results
Retrieval results
Language Model for
Calculate KL Divergence
Estimating document language model
Language Model for
New Query Model
Language Model for
Estimating query language model
No feedback
Full feedback

46. Model Based Feedback: Estimate
Assume there is a generative model to produce each word within feedback document(s)
For each word in feedback document(s), given 
Background model
1-
Feedback Documents
PC(w) w
Flip a coin
Topic words 
qF(w) w

47. Model Based Feedback: Estimate
For each word, there is a hidden variable telling which language model it comes from
the 0.12
to 0.05
it 0.04
a 0.02 …
sport
basketball
Feedback
Documents
Background
Model
pC(w|C)
1-=0.8
MLE
Estimator
Unknown
query topic
p(w|F)=?
“Basketball” …
sport =?
basketball =?
game =?
player =?
=0.2
If we know the value of hidden variable of each word ...

48. Model Based Feedback: Estimate
For each word, the hidden variable Zi = {1 (feedback), 0 (background}
Step1: estimate hidden variable based current on model parameter (Expectation)
E-step
the (0.1) basketball (0.7) game (0.6) is (0.2) ….
Step2: Update model parameters based on the guess in step1 (Maximization)
M-Step

49. Model Based Feedback: Estimate
Expectation-Maximization (EM) algorithm
Step 0: Initialize values of
Step1: (Expectation)
Step2: (Maximization)
Give =0.5

50. Model Based Feedback: Estimate
Properties of parameter 
If  is close to 0, most common words can be generated from collection language model, so more topic words in query language mode
If  is close to 1, query language model has to generate most common words, so fewer topic words in query language mode

51. Retrieval Model: Language Model
Introduction to language model
Unigram language model
Document language model estimation
Maximum Likelihood estimation
Maximum a posterior estimation
Jelinek Mercer Smoothing
Model-based feedback