1. Introduction to Statistical Modeling
Rong Jin

2. Why Statistical Modeling?
Vector space model for information retrieval
Both documents and queries are vectors in the term space
Relevance is measured by the similarity between document vectors and query vector
Many problems with vector space model
Ad-hoc term weighting schemes
Ad-hoc basis vectors
Ad-hoc similarity measurement
We need something that is much more principled !

3. A Simple Example (I) Question: how to guess which coin Alex choose?
Consider you have three coins: C1, C2, C3
Alex picked up one of the coins and flipped it six times.
You didn’t see which coin he picked out. But, you observed the results of flipping coins:
t, h, t, h, t, t
Question: how to guess which coin Alex choose?

4. A Simple Example (II)
You experimented with the three coins, say 6 times
C1: h, h, h, t, h, t
C2: t, t, h, t, t, t
C3: t, h, t, t, t, h
Given: t, h, t, h, t, t
Now, what one you think Alex choose?

5. A Simple Example (III) q: t, h, t, h, t, t  bias bq = 1/3
C1: h, h, h, t, h  bias b1 = 5/6
C2: t, t, h, t, t, t  bias b2 = 1/6
C3: t, h, t, t, t, h  bias b3 = 1/3
So, which coin you think Alex select?
A more principled approach:
Compute the likelihood p(q|Ci) for each coin

6. A Simple Example (IV) p(q|C1) = p(t, h, t, h, t, t | C1)
= p(t|C1)*p(h|C1)*p(t|C1)*p(h|C1)*p(t|C1)*p(t|C1)
= 1/6 * 5/6 * 1/6 * 5/6 * 1/6 * 1/6 ~ 5.3*10-4
Compute p(q|C2) and p(q|C3)
Which coin has the largest likelihood ?

7. A Simple Example (IV) p(q|C1) = p(t, h, t, h, t, t | C1)
= p(t|C1)*p(h|C1)*p(t|C1)*p(h|C1)*p(t|C1)*p(t|C1)
= 1/6 * 5/6 * 1/6 * 5/6 * 1/6 * 1/6 ~ 5.3*10-4
Compute p(q|C2) and p(q|C3)
p(q|C2) = 0.013, p(q|C3) = 0.02
Which coin has the largest likelihood ?

8. An Information Retrieval View
Query (q): t, h, t, h, t, t
Doc1(C1): h, h, h, t, h
Doc2(C2): t, t, h, t, t, t
Doc3(C3): t, h, t, t, t, h
Which document is ranked first if we use the vector space model?

9. An Information Retrieval View
Query (q): t, h, t, h, t, t
Doc1(C1): h, h, h, t, h
Doc2(C2): t, t, h, t, t, t
Doc3(C3): t, h, t, t, t, h
Which document is ranked first if we use the vector space model?

10. An Information Retrieval View
Query (q): t, h, t, h, t, t
Doc1(C1): h, h, h, t, h sim(D1) = 1/3*5/6+2/3*1/6 = 0.39
Doc2(C2): t, t, h, t, t, t sim(D2) = 1/3*1/6+2/3*5/6 = 0.61
Doc3(C3): t, h, t, t, t, h sim(D3) = 1/3*1/3+2/3*2/3 = 0.56
Which document is ranked first if we use the vector space model?

11. A Simple Example: Summary
q: t, h, t, h, t, t ? ? ?
C1:h, h, h, t, h
C2:t, t, h, t, h, h
C3: t, h, t, t, t, h

12. A Simple Example: Summary
q: t, h, t, h, t, t
Estimating likelihood p(q|bias)
b2 = 1/2
b3 = 1/3
b1 = 5/6
Estimating bias for each coin
C1:h, h, h, t, h
C2:t, t, h, t, h, h
C3: t, h, t, t, t, h

13. A Probabilistic Framework for Information Retrieval
q: ‘Bush Kerry’
Estimating likelihood p(q| ) ? ? ?
Estimating some statistics  for each document d1 …
d1000

14. A Probabilistic Framework for Information Retrieval
Three fundamental questions
What statistics  should be chosen to describe the characteristics of documents ?
How to estimate this statistics ?
How to compute the likelihood of generating queries given the statistics ?

15. Unigram Language Model
Probabilities for single word p(w)
={p(w) for any word w in vocabulary V}
Estimate an unigram language model
Simple counting
Given a document d, count term frequency c(w,d) for each word w. Then, p(w) = c(w,d)/|d|
How to estimate the likelihood p(q|)?

16. Estimate p(q|) q={w1, w2, …, wk}
Similar to the example of flipping coins
E.g.: q={‘bush’, ‘kerry’}
d={p(‘bush’)=0.001, p(‘kerry’)=0.02}
p(q|d)=0.001 * 0.02 = 2 * 10-5
What if the document didn’t mention word ‘bush’, instead it used phrase ‘president of united states’ ?

17. Estimate p(q|) q={w1, w2, …, wk}
Similar to the example of flipping coins
E.g.: q={‘bush’, ‘kerry’}
d={p(‘bush’)=0.001, p(‘kerry’)=0.02}
p(q|d)=0.001 * 0.02 = 2 * 10-5
What if the document didn’t mention word ‘bush’, instead it used phrase ‘president of united states’ ?

18. Illustration of Language Models for Information Retrieval
q: t, h, t, h, t, t
Estimating likelihood p(q|)=[p(h)]2[p(t)]4
2 = {p(h)=1/2, p(t)=1/2}
1 = {p(h)=1/3, p(t)=2/3}
1 = {p(h)=5/6, p(t)=1/6}
Estimating language models by counting
d1:h, h, h, t, h,h
d2:t, t, h, t, h, h
d3: t, h, t, t, t, h

19. A Simple Example: Summary
q: t, h, t, h, t, t
Estimating likelihood p(q|)=[p(h)]2[p(t)]4
2 = {p(h)=1/2, p(t)=1/2}
1 = {p(h)=1/3, p(t)=2/3}
1 = {p(h)=5/6, p(t)=1/6}
Estimating language models by counting
d1:h, h, h, t, h,h
d2:t, t, h, t, h, h
d3: t, h, t, t, t, h

20. A Simple Example: Summary
q: t, h, t, h, t, t
Estimating likelihood p(q|)=[p(h)]2[p(t)]4
2 = {p(h)=1/6, p(t)=5/6}
3 = {p(h)=1/3, p(t)=2/3}
1 = {p(h)=5/6, p(t)=1/6}
Estimating language models by counting
d1:h, h, h, t, h,h
d2:t, t, h, t, h, h
d3: t, h, t, t, t, h
Problems?

21. Problems With Unigram LM
Unigram probabilities
Insufficient for representing true documents
Simple counting for estimating unigram probabilities
It does not account for variance in documents
If you ask the same person to write the same story twice, it will be different
Most words will have zero probabilities
Sparse data problem

22. Sparse Data Problems Shrinkage Maximum a posterior (MAP) estimation
Bayesian approach

23. Shrinkage: Jelinek Mercer Smoothing
Linearly interpolate between document language model and the collection language model
Estimation based on individual document
Estimation based on the corpus
0 <  < 1: is a smoothing parameter

24. Smoothing & TF-IDF Weighting
Are they totally irrelevant ?

25. Smoothing & TF-IDF Weighting
Similar to TF.IDF weighting
irrelevant to documents