1. A Markov Random Field Model for Term Dependencies
Hongyu Li & Chaorui Chang

2. Background Dependencies exist between terms in a collection of text
Estimating statistical models for general term dependencies is infeasible due to data sparsity
Most work on modeling term dependencies in the past has focused on phrases proximity or term co-occurrences

3. Hypothesis and Solution
Dependence models will be more effective for larger collections than smaller collections
Incorporating several types of evidence into a dependence model will further improve effectiveness
Introducing Markov Random Field

4. Markov Random Field
Also called undirected graph models , model joint distributions
In the paper used to model the joint distribution 𝑃 Λ 𝑄,𝐷 over queries Q and documents D
Assume graph G consists of query nodes qi and a document node D
Joint distribution is defined by
𝑃 Λ 𝑄,𝐷 = 1 𝑍 Λ 𝑐∈𝐶(𝐺) 𝜓(𝑐;Λ)
Where C(G) is the set of cliques in G, each 𝜓(.;Λ) is a non-negative potential function

5. 3 variants of MRF model Full independence
Query terms qi are independent
Sequential dependence
Dependence between neighboring query terms
Full dependence
All query terms are in some way dependent

6. Potential functions Potential function for 2-clique
𝜓 𝑇 𝑐 = 𝜆 𝑇 log 𝑃 𝑞 𝑖 𝐷 = 𝜆 𝑇 log [ 1− 𝛼 𝐷 𝑡𝑓 𝑞 𝑖 ,𝐷 𝐷 + 𝛼 𝐷 𝑐𝑓 𝑞 𝑖 |𝐶| ]
Contiguously sets of query terms within the clique
𝜓 𝑂 𝑐 = 𝜆 𝑂 log 𝑃 #1(𝑞 𝑖 ,…, 𝑞 𝑖+𝑘 ) 𝐷 = 𝜆 𝑂 log [ 1− 𝛼 𝐷 𝑡𝑓 #1(𝑞 𝑖 ,…, 𝑞 𝑖+𝑘 ),𝐷 𝐷 + 𝛼 𝐷 𝑐𝑓 #1(𝑞 𝑖 ,…, 𝑞 𝑖+𝑘 ) |𝐶| ]
Non-contiguous sets of query terms
𝜓 𝑈 𝑐 = 𝜆 𝑈 log 𝑃 #𝑢𝑤N(𝑞 𝑖 ,…, 𝑞 𝑗 ) 𝐷 = 𝜆 𝑈 log [ 1− 𝛼 𝐷 𝑡𝑓 #𝑢𝑤N(𝑞 𝑖 ,…, 𝑞 𝑗 ),𝐷 𝐷 + 𝛼 𝐷 𝑐𝑓 #𝑢𝑤N(𝑞 𝑖 ,…, 𝑞 𝑗 ) |𝐶| ]

7. Ranking Define the ranking function
𝑃 Λ 𝐷|𝑄 = 𝑃 Λ 𝑄,𝐷 𝑃 Λ 𝑄 𝑟𝑎𝑛𝑘 log 𝑃 Λ 𝑄,𝐷 −𝑙𝑜𝑔 𝑃 Λ 𝑄
Potential function can be parameterized as 𝜓 𝑐;Λ =exp⁡( 𝜆 𝑐 𝑓 𝑐 )
𝑃 Λ 𝐷|𝑄 𝑟𝑎𝑛𝑘 𝑐∈𝐶(𝐺) 𝜆 𝑐 𝑓(𝑐) = 𝑐∈𝑇 𝜆 𝑇 𝑓 𝑇 (𝑐) + 𝑐∈𝑂 𝜆 𝑂 𝑓 𝑂 𝑐 + 𝑐∈𝑂∪𝑈 𝜆 𝑈 𝑓 𝑈 (𝑐)

8. Training Set parameter values (𝜆 𝑇 , 𝜆 𝑂 , 𝜆 𝑈 )
Train the model by directly maximizing mean average precision
Ranking function is invariant to parameter scale, thus
𝜆 𝑇 + 𝜆 𝑂 + 𝜆 𝑈 =1
Example mean average precision surface for the GOV2 collection using the full dependence model

9. 3.Experimental Results analyze the retrieval effectiveness across different collections
Journal &Press :small homogeneous collections
Web Collections: larger and less homogeneous

10. 3.1 Full Independence variant
the cliques are only members of the set T , and therefore we set 𝜆 𝑂 = 𝜆 𝑈 = 0, 𝜆 𝑇 = 1.
Ranking function :

11. AvgP refers to mean average precision, is precision at 10 ranked documents, and µ is the smoothing parameter used.
This results provide a baseline to compare the sequential and full dependence variants
Full independence variant results.

12. 3.2 Sequential Dependence variant
Models of this form have cliques in T , O, and U. Ranking function :
The unordered feature function, 𝑓 𝑈 , has a free parameter N that allows the size of the unordered window (scope of proximity) to vary.
We explore window sizes of 2, sentence(8), 50, and “unlimited” to see what impact they have on effectiveness.

13. Show very little difference across the various window sizes.
For the AP, WT10g, and GOV2 collection, the sentence-sized windows performed the best. For the WSJ collection, N = 2 performed the best.
The sequential dependence variant outperforms the full independence variant
sequential dependence variant results

14. 3.3 Full Dependence variant
Consists of cliques in T , O and U. ranking function :
We set the parameter N in the feature function 𝑓 𝑈 to be four times the number of query terms in the clique c.
We analyze the impact ordered and unordered window feature functions have on effectiveness.

15. AP collection, there is very little difference .
The results for the WSJ collection the ordered features produce a clear improvement over the unordered features, but there is very little difference between using ordered features and the combination of ordered and unordered.
The results for the two web collections, WT10g and GOV2, are similar. In both, unordered features perform better than ordered features, but the combination of both ordered and unordered features led to noticeable improvements in mean average precision.
full dependence variant results

16. Strict matching via ordered window features is more important for the smaller newswire collections, due to the homogeneous, clean nature of the documents
For the web collections, the opposite is true.

17. 4. CONCLUSIONS
Three dependence model variants are described, where each captures different dependencies between query terms.
Modeling dependencies can significantly improve retrieval effectiveness across a range of collections.
Possible future work includes exploring a wider range of potential functions, applying the model to other retrieval tasks and so on.

18. thank you