Carnegie Mellon Exact Maximum Likelihood Estimation for Word Mixtures Yi Zhang & Jamie Callan Carnegie Mellon University Wei Xu NEC C&C Research Lab

Carnegie Mellon Outline u Introduction 1. Why this problems? some retrieval applications 2. Traditional solutions: EM algorithm u New algorithm: exact MLE estimation u Experimental Results

Carnegie Mellon Query Q Document D Results Feedback Docs F={d 1, d 2, …, d n } Example 1: Model-based Feedback in the Language Modeling Approach to IR Based on Zhai&Lafferty’s slides in CIKM 2001

Carnegie Mellon  F Estimation based on Generative Mixture Model w w F={d 1, …, d n } Maximum Likelihood P(w|  ) P(w| C) 1- P(source) Background words Topic words Based on Zhai&Lafferty’s slides in CIKM 2001 Given: F, P(w|c) and Find: MLE of 

Carnegie Mellon M T :  Topic M E :  general English M I :  new E T new Example 2: Model-based Approach for Novelty Detection in Adaptive Information Filtering Based on Zhang&Callan’s paper in SIGIR 2002 Given:  general English,  Topic E T new Find: MLE of  new

Carnegie Mellon Problem Setting and Traditional Solution Using EM u Observe: data generated by a mixture multinomial distribution r=(r 1, r 2, r 3, …, r k ) u Given: interpolation weights  and , another multinomial distribution p=(p 1, p 2, p 3, …, p k ) u Find: the maximum likelihood estimation (MLE) of multinomial distribution q=(q 1, q 2, q 3, …, q k ) u Traditional solution: EM algorithm l Iterative process which can be computationally expensive l Only provide approximate solution

Carnegie Mellon Finding q (1) Under the constraints: Where: f i is observed frequency of word i

Carnegie Mellon Finding q (2) For all the q i such that q i >0, apply Lagrange multiplier method and calculate the derivatives with respect to q i : This is a close form solution for q i, if we know all i that q i >0. Theorem: All the q i greater than 0 correspond to the smallest See detailed proof in our paper…

Carnegie Mellon Algorithm for Finding Exact MLE for q

Carnegie Mellon Experiments Setting on Model Based Feedback in IR u 20 relevant documents (sampled from AP Wire News and Wall Street Journal dataset from ) for a topic as observed training data sequence. p is calculated directly as described in (Zhai&Lafferty) from documents. u There are 2352 unique words in these 20 relevant documents, which means at most 2352 q i 's are none zero, while there are p i 's are none zero.

Carnegie Mellon EM result converges to the result calculated directly by our algorithm.

Carnegie Mellon Compar ing the Speed of Our Algorithm With EM u EM stop if change of LL < 10 -  u times on PIII 500 PC

Carnegie Mellon Conclusion u We developed a new training algorithm that provide exact MLE for word mixtures u Theoretically and Empirically works well u Can be used in several language model based IR applications