Stochastic Collapsed Variational Bayesian Inference for Latent Dirichlet Allocation James Foulds 1, Levi Boyles 1, Christopher DuBois 2 Padhraic Smyth 1, Max Welling 3 1 University of California Irvine, Computer Science 2 University of California Irvine, Statistics 3 University of Amsterdam, Computer Science

Let’s say we want to build an LDA topic model on Wikipedia

LDA on Wikipedia 1 hour6 hours 12 hours 10 mins

LDA on Wikipedia 1 hour6 hours 12 hours 10 mins

LDA on Wikipedia 1 full iteration = 3.5 days! 1 hour6 hours 12 hours 10 mins

LDA on Wikipedia Stochastic variational inference 1 hour6 hours 12 hours 10 mins

LDA on Wikipedia Stochastic collapsed variational inference 1 hour6 hours 12 hours 10 mins

Available tools VB Collapsed Gibbs Sampling Collapsed VB BatchBlei et al. (2003) Griffiths and Steyvers (2004) Teh et al. (2007), Asuncion et al. (2009) Stochastic Hoffman et al. (2010, 2013) Mimno et al. (2012) (VB/Gibbs hybrid) ???

Available tools VB Collapsed Gibbs Sampling Collapsed VB BatchBlei et al. (2003) Griffiths and Steyvers (2004) Teh et al. (2007), Asuncion et al. (2009) Stochastic Hoffman et al. (2010, 2013) Mimno et al. (2012) (VB/Gibbs hybrid) ???

Outline Stochastic optimization Collapsed inference for LDA New algorithm: SCVB0 Experimental results Discussion

Stochastic Optimization for ML Batch algorithms – While (not converged) Process the entire dataset Update parameters Stochastic algorithms – While (not converged) Process a subset of the dataset Update parameters

Stochastic Optimization for ML Batch algorithms – While (not converged) Process the entire dataset Update parameters Stochastic algorithms – While (not converged) Process a subset of the dataset Update parameters

Stochastic Optimization for ML Stochastic gradient descent – Estimate the gradient Stochastic variational inference (Hoffman et al. 2010, 2013) – Estimate the natural gradient of the variational parameters Online EM (Cappe and Moulines, 2009) – Estimate E-step sufficient statistics

Collapsed Inference for LDA Marginalize out the parameters, and perform inference on the latent variables only – Simpler, faster and fewer update equations – Better mixing for Gibbs sampling – Better variational bound for VB (Teh et al., 2007)

A Key Insight VBStochastic VB Document parameters Update after every document

A Key Insight VBStochastic VB Document parameters Collapsed VB Stochastic Collapsed VB Word parameters Update after every word? Update after every document

Collapsed Inference for LDA Collapsed Variational Bayes (Teh et al., 2007) K-dimensional discrete variational distributions for each token Mean field assumption

Collapsed Gibbs sampler Collapsed Inference for LDA

Collapsed Gibbs sampler CVB0 (Asuncion et al., 2009) Collapsed Inference for LDA

CVB0 Statistics Simple sums over the variational parameters

Stochastic Optimization for ML Stochastic gradient descent – Estimate the gradient Stochastic variational inference (Hoffman et al. 2010, 2013) – Estimate the natural gradient of the variational parameters Online EM (Cappe and Moulines, 2009) – Estimate E-step sufficient statistics Stochastic CVB0 – Estimate the CVB0 statistics

Stochastic Optimization for ML Stochastic gradient descent – Estimate the gradient Stochastic variational inference (Hoffman et al. 2010, 2013) – Estimate the natural gradient of the variational parameters Online EM (Cappe and Moulines, 2009) – Estimate E-step sufficient statistics Stochastic CVB0 – Estimate the CVB0 statistics

Estimating CVB0 Statistics

Pick a random word i from a random document j

Estimating CVB0 Statistics Pick a random word i from a random document j

Stochastic CVB0 In an online algorithm, we cannot store the variational parameters But we can update them!

Stochastic CVB0 Keep an online average of the CVB0 statistics

Extra Refinements Optional burn-in passes per document Minibatches Operating on sparse counts

Stochastic CVB0 Putting it all Together

Theory Stochastic CVB0 is a Robbins Monro stochastic approximation algorithm for finding the fixed points of (a variant of) CVB0 Theorem: with an appropriate sequence of step sizes, Stochastic CVB0 converges to a stationary point of the MAP, with adjusted hyper-parameters

Experimental Results – Large Scale

Experimental Results – Small Scale Real-time or near real-time results are important for EDA applications Human participants shown the top ten words from each topic

Experimental Results – Small Scale NIPS (5 Seconds)New York Times (60 Seconds) Mean number of errors Standard deviations:

Discussion We introduced stochastic CVB0 for LDA – Combines stochastic and collapsed inference approaches – Fast – Easy to implement – Accurate Experimental results show SCVB0 is useful for both large-scale and small-scale analysis Future work: Exploit sparsity, parallelization, non-parametric extensions

Thanks! Questions?