1. Accelerated Sampling for the Indian Buffet Process
Finale Doshi-Velez and Zoubin Ghahramani
ICML 2009
Presented by: John Paisley, Duke University

2. Introduction
The IBP is a nonparametric prior for inferring the number of underlying features in a dataset, as well as which subset of these features are used by a given observation, e.g., the number of underlying notes in a piece of music and which notes are used at what times.
Gibb sampling currently has some issues: The uncollapsed Gibbs sampler is slow to mix, while the collapsed Gibbs sampler is slow to complete an iteration when the number of samples is large.
This paper presents an accelerated Gibbs sampling method for the linear-Gaussian model that is fast in both respects.

3. The IBP and Linear-Gaussian Model
The IBP story:
The first customer walks into the buffet and samples dishes.
The Nth customer samples previously tasted dishes with probability proportional to the number of previous customers who have sampled the dish, and samples more dishes.
This prior can be used for the linear-Gaussian model, to infer the number of underlying vectors that are linearly combined to construct a matrix.

4. IBP Calculations for Z
Integrating out A, the posteriors for for Z are calculated for existing features as,
and for new features as,
Collapsing out the loadings matrix, A, the likelihood term is,
which significantly increases the amount of computation that needs to be done to calculate the likelihood.

5. IBP Calculations for Z
When A is not integrated out, inference is faster because matrices do not need to be inverted. Also, when finding the posterior for a value in the nth row of Z, we don’t need to worry about the other rows of Z since the likelihood can be represented as
For the linear-Gaussian model, the posterior of A is Gaussian with,

6. Accelerated Gibbs Sampling
Goal: Develop a sampler that mixes like the collapsed sampler, but has the speed of the uncollapsed sampler.
To do this, select a window and split the data into two groups.

7. Accelerated Gibbs Sampling
By splitting the data this way, we can write the probabilities for Z as,
We therefore don’t need to worry about X-w when calculating likelihoods.

8. Accelerated Gibbs Sampling
We can efficiently compute means and covariances with data removed,
And then efficiently update the mean and covariance using all data using the statistics for the window.
And rank-one updates can be used when updating

9. Experiments (Synthetic Data)
Generate a linear-Gaussian model from the IBP prior with D = 10.

10. Experiments (real data)
Experiments were run on several real datasets for 500 iterations or 150 hours (max). Data set information is below. Per-iteration run times and performance measures are at right.

11. Experiments (real data)

12. Discussion and Conclusion
The accelerated Gibbs sampler achieved similar performance with the other two sampling methods, but at a much faster rate.
Rank-one updates are less precise (due to round-offs). It is important to sometimes invert the entire matrix.
Performance is also faster than slice-sampling. Also, it doesn’t rely on proposals (Metropolis-Hastings) or particle counts (particle filters).
Efficient computations were obtained by collapsing locally on a window using posteriors calculated from data outside of this window.