1. Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors: Matthew D. Hoffman, David M. Blei, Perry R. cook Princeton University, Department of Computer Science, 35 olden St., Princeton, NJ, 08540 USA

2. Outline ■Introduction ■Terminology ■Problem statement and contribution of this paper ■Gap-NMF Model(Gamma Process Nonnegative Matrix Factorization ) ■Variational Inference ■Definition ■Variational Objective Function ■Coordinate Ascent Optimization ■Other Approaches ■Evaluation

3. Terminology(1) ■Nonparametric Statistics: □The term non-parametric is not meant to imply that such models completely lack parameters but that the number and nature of the parameters are flexible and not fixed in advance. ■Nonnegative Matrix Factorization: □Non-negative matrix factorization (NMF) is a group of algorithms in multivariate analysis and linear algebra where a matrix, is factorized into (usually) two matrices with all elements are greater than or equal to 0 The above two definitions are cited from Wikipedia

4. Terminology(2) ■Variational Inference: □Variational inference approximates the posterior distribution with a simpler distribution, whose parameters are optimized to be close to the true posterior. ■Mean-field Variational Inference: □In mean-field variational inference, each variable is given an independent distribution, usually of the same family as its prior.

5. Outline ■Introduction ■Terminology ■Problem statement and Contribution of this Paper ■Gap-NMF Model ■Variational Inference ■Definition ■Variational Objective Function ■Coordinate Ascent Optimization ■Other Approaches ■Evaluation

6. Problem Statement and Contribution ■Research Topic: □Breaking audio spectrograms into separate sources of sound using latent variable decompositions. E.g., matrix factorization. ■A potential problem : □The number of latent variables must be specified in advance which is not always possible. ■Contribution of this paper □The paper develops Gamma Process Nonnegative Matrix Factorization (GaP-NMF), a Bayesian nonparametric approach to decompose spectrograms.

7. Outline ■Introduction ■Terminology ■Problem statement and Contribution of this Paper ■Gap-NMF Model ■Variational Inference ■Definition ■Variational Objective Function ■Coordinate Ascent Optimization ■Other Approaches ■Evaluation

8. Dataset on GaP-NMF Model ■What are given is a M by N matrix in which is the power of audio signal at time window n and frequency bin m. If the number of latent variable is specified in advance: ■Assuming the audio signal is composed of K static sound sources. The problem is to decompose, in which is M by K matrix, is K by N matrix. In which cell is the average amount of energy source k exhibits at frequency m. cell is the gain of source k at time n. ■The problem is solved by

9. GaP-NMF Model If the number of latent variable is not specified in advance: ■GaP-NMF assumes that the data is drawn according to the following generative process: Based on the formula that (Abdallah&Plumbley (2004))

10. GaP-NMF Model If the number of latent variable is not specified in advance: ■GaP-NMF assumes that the data is drawn according to the following generative process: The overall gain of the corresponding source l Based on the formula that (Abdallah&Plumbley (2004)) Used to control the number of latent variables

11. GaP-NMF Model ■The number of nonzero is the number of the latent variables K. ■If L increased towards infinity, the nonzero L which expressed by K is finite and obeys: Kingman,1993

12. Outline ■Introduction ■Terminology ■Problem statement and Contribution of this Paper ■Gap-NMF Model ■Variational Inference ■Definition ■Variational Objective Function ■Coordinate Ascent Optimization ■Other Approaches ■Evaluation

13. Definition of Variational Inference ■Variational inference approximates the posterior distribution with a simpler distribution, whose parameters are optimized to be close to the true posterior. ■Under this paper’s condition: Posterior Distribution What measured

14. Definition of Variational Inference ■Variational inference approximates the posterior distribution with a simpler distribution, whose parameters are optimized to be close to the true posterior. ■Under this paper’s condition: Variational distribution assumption with free parameters Variational DistributionPosterior Distribution Approximates What measured

15. Definition of Variational Inference ■Variational inference approximates the posterior distribution with a simpler distribution, whose parameters are optimized to be close to the true posterior. ■Under this paper’s condition: Variational distribution assumption with free parameters Variational DistributionAdjust ParametersPosterior Distribution Approximates What measured

16. Outline ■Introduction ■Terminology ■Problem statement and Contribution of this Paper ■Gap-NMF Model ■Variational Inference ■Definition ■Variational Objective Function ■Coordinate Ascent Optimization ■Other Approaches ■Evaluation

17. Variational Objective Function ■Assume each variable obeys the following Generalized Inverse-Gaussian (GIG) family:

18. Variational Objective Function ■Assume each variable obeys the following Generalized Inverse-Gaussian (GIG) family: It is Gamma family

19. Variational Objective Function ■Assume each variable obeys the following Generalized Inverse-Gaussian (GIG) family: Denotes a modified Bessel function of the second kind It is Gamma family

20. Deduction(1) ■The difference between the left and right sides is the Kullback-Leibler divergence between the true posterior and the variational distribution q. ■Kullback-Leibler divergence : for probability distributions P and Q of a discrete random variable their K–L divergence is defined to be From Jordan et al., 1999

21. Deduction(2)

22. Using Jensen’s inequality

23. Objective function ■L= ■The objective function becomes Bounded by +

24. ■Maximize the objective function defined above with the corresponding parameters. ■The distribution is obtained: ■Because these three distributions are independent, we gain approximates

25. Outline ■Introduction ■Terminology ■Problem statement and Contribution of this Paper ■Gap-NMF Model ■Variational Inference ■Definition ■Variational Objective Function ■Coordinate Ascent Optimization ■Other Approaches ■Evaluation

26. Coordinate Ascent Algorithm(1) ■The derivative of the objective function with respect to variational parameters equals to zero to obtain: ■Similarly:

27. Coordinate Ascent Algorithm(2) ■Using Lagrange multipliers, then the bound parameters become ■Then updating bound parameters and variational parameters according to equations 14,15,16,17 and18 to ultimately reaching a local minimum.

28. Outline ■Introduction ■Terminology ■Problem statement and Contribution of this Paper ■Gap-NMF Model ■Variational Inference ■Definition ■Variational Objective Function ■Coordinate Ascent Optimization ■Other Approaches ■Evaluation

29. Other Approaches ■Finite Bayesian Model ( also called GIG-NMF). ■Finite Non-Bayesian Model. ■EU-Nonnegative Matrix Factorization. ■KL-Nonnegative Matrix Factorization.

30. Outline ■Introduction ■Terminology ■Problem statement and Contribution of this Paper ■Gap-NMF Model ■Variational Inference ■Definition ■Variational Objective Function ■Coordinate Ascent Optimization ■Other Approaches ■Evaluation

31. Evaluation on Synthetic data(1) ■The data is generated according to the following model:

32. Evaluation on Synthetic data(2)

33. Evaluation on Recorded Music

34. Conclusion ■Gap-NMF model is capable of determining the number of latent source automatically. ■The key step of the paper is to use variational distribution to approximate posterior distribution. ■Gap-NMF can work well on analyzing and processing recorder music, it can be applicable to other types of audio.

35. ■Thank you!