Latent Variable Models Christopher M. Bishop

1. Density Modeling A standard approach: parametric models  a number of adaptive parameters  Gaussian distribution is widely used.  Loglikelihood method  Limitation  too flexible: parameter is so excessive  not too flexible: only uni-modal Considering mixture model, latent variable model

1.1. Latent Variables The number of parameters in normal distribution.  : d(d+1)/2 +  : d  d 2.  Assuming diagonal covariance matrix reduces  : d, but this means that t are statistically independent. Latent variables  Degree of freedom can be controlled, and correlation can be captured. Goal  to express p(t) of the variable t 1,…,t d in terms of a smaller number of latent variables x=(x 1,…,x q ) where q < d.

Cont ’ d  Joint distribution of p(t,x)  Bayesian network express the factorization

Cont ’ d  Express p(t|x) in terms of mapping from latent variables to data variables.  The definition of latent variables model is completed by specifying distribution p(u), mapping y(x;w), marginal distributino p(x).  The desired model for distribution p(t), but it is intractable in almost case.  Factor analysis: One of the simplest latent variable models

Cont ’ d  W,  : adaptive parameters  p(x): chosen to be N(0,I)  u: chosen to be zero mean Gaussian with a diagonal covariance matrix .  Then P(t) is Gaussian, with mean  and covariance matrix  +WW T.  Degree of freedom: (d+1)(q+1)-q(q+1)/2  Can capture the dominant correlations between the data variables

1.2. Mixture Distributions Uni-modal  mixture of M simpler parametric distributions  p(t|i): usually normal distribution with its own  i,  i.   i : mixing coefficients  mixing coefficients: prior probabilities for the values of the label i.  Considering indicator variable z ni.  Posterior probabilities: R ni is expectation of z ni.

Cont ’ d EM-algorithm Mixture of latent-variable models Bayesian network representation of a mixture of latent variable models. Given the values of i and x, the variables t 1,…,t d are conditionally independent.

2. Probabilistic Principal Component Analysis Summary  q principal axes v j, j  {1,…,q}  v j are q dominant eigenvectors of sample covariance matrix.  q principal components:  reconstruction vector: Disadvantage  absence of a probability density model and associated likelihood measure

2.1. Relationship to Latent Variables