1. Non-Linear Probabilistic PCA with Gaussian Process Latent Variable Models
Neil Lawrence
Machine Learning Group
Department of Computer Science
University of Sheffield, U.K.

2. Overview Principal Component Analysis
Latent Variable Model
Probabilistic derivations
Gaussian Process Latent Variable Models
Optimisation
Sparse algorithm
Results
Visualisation
Inverse Kinematics

3. Influences Density Networks – MacKay 1995
Generative Topographic Mapping – Bishop, Svensen and Williams 1997
Probabilistic PCA – Tipping and Bishop 1998

4. Notation q – dimension of latent space. d – dimension of data space.
N – number of data points.
centred data,
latent variables,
mapping matrix, W 2 <d£q.
a(i) is vector from i th row of A
ai is a vector from i th column of A

5. Reading Notation X and Y are design matrices.
Covariance given by N-1YTY.
Inner product matrix given by YYT.

6. PCA – Probabilistic Interpretation

7. PCA – Probabilistic Interpretation II

8. Equivalence of PPCA Formulations
Solution for PPCA I:
Solution for PPCA II:
Equivalence is from

9. Gaussian Processes A Gaussian Process (GP) likelihood is of the form
where K is the covariance function or kernel.
If we select the linear kernel
We see PPCA is a product of GPs.

10. Probabilistic PCA is a GPLVM
Log-likelihood:

11. Non-linear Kernel Instead of linear kernel function.
Use RBF kernel function.
Leads to non-linear GPs.

12. Ky non-linear – kernel PCA
Aside: Kernel PCA
Max likelihood ´ min Kullback Leibler.
PCA – minimise KL divergence between Gaussians
Ky non-linear – kernel PCA
Kx non-linear – GPLVM
Both non-linear – ?

13. Correlation Matching Minimisation of KL divergence.
Matches dominant correlations covariance matrices.
KPCA (with RBF kernel)
Kx global Ky local.
GPLVM (with RBF kernel)
Kx local Ky global.

14. Interpretation Kernels are similarity measures.
Express correlations between points.
GPLVM and KPCA try to match them.
cf. MDS and principal co-ordinate analysis.

15. Pros & Cons of GPLVM Pros Cons Probabilistic
Missing data straightforward.
Can sample from model given X.
Different noise models can be handled.
Kernel parameters can be optimised.
Cons
Speed of optimisation.
Optimisation is non-convex.

16. GPLVM Optimisation Gradient based optimisation wrt X, , ,  (SCG).
Penalised ML with penalty of -tr(XTX).
Example data-set
Oil flow data
Three phases of flow (stratified annular homogenous)
Twelve measurement probes
1000 data-points
We sub-sampled to 100 data-points.

17. SCG GPLVM Oil Results
2-D Manifold in 12-D space (shading is variance).

18. Efficient GPLVM Optimisation
Optimising q£N matrix X is slow.
There are correlations between data-points.

19. ‘Sparsification’ If XI is a sub-set of X.
For well chosen active set, I, |I|<<N
For n I optimise q-dimensional x(n) independently.

20. Algorithm We selected the active-set according the the IVM scheme
Select active set.
Optimise ,  and .
For all nI
Optimise xn.
For small data-sets optimise XI.
Repeat.

21. Some Results Oil data Digits data Face data Twos data
digits 0 to 4 from USPS data.
600 of each digit randomly selected.
16x16 greyscale images.
Face data
A video of Brendan Frey’s face.
1965 frames time info removed.
20x28 greyscale images.
Twos data
Cedar CD ROM digits.
700 examples of 8x8 twos.
Binary images.

22. Oil Data Full training set this time. Used RBF kernel for GPLVM
GTM used 15x15 grid and 16 nodes.
Trying to mimic experiments in original paper.

23. Oil Data
GTM
GPLVM (RBF)

24. Different Kernels RBF Kernel MLP Kernel Log likelihood: 1.12 £ 104

25. Digits RBF Kernel MLP Kernel Log likelihood: -4.49 £ 105
0 – red, 1 – blue, 2 – green, 3 – mauve, 4 – yellow.

26. Digits Data
Image from

28. Demos
Fantasy Digits.
Fantasy Brendans.

29. Twos Data So far – Gaussian Noise. Can use Binomial likelihood.
Probit noise model.
Use ADF approxmation.
Can easily extend to EP.
Practical Consequences
About d times slower, need d times more storage.
Twos data modelled as
Gaussian.
Binomial.

30. Twos Results

31. Inverse Kinematics
Style-Based Inverse Kinematics Keith Grochow, Steve L. Martin, Aaron Hertzmann, Zoran Popović. ACM Trans. on Graphics (Proc. SIGGRAPH 2004). To appear.
Learn a GPLVM on motion capture data.
Use GPLVM as ‘soft style constraint’ in combination with hard kinematics constraints.

32. Video
styleik.mov

33. Why GPLVM in IK? My thoughts: Inverse Kinematics
GPLVM is probabilistic (soft constraints).
GPLVM can capture non-linearities in the data.
Inverse Kinematics
Can be viewed as missing value problem.
GPLVM handles missing values well.
Grochow et al. also mix styles by mixing GPLVMs.

34. Rant to ML Community Source code for GPLVM available Jun ’03.
Grochow et al. Downloaded after NIPS acceptance.
Submitted Jan ’04 to SIGGRAPH.
On-line source code is good!

35. Conclusions GPLVM is a Probabilistic Non-Linear PCA
Can sample from it.
Evaluate likelihoods.
Missing data no problem.
Optimisation of X is the difficult part.
We presented a sparse optimisation algorithm.
Model has been ‘proven’ in real application.
Put your source code on-line!