Download presentation

1
**Part 2: Unsupervised Learning**

Machine Learning Techniques for Computer Vision Part 2: Unsupervised Learning Christopher M. Bishop Microsoft Research Cambridge ECCV 2004, Prague

2
**Overview of Part 2 Mixture models EM Variational Inference**

Bayesian model complexity Continuous latent variables Machine Learning Techniques for Computer Vision (ECCV 2004)

3
**The Gaussian Distribution**

Multivariate Gaussian Maximum likelihood covariance mean Machine Learning Techniques for Computer Vision (ECCV 2004)

4
**Gaussian Mixtures Linear super-position of Gaussians**

Normalization and positivity require Machine Learning Techniques for Computer Vision (ECCV 2004)

5
**Example: Mixture of 3 Gaussians**

Machine Learning Techniques for Computer Vision (ECCV 2004)

6
**Maximum Likelihood for the GMM**

Log likelihood function Sum over components appears inside the log no closed form ML solution Machine Learning Techniques for Computer Vision (ECCV 2004)

7
**EM Algorithm – Informal Derivation**

Machine Learning Techniques for Computer Vision (ECCV 2004)

8
**EM Algorithm – Informal Derivation**

M step equations Machine Learning Techniques for Computer Vision (ECCV 2004)

9
**EM Algorithm – Informal Derivation**

E step equation Machine Learning Techniques for Computer Vision (ECCV 2004)

10
**EM Algorithm – Informal Derivation**

Can interpret the mixing coefficients as prior probabilities Corresponding posterior probabilities (responsibilities) Machine Learning Techniques for Computer Vision (ECCV 2004)

11
**Time between eruptions (minutes)**

Old Faithful Data Set Time between eruptions (minutes) Duration of eruption (minutes) Machine Learning Techniques for Computer Vision (ECCV 2004)

12
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

13
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

14
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

15
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

16
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

17
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

18
**Latent Variable View of EM**

To sample from a Gaussian mixture: first pick one of the components with probability then draw a sample from that component repeat these two steps for each new data point Machine Learning Techniques for Computer Vision (ECCV 2004)

19
**Latent Variable View of EM**

Goal: given a data set, find Suppose we knew the colours maximum likelihood would involve fitting each component to the corresponding cluster Problem: the colours are latent (hidden) variables Machine Learning Techniques for Computer Vision (ECCV 2004)

20
**Incomplete and Complete Data**

Machine Learning Techniques for Computer Vision (ECCV 2004)

21
**Latent Variable Viewpoint**

Machine Learning Techniques for Computer Vision (ECCV 2004)

22
**Latent Variable Viewpoint**

Binary latent variables describing which component generated each data point Conditional distribution of observed variable Prior distribution of latent variables Marginalizing over the latent variables we obtain Machine Learning Techniques for Computer Vision (ECCV 2004)

23
**Graphical Representation of GMM**

Machine Learning Techniques for Computer Vision (ECCV 2004)

24
**Latent Variable View of EM**

Suppose we knew the values for the latent variables maximize the complete-data log likelihood trivial closed-form solution: fit each component to the corresponding set of data points We don’t know the values of the latent variables however, for given parameter values we can compute the expected values of the latent variables Machine Learning Techniques for Computer Vision (ECCV 2004)

25
**Posterior Probabilities (colour coded)**

Machine Learning Techniques for Computer Vision (ECCV 2004)

26
**Over-fitting in Gaussian Mixture Models**

Infinities in likelihood function when a component ‘collapses’ onto a data point: with Also, maximum likelihood cannot determine the number K of components Machine Learning Techniques for Computer Vision (ECCV 2004)

27
Cross Validation Can select model complexity using an independent validation data set If data is scarce use cross-validation: partition data into S subsets train on S-1 subsets test on remainder repeat and average Disadvantages computationally expensive can only determine one or two complexity parameters Machine Learning Techniques for Computer Vision (ECCV 2004)

28
**Bayesian Mixture of Gaussians**

Parameters and latent variables appear on equal footing Conjugate priors Machine Learning Techniques for Computer Vision (ECCV 2004)

29
Data Set Size Problem 1: learn the function for from 100 (slightly) noisy examples data set is computationally small but statistically large Problem 2: learn to recognize 1,000 everyday objects from 5,000,000 natural images data set is computationally large but statistically small Bayesian inference computationally more demanding than ML or MAP (but see discussion of Gaussian mixtures later) significant benefit for statistically small data sets Machine Learning Techniques for Computer Vision (ECCV 2004)

30
**Variational Inference**

Exact Bayesian inference intractable Markov chain Monte Carlo computationally expensive issues of convergence Variational Inference broadly applicable deterministic approximation let denote all latent variables and parameters approximate true posterior using a simpler distribution minimize Kullback-Leibler divergence Machine Learning Techniques for Computer Vision (ECCV 2004)

31
**General View of Variational Inference**

For arbitrary where Maximizing over would give the true posterior this is intractable by definition Machine Learning Techniques for Computer Vision (ECCV 2004)

32
**Variational Lower Bound**

Machine Learning Techniques for Computer Vision (ECCV 2004)

33
**Factorized Approximation**

Goal: choose a family of q distributions which are: sufficiently flexible to give good approximation sufficiently simple to remain tractable Here we consider factorized distributions No further assumptions are required! Optimal solution for one factor, keeping the remainder fixed coupled solutions so initialize then cyclically update message passing view (Winn and Bishop, 2004) Machine Learning Techniques for Computer Vision (ECCV 2004)

34
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

35
**Lower Bound Can also be evaluated Useful for maths/code verification**

Also useful for model comparison: Machine Learning Techniques for Computer Vision (ECCV 2004)

36
**Illustration: Univariate Gaussian**

Likelihood function Conjugate prior Factorized variational distribution Machine Learning Techniques for Computer Vision (ECCV 2004)

37
**Initial Configuration**

Machine Learning Techniques for Computer Vision (ECCV 2004)

38
After Updating Machine Learning Techniques for Computer Vision (ECCV 2004)

39
After Updating Machine Learning Techniques for Computer Vision (ECCV 2004)

40
Converged Solution Machine Learning Techniques for Computer Vision (ECCV 2004)

41
**Variational Mixture of Gaussians**

Assume factorized posterior distribution No other approximations needed! Machine Learning Techniques for Computer Vision (ECCV 2004)

42
**Variational Equations for GMM**

Machine Learning Techniques for Computer Vision (ECCV 2004)

43
Lower Bound for GMM Machine Learning Techniques for Computer Vision (ECCV 2004)

44
**VIBES Bishop, Spiegelhalter and Winn (2002)**

Machine Learning Techniques for Computer Vision (ECCV 2004)

45
ML Limit If instead we choose we recover the maximum likelihood EM algorithm Machine Learning Techniques for Computer Vision (ECCV 2004)

46
**Bound vs. K for Old Faithful Data**

Machine Learning Techniques for Computer Vision (ECCV 2004)

47
**Bayesian Model Complexity**

Machine Learning Techniques for Computer Vision (ECCV 2004)

48
**Sparse Bayes for Gaussian Mixture**

Corduneanu and Bishop (2001) Start with large value of K treat mixing coefficients as parameters maximize marginal likelihood prunes out excess components Machine Learning Techniques for Computer Vision (ECCV 2004)

49
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

50
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

51
**Summary: Variational Gaussian Mixtures**

Simple modification of maximum likelihood EM code Small computational overhead compared to EM No singularities Automatic model order selection Machine Learning Techniques for Computer Vision (ECCV 2004)

52
**Continuous Latent Variables**

Conventional PCA data covariance matrix eigenvector decomposition Minimizes sum-of-squares projection not a probabilistic model how should we choose L ? Machine Learning Techniques for Computer Vision (ECCV 2004)

53
**Probabilistic PCA Tipping and Bishop (1998)**

L dimensional continuous latent space D dimensional data space PCA factor analysis Machine Learning Techniques for Computer Vision (ECCV 2004)

54
**Probabilistic PCA Marginal distribution Advantages exact ML solution**

computationally efficient EM algorithm captures dominant correlations with few parameters mixtures of PPCA Bayesian PCA building block for more complex models Machine Learning Techniques for Computer Vision (ECCV 2004)

55
EM for PCA Machine Learning Techniques for Computer Vision (ECCV 2004)

56
EM for PCA Machine Learning Techniques for Computer Vision (ECCV 2004)

57
EM for PCA Machine Learning Techniques for Computer Vision (ECCV 2004)

58
EM for PCA Machine Learning Techniques for Computer Vision (ECCV 2004)

59
EM for PCA Machine Learning Techniques for Computer Vision (ECCV 2004)

60
EM for PCA Machine Learning Techniques for Computer Vision (ECCV 2004)

61
EM for PCA Machine Learning Techniques for Computer Vision (ECCV 2004)

62
**Bayesian PCA Bishop (1998) Gaussian prior over columns of**

Automatic relevance determination (ARD) ML PCA Bayesian PCA Machine Learning Techniques for Computer Vision (ECCV 2004)

63
**Non-linear Manifolds Example: images of a rigid object**

Machine Learning Techniques for Computer Vision (ECCV 2004)

64
**Bayesian Mixture of BPCA Models**

Machine Learning Techniques for Computer Vision (ECCV 2004)

65
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

66
**Flexible Sprites Jojic and Frey (2001)**

Automatic decomposition of video sequence into background model ordered set of masks (one per object per frame) foreground model (one per object per frame) Machine Learning Techniques for Computer Vision (ECCV 2004)

67
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

68
**Transformed Component Analysis**

Generative model Now include transformations (translations) Extend to L layers Inference intractable so use variational framework Machine Learning Techniques for Computer Vision (ECCV 2004)

69
**Machine Learning Techniques for Computer Vision (ECCV 2004)**

70
**Bayesian Constellation Model**

Li, Fergus and Perona (2003) Object recognition from small training sets Variational treatment of fully Bayesian model Machine Learning Techniques for Computer Vision (ECCV 2004)

71
**Bayesian Constellation Model**

Machine Learning Techniques for Computer Vision (ECCV 2004)

72
**Summary of Part 2 Discrete and continuous latent variables**

EM algorithm Build complex models from simple components represented graphically incorporates prior knowledge Variational inference Bayesian model comparison Machine Learning Techniques for Computer Vision (ECCV 2004)

Similar presentations

© 2019 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google