1. Lecture 3-4 Clustering (1hr) Gaussian Mixture and EM (1hr)
Tae-Kyun Kim

2. Vector Clustering
2D data vectors (green) are grouped to two homogenous clusters (blue and red).
Clustering is achieved by an iterative algorithm (left to right). The cluster centers are marked x.

3. Pixel Clustering (Image Quantisation)
Image pixels are represented by 3D vectors of R,G,B values.
The vectors are grouped to K=10,3,2 clusters, and represented by the mean values of the respective clusters. R G B ``

4. Patch Clustering (BoW in Lecture 9-10)
Image patches are harvested around feature points in a large number of images.
They are represented by finite dimensional vectors, and clustered to form a visual dictionary.
SIFT 20
or raw pixels
D=400 20 … …… ……
dimension D
K codewords …

5. Image Clustering
Whole images are represented as finite dimensional vectors.
Homogenous vectors are grouped together in Euclidean space. ……

6. K-means vs GMM
Two representative techniques are k-means and Gaussian Mixture Model (GMM).
K-means assigns data points to the nearest clusters, while GMM assigns data to the Gaussian densities that best represent the data.
Hard clustering: a data point is assigned only one cluster.
Soft clustering: a data point is assigned multiple Gaussians probabilistically.

7. Matrix and Vector Derivatives

9. K-means Clustering

11. till converge

12. K=2
rnk μ 1 μ 2

13. Convergence proof (yes)
Global minimum (no)

14. V= 𝐱_𝑛−𝐮_𝑘 𝑇 Σ k −1 (𝐱_𝑛−𝐮_𝑘)

15. Statistical Pattern Recognition Toolbox for Matlab
…\stprtool\probab\cmeans.m
…\stprtool\probab\cmeans_tk.m

16. Mixture of Gaussians

20. Maximum Likelihood
s.t.

22. max f(x) s.t. g(x)=0 max f(x) + 𝜆g(x) objective ftn. f(x)
constraints g(x)
max f(x) s.t. g(x)=0
max f(x) + 𝜆g(x)

24. till converge

27. Statistical Pattern Recognition Toolbox for Matlab
…\stprtool\visual\pgmm.m
…\stprtool\demos\demo_emgmm.m

28. Supplementary Material

29. Information Theory (for Lecture 7-8)

31. Advanced topic (optional)

32. EM Algorithm in General