Lecture 3-4 Clustering (1hr) Gaussian Mixture and EM (1hr) Tae-Kyun Kim

Vector Clustering 2D data vectors (green) are grouped to two homogenous clusters (blue and red). Clustering is achieved by an iterative algorithm (left to right). The cluster centers are marked x.

Pixel Clustering (Image Quantisation) Image pixels are represented by 3D vectors of R,G,B values. The vectors are grouped to K=10,3,2 clusters, and represented by the mean values of the respective clusters. R G B ``

Patch Clustering (BoW in Lecture 9-10) Image patches are harvested around feature points in a large number of images. They are represented by finite dimensional vectors, and clustered to form a visual dictionary. SIFT 20 or raw pixels D=400 20 … …… …… dimension D K codewords …

Image Clustering Whole images are represented as finite dimensional vectors. Homogenous vectors are grouped together in Euclidean space. ……

K-means vs GMM Two representative techniques are k-means and Gaussian Mixture Model (GMM). K-means assigns data points to the nearest clusters, while GMM assigns data to the Gaussian densities that best represent the data. Hard clustering: a data point is assigned only one cluster. Soft clustering: a data point is assigned multiple Gaussians probabilistically.

Matrix and Vector Derivatives

K-means Clustering

till converge

K=2 rnk μ 1 μ 2

Convergence proof (yes) Global minimum (no)

V= 𝐱_𝑛−𝐮_𝑘 𝑇 Σ k −1 (𝐱_𝑛−𝐮_𝑘)

Statistical Pattern Recognition Toolbox for Matlab http://cmp.felk.cvut.cz/cmp/software/stprtool/ …\stprtool\probab\cmeans.m …\stprtool\probab\cmeans_tk.m

Mixture of Gaussians

Maximum Likelihood s.t.

max f(x) s.t. g(x)=0 max f(x) + 𝜆g(x) objective ftn. f(x) constraints g(x) max f(x) s.t. g(x)=0 max f(x) + 𝜆g(x) http://en.wikipedia.org/wiki/Lagrange_multiplier

till converge

Statistical Pattern Recognition Toolbox for Matlab http://cmp.felk.cvut.cz/cmp/software/stprtool/ …\stprtool\visual\pgmm.m …\stprtool\demos\demo_emgmm.m

Supplementary Material

Information Theory (for Lecture 7-8)

Advanced topic (optional) http://www.iis.ee.ic.ac.uk/~tkkim/mlcv/lecture_clustering_em.pdf

EM Algorithm in General