1. Intro. ANN & Fuzzy Systems Lecture 23 Clustering (4)

2. Intro. ANN & Fuzzy Systems (C) 2001 by Yu Hen Hu 2 Outline Clustering as Density Estimation Non-parametric density estimate: Parzen Windows Mixture Density Estimate

3. Intro. ANN & Fuzzy Systems (C) 2001 by Yu Hen Hu 3 Density Estimate Probability density function p(x) describes the distribution of samples in the feature space. p(x) can be estimated using –Non-parametric method: parzen window (kernel) method –Parametric method: mixture Gaussian density model It is related to clustering when each cluster is modeled with a uni-Gaussian density

4. Intro. ANN & Fuzzy Systems (C) 2001 by Yu Hen Hu 4 Probability Density Let f(x) denote the prob. density function of x over a feature space X. Let {x 1, …, x N } be i.i.d. samples drawn from X, then pr. K of them fall within , P k has a binomial distribution, and E(K) = K/N K should be increased as N does, and V should be decreased as N does Let V be the volume enclosed by , then the pr. density f(x) can be estimated as K/(NV). –V should be small enough that f(x) are relatively constant over . –But with N finite, K will approach 0 when V does. 3 conditions:

5. Intro. ANN & Fuzzy Systems (C) 2001 by Yu Hen Hu 5 Parzen Windows A non-parametric method using interpolation functions Let be such that {x(k); 1  k  N} are drawn from unknown density p(x). Set V N = h N d where h N is a smoothing parameter and d is the dimension of x. Then the estimate of p(x) is: Conditions: and Example of window function: kerneldemo.m

6. Intro. ANN & Fuzzy Systems (C) 2001 by Yu Hen Hu 6 Mixture Density Estimation Mixture density model: p(i): prior (mixing) prob. p(x|  (i),i): component density Gaussian mixture model (GMM) Let {x k ; 1  k  N} be drawn from the mixture density, goal is to estimate  (i) and p(i) (negative) log likelihood

7. Intro. ANN & Fuzzy Systems (C) 2001 by Yu Hen Hu 7 EM Algorithm for GMM We want to minimize the negative log likelihood but p(i)s are unknown (missing data). EM algorithm: –Expectation: Given current estimates of p(x|  (i), i} (i.e. {  i }, and {  i }), find {p(i)}. –Maximization: With current estimates of {p(i)}, update estimates of p(x|  (i), i} (i.e. {  i }, and {  i }) by minimizing negative log likelihood. Expectation step: To minimize subject to  p(i) = 1, and p(i)  0. Use Lagrange multiplier, one may find

8. Intro. ANN & Fuzzy Systems (C) 2001 by Yu Hen Hu 8 EM: Maximization Step Parameters  (µ and  ) can be estimated as: The formula for updating {  i }, and {  i } is similar to the so-called fuzzy c-mean clustering algorithm.