1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 29 Nov 11, 2005 Nanjing University of Science & Technology

2 Lecture 29 Topics 1. Review Fuzzy Sets and Fuzzy Partitions 2. Fuzzy C- Means Clustering Algorithm Preliminaries 3. Fuzzy C-Means Clustering Algorithm Details 4. Fuzzy C-Means Clustering Algorithm Example 5. Comments about Fuzzy Clustering

3 Example of a Fuzzy Set: Define a Fuzzy set A by the following membership function Or equivalently Example Review

4 S Function defined on S Review

5 Membership Functions for Fuzzy Clusters Domain Pattern Vectors Membership Function for F 1 Membership Function for F 2 EXAMPLE Review

6 A Fuzzy Partition F, of a set S, is defined by its membership functions for the fuzzy sets F k : k =1, 2,..., K ) ] Fuzzy Partition Review

7 where Each value bounded by 0 and 1 Sum of each columns values =1 Sum of each row is less than n Review

8 Conversion of Fuzzy Clustering into Crisp Clustering ) ] C C C Fuzzy Clusters Crisp Clustering All membership values are equal to 1 or 0 Only one 1 in each column where membership value for a Fuzzy cluster is a maximum

9 Fuzzy Clusters Crisp Clusters from Fuzzy Clusters Cl 1 : [ ] Cl 2 : [ ] Cl 3 : [ ] Cl 1 : [ ] Cl 2 : [ ] Cl 3 : [ ] Example: Given Fuzzy clusters below convert to Crisp Clusters

10 Cl 1 : [ ] Cl 2 : [ ] Cl 3 : [ ] Cl 1 : { x 1, x 2 } Cl 2 : { x 4, x 6 } Cl 3 : { x 3, x 5, x 7 } Cl 1 : { x 1 x 2 } Cl 2 : { x 4 x 6 } Cl 3 : { x 3 x 5 x 7 } Answer: Crisp Clusters

11 Fuzzy C-Means Clustering Preliminary Given a set S composed of pattern vectors which we wish to cluster ) ] C C C S = { x 1, x 2,..., x N } Define C Cluster Membership Functions...

12 Define C Cluster Centroids as follows Let V i be the Cluster Centroid for Fuzzy Cluster Cl i, i = 1, 2, …, C Define a Performance Objective J m as where

13 The Fuzzy C-Means Algorithm minimizes J m by selecting V i and i, i =1, 2, …, C by an alternating iterative procedure as described in the algorithm’s details m = Fuzziness Index (m >1 ) Higher numbers being more fuzzy A is a symmetric positive definite matrix N s is total number of pattern vectors Definitions

14 Fuzzy C-Means Clustering Algorithm (a) Flow Diagram No Yes

15 Fuzzy C-Means Clustering Algorithm (b) Details of Steps in the Algorithm

16 Step 1: Initialization Select initial membership functions such that This is equivalent to specifying fuzzy clusters F 1, F 2, …, F C

17 One Method to accomplish this selection is to chose r ki randomly from the open interval (0, 1) and then Normalize

18 Step 2: Computation of Fuzzy Centroids Compute the Fuzzy Centroids as

19 Step 3: Compute New Fuzzy Membership Functions Using the V i, i = 1, 2, …, C from step 2 compute i (k)

20 Alternative method for computing the Membership Functions where

21 If membership functions do not change convergence has occurred If algorithm converges then the i represent the fuzzy clusters and we Stop If Convergence has not occurred and the number of iterations is less than some preassigned maximum value (MAXIT) then return to step 2 If otherwise then Stop with no solution. Step 4: Check for Convergence

22 Usually convergences Answer not unique as depends upon initial conditions. Can converge to a local minimum Can be used to produce hard clusters Properties of Fuzzy C-Means Algorithm

23 Example – Application of Fuzzy Clustering Algorithm Given the following set of data vectors (a) Perform a Fuzzy Clustering of the data using Fuzzy C-Means Algorithm to obtain two fuzzy clusters. Try several initial conditions. Is the result unique? (use MAXIT=1000) (b) Using the results of (a) give a crisp clustering of the data. (c) Repeat (a) and (b) for three Fuzzy Clusters.

24 Plot of Data for Fuzzy clustering example

25 (a) Solution for two clusters Fuzzy cluster membership functions randomly selected Calculation of Fuzzy Centroids

26 Calculation of New Membership Functions Calculation of Performance Not the same as preceding iteration Membership function (No convergence) Number of iterations not greater than 1000 therefore the iterations continue.

27 Results Converge at Iteration 17 Cluster membership Functions Performance Measure Cluster Centroids

28 (a) Final Cluster membership Functions Cl 2 : Cl 1 :

29 (b) Solution Crisp Clustering Cl 1 = { x 1, x 2, x 3 } Cl 2 = { x 4, x 5 } Crisp Membership functions Set Assignment Cl 2 : Cl 1 : Fuzzy Membership functions

30 J m = (c) Solution for three fuzzy clusters Applying the Fuzzy Clustering Algorithm convergence was obtained in ?? iterations as Final Cluster membership functions Cl 1 : Cl 2 : Cl 3 :

31 (c) Solution Crisp Clustering Cl 1 = { x 1, x 2 } Cl 2 = { x 4, x 5 } Cl 3 = { x 3 } Membership functions Set Assignment F 1 : [ 1, 1, 0, 0, 0 ] F 2 : [ 0, 0, 0, 1, 1 ] F 3 : [ 0, 0, 1, 0, 0 ] “Crisp Clusters”

32 Comments: Fuzzy Clustering Can be used to Produce Hard Clustering The larger the value of m the fuzzier the clusters The Fuzzy algorithm is relatively stable and usually converges in a reasonable number of iterations The Fuzzy algorithm is relatively insensitve to initial conditions

33 Of the two different fuzzy clusterings given below, which clustering is the Fuzzier ??? Cl 1 : [ ] Cl 2 : [ ] Cl 1 : [ ] Cl 2 : [ ] # 1 # 2 or

34 Cl 1 : [ ] Cl 2 : [ ] Cl 1 : [ ] Cl 2 : [ ] # 1 # 2 ANSWER: #1 is the fuzzier of the two different clusterings

35 Why is #1 the Fuzzier of the two ???

36 Why is #1 the Fuzzier of the two ??? ANSWER: Because the cluster membership functions contain many entries close to 0.5 ( for the two class case) as opposed to values close to 0 and 1. For the M class case values close to 1/M would indicate most fuzziness.. * *

37 Lecture 29 Topics 1. Reviewed Fuzzy Sets and Fuzzy Partitions 2. Presented Fuzzy C- Means Clustering Algorithm Preliminaries 3. Gave Fuzzy C-Means Clustering Algorithm Details 4. Showed an Example of the Fuzzy C- Means Clustering Algorithm 5. Made a few comments about Fuzzy Clustering in General.

38 End of Lecture 29