1. Semi-Supervised Clustering Jieping Ye Department of Computer Science and Engineering Arizona State University http://www.public.asu.edu/~jye02

2. Outline Overview of clustering and classification What is semi-supervised learning? –Semi-supervised clustering –Semi-supervised classification Semi-supervised clustering –What is semi-supervised clustering? –Why semi-supervised clustering? –Semi-supervised clustering algorithms

3. Supervised classification versus unsupervised clustering Unsupervised clustering Group similar objects together to find clusters Minimize intra-class distance Maximize inter-class distance Supervised classification Class label for each training sample is given –Build a model from the training data –Predict class label on unseen future data points

4. What is clustering? Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups Inter-cluster distances are maximized Intra-cluster distances are minimized

5. What is Classification?

6. Clustering algorithms K-Means Hierarchical clustering Graph based clustering (Spectral clustering) Bi-clustering

7. Classification algorithms K-Nearest-Neighbor classifiers Naïve Bayes classifier Linear Discriminant Analysis (LDA) Support Vector Machines (SVM) Logistic Regression Neural Networks

8. Supervised Classification Example....

9. ....................

10. ....................

11. . Unsupervised Clustering Example...................

12. ....................

13. Semi-Supervised Learning Combines labeled and unlabeled data during training to improve performance: –Semi-supervised classification: Training on labeled data exploits additional unlabeled data, frequently resulting in a more accurate classifier. –Semi-supervised clustering: Uses small amount of labeled data to aid and bias the clustering of unlabeled data. Unsupervised clustering Semi-supervised learning Supervised classification

14. . Semi-Supervised Classification Example...................

15. ....................

16. Semi-Supervised Classification Algorithms: –Semisupervised EM [Ghahramani:NIPS94,Nigam:ML00]. –Co-training [Blum:COLT98]. –Transductive SVM’s [Vapnik:98,Joachims:ICML99]. –Graph based algorithms Assumptions: –Known, fixed set of categories given in the labeled data. –Goal is to improve classification of examples into these known categories.

17. . Semi-Supervised Clustering Example...................

18. ....................

19. . Second Semi-Supervised Clustering Example...................

20. ....................

21. Semi-supervised clustering: problem definition Input: –A set of unlabeled objects, each described by a set of attributes (numeric and/or categorical) –A small amount of domain knowledge Output: –A partitioning of the objects into k clusters (possibly with some discarded as outliers) Objective: –Maximum intra-cluster similarity –Minimum inter-cluster similarity –High consistency between the partitioning and the domain knowledge

22. Why semi-supervised clustering? Why not clustering? –The clusters produced may not be the ones required. –Sometimes there are multiple possible groupings. Why not classification? –Sometimes there are insufficient labeled data. Potential applications –Bioinformatics (gene and protein clustering) –Document hierarchy construction –News/email categorization –Image categorization

23. Semi-Supervised Clustering Domain knowledge –Partial label information is given –Apply some constraints (must-links and cannot-links) Approaches –Search-based Semi-Supervised Clustering Alter the clustering algorithm using the constraints –Similarity-based Semi-Supervised Clustering Alter the similarity measure based on the constraints –Combination of both

24. Search-Based Semi-Supervised Clustering Alter the clustering algorithm that searches for a good partitioning by: –Modifying the objective function to give a reward for obeying labels on the supervised data [Demeriz:ANNIE99]. –Enforcing constraints (must-link, cannot-link) on the labeled data during clustering [Wagstaff:ICML00, Wagstaff:ICML01]. –Use the labeled data to initialize clusters in an iterative refinement algorithm (kMeans,) [Basu:ICML02].

25. Overview of K-Means Clustering K-Means is a partitional clustering algorithm based on iterative relocation that partitions a dataset into K clusters. Algorithm: Initialize K cluster centers randomly. Repeat until convergence: –Cluster Assignment Step: Assign each data point x to the cluster X l, such that L 2 distance of x from (center of X l ) is minimum –Center Re-estimation Step: Re-estimate each cluster center as the mean of the points in that cluster

26. K-Means Objective Function Locally minimizes sum of squared distance between the data points and their corresponding cluster centers: Initialization of K cluster centers: –Totally random –Random perturbation from global mean –Heuristic to ensure well-separated centers

27. K Means Example

28. K Means Example Randomly Initialize Means x x

29. K Means Example Assign Points to Clusters x x

30. K Means Example Re-estimate Means x x

31. K Means Example Re-assign Points to Clusters x x

32. K Means Example Re-estimate Means x x

33. K Means Example Re-assign Points to Clusters x x

34. K Means Example Re-estimate Means and Converge x x

35. Semi-Supervised K-Means Partial label information is given –Seeded K-Means –Constrained K-Means Constraints (Must-link, Cannot-link) –COP K-Means

36. Semi-Supervised K-Means for partially labeled data Seeded K-Means: –Labeled data provided by user are used for initialization: initial center for cluster i is the mean of the seed points having label i. –Seed points are only used for initialization, and not in subsequent steps. Constrained K-Means: –Labeled data provided by user are used to initialize K- Means algorithm. –Cluster labels of seed data are kept unchanged in the cluster assignment steps, and only the labels of the non- seed data are re-estimated.

37. Seeded K-Means Use labeled data to find the initial centroids and then run K-Means. The labels for seeded points may change.

38. Seeded K-Means Example

39. Seeded K-Means Example Initialize Means Using Labeled Data x x

40. Seeded K-Means Example Assign Points to Clusters x x

41. Seeded K-Means Example Re-estimate Means x x

42. Seeded K-Means Example Assign points to clusters and Converge x x the label is changed

43. Exercise 0 1 3 6 9 15 20 21 22 Compute the clustering using seeded Kmeans

44. Constrained K-Means Use labeled data to find the initial centroids and then run K-Means. The labels for seeded points will not change.

45. Constrained K-Means Example

46. Constrained K-Means Example Initialize Means Using Labeled Data x x

47. Constrained K-Means Example Assign Points to Clusters x x

48. Constrained K-Means Example Re-estimate Means and Converge x x

49. Exercise 0 1 3 6 9 15 20 21 22 Compute the clustering using constrained Kmeans

50. COP K-Means COP K-Means [Wagstaff et al.: ICML01] is K- Means with must-link (must be in same cluster) and cannot-link (cannot be in same cluster) constraints on data points. Initialization: Cluster centers are chosen randomly, but as each one is chosen any must-link constraints that it participates in are enforced (so that they cannot later be chosen as the center of another cluster). Algorithm: During cluster assignment step in COP- K-Means, a point is assigned to its nearest cluster without violating any of its constraints. If no such assignment exists, abort.

51. COP K-Means Algorithm

52. Illustration x x Must-link Determine its label Assign to the red class

53. Illustration x x Cannot-link Determine its label Assign to the red class

54. Illustration x x Cannot-link Determine its label The clustering algorithm fails Must-link

55. Similarity-based semi-supervised clustering Alter the similarity measure based on the constraints Paper: From Instance-level Constraints to Space- Level Constraints: Making the Most of Prior Knowledge in Data Clustering. D. Klein et al. Two types of constraints: Must-links and Cannot-links Clustering algorithm: Hierarchical clustering

56. Constraints

57. Overview of Hierarchical Clustering Algorithm Agglomerative versus Divisive Basic algorithm of Agglomerative clustering 1.Compute the distance matrix 2.Let each data point be a cluster 3.Repeat 4.Merge the two closest clusters 5.Update the distance matrix 6.Until only a single cluster remains Key operation is the update of the distance between two clusters

58. How to Define Inter-Cluster Distance p1 p3 p5 p4 p2 p1p2p3p4p5......... Distance? MIN MAX Group Average Distance Between Centroids distance matrix

59. Must-link constraints Distance between must-links pair to zero. Derive a new metric by running an all-pairs- shortest distances algorithm. –It is still a metric –Faithful to the original metric –Computational complexity: O( N 2 C) C: number of points involved in must-link constraints N: total number of points

60. New distance matrix based on must-link constraints p1 p3 p5 p4 p2 p1p2p3p4p5......... New distance matrix Hierarchical clustering can be carried out based on the new distance matrix. What is missing?

61. Cannot-link constraint Run hierarchical clustering with complete link (MAX) –The distance between two clusters is determined by the largest distance. Set the distance between cannot-link pair to be The new distance matrix does not define a metric. –Work very well in practice

62. Constrained complete-link clustering algorithm Derive a new distance Matrix based on both Types of constraints

63. Illustration 1 2 3 4 5 00.20.50.10.8 00.40.20.6 00.30.2 00.5 0 Initial distance matrix

64. New distance matrix 00.20.50.10.8 00.40.20.6 00.30.2 00.5 0 000.1 0.8 00.2 0.6 000.2 0 0 Must-links: 1—2, 3—4 Cannot-links: 2--3 0.9

65. Hierarchical clustering 1 and 2 form a cluster, and 3 and 4 form another cluster 00.90.8 00.2 0 1,23,4 5 1,2 3,4 5 1 2 3 4 5

66. Summary Seeded and Constrained K-Means: partially labeled data COP K-Means: constraints (Must-link and Cannot-link) Constrained K-Means and COP K-Means require all the constraints to be satisfied. –May not be effective if the seeds contain noise. Seeded K-Means use the seeds only in the first step to determine the initial centroids. –Less sensitive to the noise in the seeds. Semi-supervised hierarchical clustering