1. 11/21/2015 1 Data Mining: Concepts and Techniques (3 rd ed.) — Chapter 11 — Jiawei Han, Micheline Kamber, and Jian Pei University of Illinois at Urbana-Champaign & Simon Fraser University ©2009 Han, Kamber & Pei. All rights reserved. 1

2. November 21, 2015Data Mining: Concepts and Techniques2

3. 3 Chapter 11. Cluster Analysis: Advanced Methods Statistics-Based Clustering Clustering High-Dimensional Data Semi-Supervised Learning and Active Learning Constraint-Based Clustering Bi-Clustering and co-Clustering Collaborative filtering Spectral Clustering Evaluation of Clustering Quality Summary 3

4. 4 Model-Based Clustering What is model-based clustering? Assumption: a cluster is generated by a model such as a probability distribution A model (e.g., Gaussian distribution) is determined by a set of parameters Task: optimize the fit between the given data and some mathematical models by learning the parameters of the models Typical methods Statistical approach EM (Expectation maximization), AutoClass Neural network approach SOM (Self-Organizing Feature Map)

5. Mixture Models A cluster can be modeled as a probability distribution Practically, assume a distribution can be approximated well using multivariate normal distribution Multiple clusters is a mixture of different probability distributions A data set is a set of observations from a mixture of models 5

6. Object Probability Suppose there are k clusters and a set X of m objects Let the j-th cluster have parameter  j = (  j,  j ) The probability that a point is in the j-th cluster is w j, where w 1 + …+ w k = 1 The probability of an object x is 6

7. Example 7

8. Maximal Likelihood Estimation Maximum likelihood principle: If we know a set of objects are from one distribution, but do not know the parameter, we can choose the parameter maximizing the probability Maximize Equivalently, maximize 8

9. The EM (Expectation Maximization) Algorithm Expectation Maximization algorithm Select an initial set of model parameters Repeat Expectation Step: For each object, calculate the probability that it belongs to each distribution  i, i.e., prob(x i |  i ) Maximization Step: Given the probabilities from the expectation step, find the new estimates of the parameters that maximize the expected likelihood Until the parameters are stable 9

10. Advantages and Disadvantages Mixture models are more general than k-means and fuzzy c-means Clusters can be characterized by a small number of parameters The results may satisfy the statistical assumptions of the generative models Computationally expensive Need large data sets Hard to estimate the number of clusters 10

11. 11 Neural Network Approaches Neural network approaches Represent each cluster as an exemplar, acting as a “prototype” of the cluster New objects are distributed to the cluster whose exemplar is the most similar according to some distance measure Typical methods SOM (Soft-Organizing feature Map) Competitive learning Involves a hierarchical architecture of several units (neurons) Neurons compete in a “winner-takes-all” fashion for the object currently being presented

12. 12 Self-Organizing Feature Map (SOM) SOMs, also called topological ordered maps, or Kohonen Self- Organizing Feature Map (KSOMs) It maps all the points in a high-dimensional source space into a 2 to 3-d target space, s.t., the distance and proximity relationship (i.e., topology) are preserved as much as possible Similar to k-means: cluster centers tend to lie in a low-dimensional manifold in the feature space Clustering is performed by having several units competing for the current object The unit whose weight vector is closest to the current object wins The winner and its neighbors learn by having their weights adjusted SOMs are believed to resemble processing that can occur in the brain Useful for visualizing high-dimensional data in 2- or 3-D space

13. 13 Web Document Clustering Using SOM The result of SOM clustering of 12088 Web articles The picture on the right: drilling down on the keyword “mining” Based on websom.hut.fi Web page

14. 14 Chapter 11. Cluster Analysis: Advanced Methods Statistics-Based Clustering Clustering High-Dimensional Data Semi-Supervised Learning and Active Learning Constraint-Based Clustering Bi-Clustering and co-Clustering Collaborative filtering Spectral Clustering Evaluation of Clustering Quality Summary 14

15. 15 Clustering High-Dimensional Data Clustering high-dimensional data Many applications: text documents, DNA micro-array data Major challenges: Many irrelevant dimensions may mask clusters Distance measure becomes meaningless—due to equi-distance Clusters may exist only in some subspaces Methods Feature transformation: only effective if most dimensions are relevant PCA & SVD useful only when features are highly correlated/redundant Feature selection: wrapper or filter approaches useful to find a subspace where the data have nice clusters Subspace-clustering: find clusters in all the possible subspaces CLIQUE, ProClus, and frequent pattern-based clustering

16. 16 The Curse of Dimensionality (graphs adapted from Parsons et al. KDD Explorations 2004) Data in only one dimension is relatively packed Adding a dimension “stretch” the points across that dimension, making them further apart Adding more dimensions will make the points further apart — high dimensional data is extremely sparse Distance measure becomes meaningless — due to equi-distance

17. 17 Why Subspace Clustering? (adapted from Parsons et al. SIGKDD Explorations 2004) Clusters may exist only in some subspaces Subspace-clustering: find clusters in all the subspaces

18. 18 CLIQUE (Clustering In QUEst) Agrawal, Gehrke, Gunopulos, Raghavan (SIGMOD’98) Automatically identifying subspaces of a high dimensional data space that allow better clustering than original space CLIQUE can be considered as both density-based and grid-based It partitions each dimension into the same number of equal length interval It partitions an m-dimensional data space into non-overlapping rectangular units A unit is dense if the fraction of total data points contained in the unit exceeds the input model parameter A cluster is a maximal set of connected dense units within a subspace

19. 19 CLIQUE: The Major Steps Partition the data space and find the number of points that lie inside each cell of the partition. Identify the subspaces that contain clusters using the Apriori principle Identify clusters Determine dense units in all subspaces of interests Determine connected dense units in all subspaces of interests. Generate minimal description for the clusters Determine maximal regions that cover a cluster of connected dense units for each cluster Determination of minimal cover for each cluster

20. 20 Salary (10,000) 2030405060 age 5 4 3 1 2 6 7 0 2030405060 age 5 4 3 1 2 6 7 0 Vacation (week) age Vacation Salary 3050  = 3

21. 21 Strength and Weakness of CLIQUE Strength automatically finds subspaces of the highest dimensionality such that high density clusters exist in those subspaces insensitive to the order of records in input and does not presume some canonical data distribution scales linearly with the size of input and has good scalability as the number of dimensions in the data increases Weakness The accuracy of the clustering result may be degraded at the expense of simplicity of the method

22. November 21, 2015Data Mining: Concepts and Techniques22 Frequent Pattern-Based Approach Clustering high-dimensional space (e.g., clustering text documents, microarray data) Projected subspace-clustering: which dimensions to be projected on? CLIQUE, ProClus Feature extraction: costly and may not be effective? Using frequent patterns as “features” Clustering by pattern similarity in micro-array data (pClustering) [H. Wang, W. Wang, J. Yang, and P. S. Yu. Clustering by pattern similarity in large data sets, SIGMOD’02]

23. 23 Clustering by Pattern Similarity (p-Clustering) Left figure: Micro-array “raw” data shows 3 genes and their values in a multi-D space: Difficult to find their patterns Right two: Some subsets of dimensions form nice shift and scaling patterns No globally defined similarity/distance measure Clusters may not be exclusive An object can appear in multiple clusters

24. November 21, 2015Data Mining: Concepts and Techniques24 Why p-Clustering? Microarray data analysis may need to Clustering on thousands of dimensions (attributes) Discovery of both shift and scaling patterns Clustering with Euclidean distance measure? — cannot find shift patterns Clustering on derived attribute A ij = a i – a j ? — introduces N(N-1) dimensions Bi-cluster (Y. Cheng and G. Church. Biclustering of expression data. ISMB’00 ) using transformed mean-squared residue score matrix (I, J) Where A submatrix is a δ-cluster if H(I, J) ≤ δ for some δ > 0 Problems with bi-cluster No downward closure property Due to averaging, it may contain outliers but still within δ-threshold

25. p-Clustering: Clustering by Pattern Similarity P-score: the similarity between two objects r x, r y on two attributes a u, a v δ-pCluster: If for any 2 by 2 matrix X, pScore(X) ≤ δ (δ > 0) Properties of δ-pCluster Downward closure Clusters are more homogeneous than bi-cluster (thus the name: pair-wise Cluster) MaPle (Pei et al. 2003): Efficient mining of maximum p-clusters For scaling patterns, taking logarithmic on will lead to the pScore form

26. 26 Chapter 11. Cluster Analysis: Advanced Methods Statistics-Based Clustering Clustering High-Dimensional Data Semi-Supervised Learning and Active Learning Constraint-Based Clustering Bi-Clustering and co-Clustering Collaborative filtering Spectral Clustering Evaluation of Clustering Quality Summary 26

27. Why Semi-Supervised Learning? Sparsity in data: training examples cannot cover the data space well unlabeled data can help to address sparsity 27

28. Semi-Supervised Learning Methods Many methods exist: EM with generative mixture models, self-training, co-training, data-based methods, transductive SVM, graph-based methods, … Inductive methods and Transductive methods Transductive methods: only label the available unlabeled data – not generating a classifier Inductive methods: not only produce labels for unlabeled data, but also generate a classifier Algorithmic methods Classifier-based methods: start from an initial classifier, and iteratively enhance it Data-based methods: find an inherent geometry in the data, and use the geometry to find a good classifier 28

29. Co-Training 29

30. Graph Mincuts Positive samples as sources and negative samples as sinks Unlabeled samples are connected to other samples with weights based on similarity Objective: find a minimum set of edges to remove so that all flows from sources to sinks are blocked 30

31. 31 Chapter 11. Cluster Analysis: Advanced Methods Statistics-Based Clustering Clustering High-Dimensional Data Semi-Supervised Learning and Active Learning Constraint-Based Clustering Bi-Clustering and co-Clustering Collaborative filtering Spectral Clustering Evaluation of Clustering Quality Summary 31

32. 32 Why Constraint-Based Cluster Analysis? Need user feedback: Users know their applications the best Less parameters but more user-desired constraints, e.g., an ATM allocation problem: obstacle & desired clusters

33. 33 A Classification of Constraints in Cluster Analysis Clustering in applications: desirable to have user-guided (i.e., constrained) cluster analysis Different constraints in cluster analysis: Constraints on individual objects (do selection first) Cluster on houses worth over $300K Constraints on distance or similarity functions Weighted functions, obstacles (e.g., rivers, lakes) Constraints on the selection of clustering parameters # of clusters, MinPts, etc. User-specified constraints Contain at least 500 valued customers and 5000 ordinary ones Semi-supervised: giving small training sets as “constraints” or hints

34. 34 Clustering With Obstacle Objects Tung, Hou, and Han. Spatial Clustering in the Presence of Obstacles, ICDE'01 K-medoids is more preferable since k-means may locate the ATM center in the middle of a lake Visibility graph and shortest path Triangulation and micro-clustering Two kinds of join indices (shortest- paths) worth pre-computation VV index: indices for any pair of obstacle vertices MV index: indices for any pair of micro-cluster and obstacle indices

35. 35 An Example: Clustering With Obstacle Objects Taking obstacles into accountNot Taking obstacles into account

36. 36 User-Guided Clustering name office position Professor course-id name area course semester instructor office position Student name student course semester unit Register grade professor student degree Advise name Group person group Work-In area year conf Publication title Publish author Target of clustering User hint Course Open-course X. Yin, J. Han, P. S. Yu, “Cross-Relational Clustering with User's Guidance”, KDD'05 User usually has a goal of clustering, e.g., clustering students by research area User specifies his clustering goal to CrossClus

37. 37 Comparing with Classification User-specified feature (in the form of attribute) is used as a hint, not class labels The attribute may contain too many or too few distinct values, e.g., a user may want to cluster students into 20 clusters instead of 3 Additional features need to be included in cluster analysis All tuples for clustering User hint

38. 38 Comparing with Semi-Supervised Clustering Semi-supervised clustering: User provides a training set consisting of “similar” (“must-link) and “dissimilar” (“cannot link”) pairs of objects User-guided clustering: User specifies an attribute as a hint, and more relevant features are found for clustering All tuples for clustering Semi-supervised clustering All tuples for clustering User-guided clustering x

39. 39 Why Not Semi-Supervised Clustering? Much information (in multiple relations) is needed to judge whether two tuples are similar A user may not be able to provide a good training set It is much easier for a user to specify an attribute as a hint, such as a student’s research area Tom SmithSC1211TA Jane ChangBI205RA Tuples to be compared User hint

40. 40 CrossClus: An Overview Measure similarity between features by how they group objects into clusters Use a heuristic method to search for pertinent features Start from user-specified feature and gradually expand search range Use tuple ID propagation to create feature values Features can be easily created during the expansion of search range, by propagating IDs Explore three clustering algorithms: k-means, k-medoids, and hierarchical clustering

41. 41 Multi-Relational Features A multi-relational feature is defined by: A join path, e.g., Student → Register → OpenCourse → Course An attribute, e.g., Course.area (For numerical feature) an aggregation operator, e.g., sum or average Categorical feature f = [Student → Register → OpenCourse → Course, Course.area, null] TupleAreas of courses DBAITH t1t1 550 t2t2 037 t3t3 154 t4t4 505 t5t5 334 areas of courses of each student TupleFeature f DBAITH t1t1 0.5 0 t2t2 00.30.7 t3t3 0.10.50.4 t4t4 0.50 t5t5 0.3 0.4 Values of feature f f(t1)f(t1) f(t2)f(t2) f(t3)f(t3) f(t4)f(t4) f(t5)f(t5)

42. 42 Representing Features Similarity between tuples t 1 and t 2 w.r.t. categorical feature f Cosine similarity between vectors f(t 1 ) and f(t 2 ) Most important information of a feature f is how f groups tuples into clusters f is represented by similarities between every pair of tuples indicated by f The horizontal axes are the tuple indices, and the vertical axis is the similarity This can be considered as a vector of N x N dimensions Similarity vector V f

43. 43 Similarity Between Features Feature f (course)Feature g (group) DBAITHInfo sysCog sciTheory t1t1 0.5 0100 t2t2 00.30.7001 t3t3 0.10.50.400.5 t4t4 0 0 t5t5 0.3 0.40.5 0 Values of Feature f and g Similarity between two features – cosine similarity of two vectors VfVf VgVg

44. 44 Computing Feature Similarity Tuples Feature f Feature g DB AI TH Info sys Cog sci Theory Similarity between feature values w.r.t. the tuples sim(f k,g q )= Σ i=1 to N f(t i ).p k ∙g(t i ).p q DB Info sys Tuple similarities, hard to compute Feature value similarities, easy to compute DB AI TH Info sys Cog sci Theory Compute similarity between each pair of feature values by one scan on data

45. 45 Searching for Pertinent Features Different features convey different aspects of information Features conveying same aspect of information usually cluster tuples in more similar ways Research group areas vs. conferences of publications Given user specified feature Find pertinent features by computing feature similarity Research group area Advisor Conferences of papers Research area GPA Number of papers GRE score Academic Performances Nationality Permanent address Demographic info

46. 46 Heuristic Search for Pertinent Features Overall procedure 1. Start from the user- specified feature 2. Search in neighborhood of existing pertinent features 3. Expand search range gradually name office position Professor office position Student name student course semester unit Register grade professor student degree Advise person group Work-In name Group areayear conf Publication title Publish author Target of clustering User hint course-id name area Course course semester instructor Open-course 1 2 Tuple ID propagation is used to create multi-relational features IDs of target tuples can be propagated along any join path, from which we can find tuples joinable with each target tuple

47. 47 Clustering with Multi-Relational Features Given a set of L pertinent features f 1, …, f L, similarity between two tuples Weight of a feature is determined in feature search by its similarity with other pertinent features Clustering methods CLARANS [Ng & Han 94], a scalable clustering algorithm for non-Euclidean space K-means Agglomerative hierarchical clustering

48. 48 Experiments: Compare CrossClus with Baseline: Only use the user specified feature PROCLUS [Aggarwal, et al. 99]: a state-of-the-art subspace clustering algorithm Use a subset of features for each cluster We convert relational database to a table by propositionalization User-specified feature is forced to be used in every cluster RDBC [Kirsten and Wrobel’00] A representative ILP clustering algorithm Use neighbor information of objects for clustering User-specified feature is forced to be used

49. 49 Measure of Clustering Accuracy Accuracy Measured by manually labeled data We manually assign tuples into clusters according to their properties (e.g., professors in different research areas) Accuracy of clustering: Percentage of pairs of tuples in the same cluster that share common label This measure favors many small clusters We let each approach generate the same number of clusters

50. 50 DBLP Dataset

51. 51 Chapter 11. Cluster Analysis: Advanced Methods Statistics-Based Clustering Clustering High-Dimensional Data Semi-Supervised Learning and Active Learning Constraint-Based Clustering Bi-Clustering and co-Clustering Collaborative filtering Spectral Clustering Evaluation of Clustering Quality Summary 51

52. Bi-clustering and Co-clustering Biclustering, co-clustering, or two-mode clustering allows simultaneous clustering of the rows and columns of a matrix Given a set of m rows in n columns (i.e., an m×n matrix), a biclustering algorithm generates biclusters – a subset of rows which exhibit similar behavior across a subset of columns, or vice versa 52

53. 53 Chapter 11. Cluster Analysis: Advanced Methods Statistics-Based Clustering Clustering High-Dimensional Data Semi-Supervised Learning and Active Learning Constraint-Based Clustering Bi-Clustering and co-Clustering Collaborative filtering Spectral Clustering Evaluation of Clustering Quality Summary 53

54. Collaborative Filtering Collaborative filtering (CF) is the process of filtering for information or patterns using techniques involving collaboration among multiple agents, viewpoints, data sources, etc. Applications involving very large data sets: sensing and monitoring data, financial data, electronic commerce and web 2.0 applications Example: a method of making automatic predictions (filtering) about the interests of a user by collecting taste information from many users (collaborating) Assumption: those who agreed in the past tend to agree again in the future 54

55. Framework A general 2-step process Look for users who share the same rating patterns with the active user (the one for which the prediction is made) Use the ratings of those found in step 1 to calculate a prediction Item-based filtering (used in Amazon.com) Build an item-to-item matrix determining the relationships between each pair of items Infer the user’s taste (i.e., the prediction) using the matrix 55

56. 56 Chapter 11. Cluster Analysis: Advanced Methods Statistics-Based Clustering Clustering High-Dimensional Data Semi-Supervised Learning and Active Learning Constraint-Based Clustering Bi-Clustering and co-Clustering Collaborative filtering Spectral Clustering Evaluation of Clustering Quality Summary 56

57. Spectral Clustering Given a set of data points A, a similarity matrix S may be defined where S ij represents a measure of the similarity between points i and j (i, j ∊ A) Spectral clustering makes use of the spectrum of the similarity matrix of the data to perform dimensionality reduction for clustering in fewer dimensions In functional analysis, the spectrum of a bounded operator is a generalization of eigenvalues for matrices A complex number λ is said to be in the spectrum of a bounded linear operator T if λI − T is not invertible, where I is the identity operator 57

58. Shi-Malik Algorithm Given a set S of points, the algorithm partitions the points into two sets S 1 and S 2 Let v be the eigenvector v corresponding to the second-smallest eigenvalue of the Laplacian matrix L of S L = I – D –1/2 SD –1/2, where D is the diagonal matrix Let m be the median of the components in v Place all points whose component in v is greater than m in S 1, and the rest in S 2 58

59. Example 59 Extraction from http://www.kimbly.com/blog/000489.html

60. 60 Chapter 11. Cluster Analysis: Advanced Methods Statistics-Based Clustering Clustering High-Dimensional Data Semi-Supervised Learning and Active Learning Constraint-Based Clustering Bi-Clustering and co-Clustering Collaborative filtering Spectral Clustering Evaluation of Clustering Quality Summary 60

61. 61 Summary Cluster analysis groups objects based on their similarity and has wide applications Measure of similarity can be computed for various types of data Clustering algorithms can be categorized into partitioning methods, hierarchical methods, density-based methods, grid-based methods, and model-based methods Outlier detection and analysis are very useful for fraud detection, etc. and can be performed by statistical, distance-based or deviation-based approaches There are still lots of research issues on cluster analysis

62. 62 Problems and Challenges Considerable progress has been made in scalable clustering methods Partitioning: k-means, k-medoids, CLARANS Hierarchical: BIRCH, ROCK, CHAMELEON Density-based: DBSCAN, OPTICS, DenClue Grid-based: STING, WaveCluster, CLIQUE Model-based: EM, Cobweb, SOM Frequent pattern-based: pCluster Constraint-based: COD, constrained-clustering Current clustering techniques do not address all the requirements adequately, still an active area of research

63. 63 References G. J. McLachlan and K.E. Bkasford. Mixture Models: Inference and Applications to Clustering. John Wiley and Sons, 1988. P. Michaud. Clustering Techniques. Future Generation Computer Systems, 13, 1997. A. K. Jain and R. C. Dubes. Algorithms for Clustering Data. Printice Hall, 1988. L. Parsons, E. Haque and H. Liu, Subspace Clustering for High Dimensional Data: A Review, SIGKDD Explorations, 6(1), June 2004 E. Schikuta. Grid clustering: An efficient hierarchical clustering method for very large data sets. Proc. 1996 Int. Conf. on Pattern Recognition,. A. K. H. Tung, J. Han, L. V. S. Lakshmanan, and R. T. Ng. Constraint-Based Clustering in Large Databases, ICDT'01. A. K. H. Tung, J. Hou, and J. Han. Spatial Clustering in the Presence of Obstacles, ICDE'01 H. Wang, W. Wang, J. Yang, and P.S. Yu. Clustering by pattern similarity in large data sets, SIGMOD’ 02. X. Yin, J. Han, and P.S. Yu, “Cross-Relational Clustering with User's Guidance”, in Proc. 2005 Int. Conf. on Knowledge Discovery and Data Mining (KDD'05), Chicago, IL, Aug. 2005.Cross-Relational Clustering with User's Guidance

64. 64

65. Slides Not to Be Used in Class 65

66. Chapter 11. Cluster Analysis: Advanced Methods Statistics-Based Clustering Model-Based Clustering: The Expectation-Maximization Method Neural Network Approach (SOM) Fuzzy and non-crisp clustering Clustering High-Dimensional Data Why Subspace Clustering?—Challenges on Clustering High-Dimensional Data PROCLUS: A Dimension-Reduction Subspace Clustering Method Frequent Pattern-Based Clustering Methods Semi-Supervised Learning and Active Learning Semi-supervised clustering Classification of partially labeled data Constraint-Based and User-Guided Cluster Analysis Clustering with Obstacle Objects User-Constrained Cluster Analysis User-Guided Cluster Analysis Bi-Clustering and co-Clustering Collaborative filtering Clustering-based approach Classification-Based Approach Frequent Pattern-Based Approach Spectral Clustering Evaluation of Clustering Quality Summary 66

67. 67 Conceptual Clustering Conceptual clustering A form of clustering in machine learning Produces a classification scheme for a set of unlabeled objects Finds characteristic description for each concept (class) COBWEB (Fisher’87) A popular a simple method of incremental conceptual learning Creates a hierarchical clustering in the form of a classification tree Each node refers to a concept and contains a probabilistic description of that concept

68. 68 COBWEB Clustering Method A classification tree

69. 69 More on Conceptual Clustering Limitations of COBWEB The assumption that the attributes are independent of each other is often too strong because correlation may exist Not suitable for clustering large database data – skewed tree and expensive probability distributions CLASSIT an extension of COBWEB for incremental clustering of continuous data suffers similar problems as COBWEB AutoClass (Cheeseman and Stutz, 1996) Uses Bayesian statistical analysis to estimate the number of clusters Popular in industry