The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL Bi-Clustering COMP Seminar Spring 2011

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 2 Data Mining: Clustering Where K-means clustering minimizes

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 3 Clustering by Pattern Similarity (p-Clustering) The micro-array “raw” data shows 3 genes and their values in a multi-dimensional space  Parallel Coordinates Plots  Difficult to find their patterns “non-traditional” clustering

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 4 Clusters Are Clear After Projection

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 5 Motivation E-Commerce: collaborative filtering Movie 1 Movie 2 Movie 3 Movie 4 Movie 5 Movie 6 Movie 7 Viewer Viewer Viewer Viewer Viewer 55534

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 6 Motivation

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 7 Motivation Movie 1 Movie 2 Movie 3 Movie 4 Movie 5 Movie 6 Movie 7 Viewer Viewer Viewer Viewer Viewer 55534

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 8 Motivation

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 9 Motivation DNA microarray analysis CH1ICH1BCH1DCH2ICH2B CTFC VPS EFB SSA FUN SP MDM CYS DEP NTG

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 10 Motivation

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 11 Motivation Strong coherence exhibits by the selected objects on the selected attributes.  They are not necessarily close to each other but rather bear a constant shift.  Object/attribute bias bi-cluster

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 12 Challenges The set of objects and the set of attributes are usually unknown. Different objects/attributes may possess different biases and such biases  may be local to the set of selected objects/attributes  are usually unknown in advance May have many unspecified entries

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 13 Previous Work Subspace clustering  Identifying a set of objects and a set of attributes such that the set of objects are physically close to each other on the subspace formed by the set of attributes. Collaborative filtering: Pearson R  Only considers global offset of each object/attribute.

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 14 bi-cluster Consists of a (sub)set of objects and a (sub)set of attributes  Corresponds to a submatrix  Occupancy threshold   Each object/attribute has to be filled by a certain percentage.  Volume: number of specified entries in the submatrix  Base: average value of each object/attribute (in the bi-cluster)

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 15 bi-cluster CH1ICH1BCH1DCH2ICH2BObj base CTFC3 VPS EFB SSA1 FUN14 SP07 MDM10 CYS DEP1 NTG1 Attr base

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 16 bi-cluster Perfect  -cluster Imperfect  -cluster  Residue: d IJ d Ij d iJ d ij

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 17 bi-cluster The smaller the average residue, the stronger the coherence. Objective: identify  -clusters with residue smaller than a given threshold

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 18 Cheng-Church Algorithm Find one bi-cluster. Replace the data in the first bi-cluster with random data Find the second bi-cluster, and go on. The quality of the bi-cluster degrades (smaller volume, higher residue) due to the insertion of random data.

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 19 The FLOC algorithm Generating initial clusters Determine the best action for each row and each column Perform the best action of each row and column sequentially Improved? Y N

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 20 The FLOC algorithm Action: the change of membership of a row(or column) with respect to a cluster column row M+N actions are Performed at each iteration N=3 M=4

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 21 The FLOC algorithm Gain of an action: the residue reduction incurred by performing the action Order of action:  Fixed order  Random order  Weighted random order Complexity: O((M+N)MNkp) 

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 22 The FLOC algorithm Additional features  Maximum allowed overlap among clusters  Minimum coverage of clusters  Minimum volume of each cluster Can be enforced by “temporarily blocking” certain action during the mining process if such action would violate some constraint.

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 23 Performance Microarray data: 2884 genes, 17 conditions  100 bi-clusters with smallest residue were returned.  Average residue =  The average residue of clusters found via the state of the art method in computational biology field is  The average volume is 25% bigger  The response time is an order of magnitude faster

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 24 Conclusion Remark The model of bi-cluster is proposed to capture coherent objects with incomplete data set.  base  residue Many additional features can be accommodated (nearly for free).

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 25 Coherent Cluster Want to accommodate noise but not outliers

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 26 Coherent Cluster Coherent cluster  Subspace clustering pair-wise disparity  For a 2  2 (sub)matrix consisting of objects {x, y} and attributes {a, b} x y ab d xa d ya d xb d yb x y ab z attribute mutual bias of attribute a mutual bias of attribute b

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 27 Coherent Cluster  A 2  2 (sub)matrix is a  -coherent cluster if its D value is less than or equal to .  An m  n matrix X is a  -coherent cluster if every 2  2 submatrix of X is  -coherent cluster.  A  -coherent cluster is a maximum  -coherent cluster if it is not a submatrix of any other  -coherent cluster.  Objective: given a data matrix and a threshold , find all maximum  -coherent clusters.

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 28 Coherent Cluster Challenges:  Finding subspace clustering based on distance itself is already a difficult task due to the curse of dimensionality.  The (sub)set of objects and the (sub)set of attributes that form a cluster are unknown in advance and may not be adjacent to each other in the data matrix.  The actual values of the objects in a coherent cluster may be far apart from each other.  Each object or attribute in a coherent cluster may bear some relative bias (that are unknown in advance) and such bias may be local to the coherent cluster.

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 29 Coherent Cluster Compute the maximum coherent attribute sets for each pair of objects Construct the lexicographical tree Post-order traverse the tree to find maximum coherent clusters Two-way Pruning

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 30 Coherent Cluster Observation: Given a pair of objects {o 1, o 2 } and a (sub)set of attributes {a 1, a 2, …, a k }, the 2  k submatrix is a  -coherent cluster iff, for every attribute a i, the mutual bias (d o1ai – d o2ai ) does not differ from each other by more than . a1a1 a2a2 a3a3 a4a4 a5a o1o1 o2o2  [2, 3.5] If  = 1.5, then {a 1,a 2,a 3,a 4,a 5 } is a coherent attribute set (CAS) of (o 1,o 2 ).

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 31 Coherent Cluster Observation: given a subset of objects {o 1, o 2, …, o l } and a subset of attributes {a 1, a 2, …, a k }, the l  k submatrix is a  -coherent cluster iff {a 1, a 2, …, a k } is a coherent attribute set for every pair of objects (o i,o j ) where 1  i, j  l. a1a1 a5a5 a6a6 a7a7 a2a2 a3a3 a4a4 o1o1 o3o3 o4o4 o5o5 o6o6 o2o2

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 32 a1a1 a2a2 a3a3 a4a4 a5a r1r1 r2r2 Coherent Cluster Strategy: find the maximum coherent attribute sets for each pair of objects with respect to the given threshold .  = r1r1 r2r2 a2a2 2 a3a3 3.5 a4a4 2 a5a5 2.5 a1a1 3 1 The maximum coherent attribute sets define the search space for maximum coherent clusters.

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 33 Two Way Pruning a0a1a2 o0142 o1255 o2365 o o (o0,o2) →(a0,a1,a2) (o1,o2) →(a0,a1,a2) (a0,a1) →(o0,o1,o2) (a0,a2) →(o1,o2,o3) (a1,a2) →(o1,o2,o4) (a1,a2) →(o0,o2,o4) (o0,o2) →(a0,a1,a2) (o1,o2) →(a0,a1,a2) (a0,a1) →(o0,o1,o2) (a0,a2) →(o1,o2,o3) (a1,a2) →(o1,o2,o4) (a1,a2) →(o0,o2,o4) delta=1 nc =3 nr = 3 MCAS MCOS

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 34 Coherent Cluster Strategy: grouping object pairs by their CAS and, for each group, find the maximum clique(s). Implementation: using a lexicographical tree to organize the object pairs and to generate all maximum coherent clusters with a single post-order traversal of the tree. objects attributes

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 35 (o 0,o 1 ) : {a 0,a 1 }, {a 2,a 3 } (o 0,o 2 ) : {a 0,a 1,a 2,a 3 } (o 0,o 4 ) : {a 1,a 2 } (o 1,o 2 ) : {a 0,a 1,a 2 }, {a 2,a 3 } (o 1,o 3 ) : {a 0,a 2 } (o 1,o 4 ) : {a 1,a 2 } (o 2,o 3 ) : {a 0,a 2 } (o 2,o 4 ) : {a 1,a 2 } a0a0 a1a1 a2a2 a3a3 o0o o1o o2o o3o o4o a0a0 a1a1 a2a2 a2a2 a3a3 a1a1 a2a2 a2a2 a3a3 (o0,o1)(o0,o1) (o1,o2)(o1,o2) (o0,o2)(o0,o2) (o1,o3)(o1,o3) (o2,o3)(o2,o3) (o0,o4)(o0,o4) (o1,o4)(o1,o4) (o2,o4)(o2,o4) (o0,o1)(o0,o1) (o1,o2)(o1,o2) assume  = 1 {a 0,a 1 } : (o 0,o 1 ) {a 0,a 2 } : (o 1,o 3 ),(o 2,o 3 ) {a 1,a 2 } : (o 0,o 4 ),(o 1,o 4 ),(o 2,o 4 ) {a 2,a 3 } : (o 0,o 1 ),(o 1,o 2 ) {a 0,a 1,a 2 } : (o 1,o 2 ) {a 0,a 1,a 2,a 3 } : (o 0,o 2 ) (o1,o2)(o1,o2) (o1,o2)(o1,o2) (o1,o2)(o1,o2) (o0,o2)(o0,o2) (o0,o2)(o0,o2) (o0,o2)(o0,o2) (o0,o2)(o0,o2) (o0,o2)(o0,o2)

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 36 a0a0 a1a1 a2a2 a2a2 a3a3 a1a1 a2a2 a2a2 a3a3 (o0,o1)(o0,o1) (o1,o2)(o1,o2) (o0,o2)(o0,o2) (o1,o3)(o1,o3) (o2,o3)(o2,o3) (o0,o4)(o0,o4) (o1,o4)(o1,o4) (o2,o4)(o2,o4) (o0,o1)(o0,o1) (o1,o2)(o1,o2) (o0,o2)(o0,o2) (o0,o2)(o0,o2) a3a3 (o0,o2)(o0,o2) a3a3 (o0,o2)(o0,o2) a3a3 {o 0,o 2 }  {a 0,a 1,a 2,a 3 } (o1,o2)(o1,o2) (o0,o2)(o0,o2)(o1,o2)(o1,o2) (o0,o2)(o0,o2)(o1,o2)(o1,o2) (o0,o2)(o0,o2) {o 1,o 2 }  {a 0,a 1,a 2 } {o 0,o 1,o 2 }  {a 0,a 1 } a3a3 (o0,o2)(o0,o2) a3a3 (o0,o2)(o0,o2) (o0,o2)(o0,o2) {o 1,o 2,o 3 }  {a 0,a 2 } {o 0,o 2,o 4 }  {a 1,a 2 } {o 1,o 2,o 4 }  {a 1,a 2 } {o 0,o 1,o 2 }  {a 2,a 3 }

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 37 Coherent Cluster High expressive power  The coherent cluster can capture many interesting and meaningful patterns overlooked by previous clustering methods. Efficient and highly scalable Wide applications  Gene expression analysis  Collaborative filtering subspace cluster coherent cluster

The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL 38 Remark Comparing to Bicluster  Can well separate noises and outliers  No random data insertion and replacement  Produce optimal solution