1. Subspace Clustering/Biclustering
CS 685: Special Topics in Data Mining
Spring 2008
Jinze Liu

2. Data Mining: Clustering
K-means clustering minimizes
Where

3. Clustering by Pattern Similarity (p-Clustering)
The micro-array “raw” data shows 3 genes and their values in a multi-dimensional space
Parallel Coordinates Plots
Difficult to find their patterns
“non-traditional” clustering

4. Clusters Are Clear After Projection

5. Motivation E-Commerce: collaborative filtering Movie 1 Movie 2 Movie 3
Viewer 1 1 2 4 3 5
Viewer 2 6 7
Viewer 3
Viewer 4
Viewer 5

6. Motivation

7. Motivation Movie 1 Movie 2 Movie 3 Movie 4 Movie 5 Movie 6 Movie 7
Viewer 1 1 2 4 3 5
Viewer 2 6 7
Viewer 3
Viewer 4
Viewer 5

8. Motivation

9. Gene Expression Data

10. Biclustering of Gene Expression Data
Genes not regulated under all conditions
Genes regulated by multiple factors/processes concurrently
Key to determine function of genes
Key to determine classification of conditions

11. Motivation DNA microarray analysis CH1I CH1B CH1D CH2I CH2B CTFC3 4392
284
4108
280
228
VPS8
401
281
120
275
298
EFB1
318 37
277
215
SSA1
292
109
580
238
FUN14
2857
285
2576
271
226
SP07
290 48
224
MDM10
538
272
266
236
CYS3
322
288 41
278
219
DEP1
312 40
273
232
NTG1
329
296 33
274

12. Motivation

13. Motivation
Strong coherence exhibits by the selected objects on the selected attributes.
They are not necessarily close to each other but rather bear a constant shift.
Object/attribute bias
bi-cluster

14. Challenges
The set of objects and the set of attributes are usually unknown.
Different objects/attributes may possess different biases and such biases
may be local to the set of selected objects/attributes
are usually unknown in advance
May have many unspecified entries

15. What’s Biclustering?
Given an n x m matrix, A, find a set of submatrices, Bk, such that the contents of each Bk follow a desired pattern
Row/Column order need not be consistent between different Bks

16. Bipartite Graphs
Matrix can be thought of as a Graph Rows are one set of vertices L, Columns are another set R
Edges are weighted by the corresponding entries in the matrix If all weights are binary, biclustering becomes biclique finding

17. Bicluster Structures

18. Previous Work Subspace clustering Collaborative filtering: Pearson R
Identifying a set of objects and a set of attributes such that the set of objects are physically close to each other on the subspace formed by the set of attributes.
Collaborative filtering: Pearson R
Only considers global offset of each object/attribute.

19. bi-cluster
Consists of a (sub)set of objects and a (sub)set of attributes
Corresponds to a submatrix
Occupancy threshold 
Each object/attribute has to be filled by a certain percentage.
Volume: number of specified entries in the submatrix
Base: average value of each object/attribute (in the bi-cluster)

20. bi-cluster CH1I CH1B CH1D CH2I CH2B Obj base CTFC3 VPS8 401 120 298
273
EFB1
318 37
215
190
SSA1
FUN14
SP07
MDM10
CYS3
322 41
219
194
DEP1
NTG1
Attr base
347 66
244

21. Bicluster Structures

22. Cheng and Church Example:
Correlation between any two columns = correlation between any two rows = 1.
aij = aiJ + aIj – aIJ, where aiJ = mean of row i, aIj = mean of column j, aIJ = mean of A.
Biological meaning: the genes have the same (amount of) response to the conditions.
Back.: 5
Col 0: 1
Col 1: 3
Col 2: 2
Row 0: 2 8 10 9
Row 1: 4 12 11
Row 2: 1 7

23. bi-cluster Perfect -cluster Imperfect -cluster Residue: dij diJ dIJ

24. Cheng and Church Model:
A bicluster is represented the submatrix A of the whole expression matrix (the involved rows and columns need not be contiguous in the original matrix).
Each entry Aij in the bicluster is the superposition (summation) of:
The background level
The row (gene) effect
The column (condition) effect
A dataset contains a number of biclusters, which are not necessarily disjoint.

25. Cheng and Church Finding the largest -bicluster:
The problem of finding the largest square -bicluster (|I| = |J|) is NP-hard.
Objective function for heuristic methods (to minimize): => sum of the components from each row and column, which suggests simple greedy algorithms to evaluate each row and column independently.

26. Cheng and Church Greedy methods:
Algorithm 0: Brute-force deletion (skipped)
Algorithm 1: Single node deletion
Parameter(s):  (maximum squared residue).
Initialization: the bicluster contains all rows and columns.
Iteration:
Compute all aIj, aiJ, aIJ and H(I, J) for reuse.
Remove a row or column that gives the maximum decrease of H.
Termination: when no action will decrease H or H <= .
Time complexity: O(MN)

27. Cheng and Church Greedy methods:
Algorithm 2: Multiple node deletion (take one more parameter . In iteration step 2, delete all rows and columns with row/column residue > H(I, J)).
Algorithm 3: Node addition (allow both additions and deletions of rows/columns).

28. Cheng and Church
Handling missing values and masking discovered biclusters: replace by random numbers so that no recognizable structures will be introduced.
Data preprocessing:
Yeast: x  100log(105x)
Lymphoma: x  100x (original data is already log-transformed)

29. Cheng and Church
Some results on yeast cell cycle data (288417):

30. Cheng and Church Some results on lymphoma data (402696):
No. of genes, no. of conditions
4, 96
10, 29
11, 25
103, 25
127, 13
13, 21
10, 57
2, 96
25, 12
9, 51
3, 96

31. Cheng and Church Discussion:
Biological validation: comparing with the clusters in previously published results.
No evaluation of the statistical significance of the clusters.
Both the model and the algorithm are not tailored for discovering multiple non-disjoint clusters.
Normalization is of utmost importance for the model, but this issue is not well-discussed.

32. The FLOC algorithm Generating initial clusters
Determine the best action for
each row and each column
Perform the best action of each
row and column sequentially Y
Improved? N

33. The FLOC algorithm
Action: the change of membership of a row(or column) with respect to a cluster
column
M=4 1 2 3 4
row 3 4 2 2 1
M+N actions are
Performed at
each iteration 2 1 3 2 3
N=3 3 4 2 4

34. Performance Microarray data: 2884 genes, 17 conditions
100 bi-clusters with smallest residue were returned.
Average residue = 10.34
The average residue of clusters found via the state of the art method in computational biology field is 12.54
The average volume is 25% bigger
The response time is an order of magnitude faster

35. Coherent Cluster
Want to accommodate noises but not outliers

36. Coherent Cluster Coherent cluster pair-wise disparity
Subspace clustering
pair-wise disparity
For a 22 (sub)matrix consisting of objects {x, y} and attributes {a, b} x y a b z x y a b
dxa
dya
dxb
dyb
mutual bias
of attribute a
mutual bias
of attribute b
attribute

37. Coherent Cluster
A 22 (sub)matrix is a -coherent cluster if its D value is less than or equal to .
An mn matrix X is a -coherent cluster if every 22 submatrix of X is -coherent cluster.
A -coherent cluster is a maximum -coherent cluster if it is not a submatrix of any other -coherent cluster.
Objective: given a data matrix and a threshold , find all maximum -coherent clusters.

38. Coherent Cluster Challenges:
Finding subspace clustering based on distance itself is already a difficult task due to the curse of dimensionality.
The (sub)set of objects and the (sub)set of attributes that form a cluster are unknown in advance and may not be adjacent to each other in the data matrix.
The actual values of the objects in a coherent cluster may be far apart from each other.
Each object or attribute in a coherent cluster may bear some relative bias (that are unknown in advance) and such bias may be local to the coherent cluster.

39. References
J. Young, W. Wang, H. Wang, P. Yu, Delta-cluster: capturing subspace correlation in a large data set, Proceedings of the 18th IEEE International Conference on Data Engineering (ICDE), pp , 2002.
H. Wang, W. Wang, J. Young, P. Yu, Clustering by pattern similarity in large data sets, to appear in Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD), 2002.
Y. Sungroh,  C. Nardini, L. Benini, G. De Micheli, Enhanced pClustering and its applications to gene expression data Bioinformatics and Bioengineering, 2004.
J. Liu and W. Wang, OP-Cluster: clustering by tendency in high dimensional space, ICDM’03.