Vladyslav Kolbasin Stable Clustering

Clustering data Clustering is part of exploratory process Standard definition:  Clustering - grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups There is no “true” solution for clustering  We don't have any “true Y values”  Usually we want to do some data exploration or simplification or even find some data taxonomy  Usually we don't have precise mathematical definition of clustering Usually we iterate through different methods that have different mathematical target function, then use some best method 2

Usual clustering methods Methods:  K-means  Hierarchical clustering  Spectral clustering  DBScan  BIRCH  … Issues:  Need to estimate clusters count  Non-determinism  Non-stability 3

Are standard methods stable? Kmeans 4

Are standard methods stable? Hierarchical clustering 5

Audience data A lot of attributes: All attributes are binary There are several data providers There is no very important attributes 6

Stability importance Data comes from different providers and it is very noisy It is unlikely that results will change from run to run  Usually audience doesn't change a lot in short period Many algorithms “explode” when we increase data size  Non linear complexity of clustering  Best count of clusters move to higher values for bigger data 7

Let's add some additional requirement to clustering:  clustering result should be a structure on the data set that is “stable” So there should be similar results when:  We change some small portion of data  We apply clustering onto several datasets from the same underlying model  Apply clustering onto several subsets of initial dataset We don't want to process gigabytes and terabytes to get several stable clusters which are independent of randomness in sampling 8 Stable clustering. Requirements

Natural restrictions:  We don't want to have too many clusters  We don't want to have too small or too big clusters Too small clusters are usually useless for further processing Too big clusters do not bring significantly new information  Some points can be noise points, so let try to find only significant tendencies It will be big benefit if we can easily scale results  To be able to look at inner structure of selected cluster without full rerun Any additional instruments for manual analysis of clustering are welcome 9

Stable clustering ideas Do not use whole dataset, but use many small sub samples Use several samplings to mine as much as possible information from data Average all clustering on samples to get stable result 10

Stable clustering algorithm 1)Select N samples of whole dataset 2)Do clustering for each sample  So for each sample we have set of clusters (possibly very different) 3)Do some clusters' preprocessing 4)Associate clusters from different samples to each other  Build some relationship structure - clusters graph  Set relationship measure - distance measure 5)Do clustering on relationship structure  Do communities search 11

2. Sample clustering Any clustering method:  Kmeans  Hierarchical clustering  … It is conveniently to use hierarchical clustering:  It is rather fast clustering method  We can estimate clusters count using natural restrictions, not using special criteria like we usually do for kmeans  We can deep into internal structure without any additional calculations 12

2.1. Dendrogram clustering Recursive splitting of large clusters With natural restrictions:  Set max possible cluster size (in %)  Set min cluster size (in %), any smaller cluster – noise  Max count of splits  … 13

2.1. Dendrogram clustering 14

3. Do clusters' preprocessing Reduce noise points Cluster smoothing Make clusters more convenient for associating:  Cluster can be similar to several other clusters (1-to-many)  If split it, it can transform into: 1-to-1 & 1-to-1 clusters And some other heuristics... 15

4. Associate clusters from different samples to each other How similar to each other are clusters? Set relationship measure:  Simplest measure - distance between cluster's centers  But we can use any suitable measure 16

4. Associate clusters from different samples to each other Clusters relationship structure - clusters graph  But we are not interested in edges for very different clusters So need some threshold:  Can estimate manually, then hard-code  Can estimate automatically 17

5. Communities search in networks Methods:  walktrap.community  edge.betweenness.community  fastgreedy.community  spinglass.community  … It is possible that some clusters will not be in any community. Then will mark these clusters as special type community 18

5.1 Community structure detection based on edge betweenness edge.betweenness.community() implements Girvan–Newman algorithm Betwenness - the number of geodesics (shortest paths) going through an edge Algorithm:  Calculate edge-betweenness for all edges  Remove the edge with highest betweenness  Recalculate betweenness  Repeat until all edges are removed, or modularity function is optimized (depending on variation) 19

5. Communities examples 20

Algorithm analysis 21

Algorithm analysis 22

Algorithm analysis 23

Summary Issues in clustering algorithms Why stability is important for business questions? 2 staged clustering algorithm  1 st stage – apply simple clustering on samples  2 nd stage – do clustering on cluster graph Real data clustering example Algorithm can be simply parallelized:  Most time is spent on 2 nd step 24