Linked Based Clustering and Its Theoretical Foundations Paper written by Margareta Ackerman and Shai Ben-David Yan T. Yang Presented by Yan T. Yang.

Slides:

Advertisements

Similar presentations

Different types of data e.g. Continuous data:height Categorical data ordered (nominal):growth rate very slow, slow, medium, fast, very fast not ordered:fruit.

Advertisements

SEEM Tutorial 4 – Clustering. 2 What is Cluster Analysis?  Finding groups of objects such that the objects in a group will be similar (or.

Hierarchical Clustering

Cluster Analysis: Basic Concepts and Algorithms

1 CSE 980: Data Mining Lecture 16: Hierarchical Clustering.

Hierarchical Clustering. Produces a set of nested clusters organized as a hierarchical tree Can be visualized as a dendrogram – A tree-like diagram that.

Hierarchical Clustering, DBSCAN The EM Algorithm

O(N 1.5 ) divide-and-conquer technique for Minimum Spanning Tree problem Step 1: Divide the graph into  N sub-graph by clustering. Step 2: Solve each.

Effects of Rooting on Phylogenic Algorithms Margareta Ackerman Joint work with David Loker and Dan Brown.

Data Mining Cluster Analysis: Basic Concepts and Algorithms

Characterization of Linkage-Based Algorithms Margareta Ackerman Joint work with Shai Ben-David and David Loker University of Waterloo To appear in COLT.

Introduction to Bioinformatics

Important clustering methods used in microarray data analysis Steve Horvath Human Genetics and Biostatistics UCLA.

IT 433 Data Warehousing and Data Mining Hierarchical Clustering Assist.Prof.Songül Albayrak Yıldız Technical University Computer Engineering Department.

Discerning Linkage-Based Algorithms Among Hierarchical Clustering Methods Margareta Ackerman and Shai Ben-David IJCAI 2011.

6-1 ©2006 Raj Jain Clustering Techniques  Goal: Partition into groups so the members of a group are as similar as possible and different.

UPGMA Algorithm.  Main idea: Group the taxa into clusters and repeatedly merge the closest two clusters until one cluster remains  Algorithm  Add a.

Clustering… in General In vector space, clusters are vectors found within  of a cluster vector, with different techniques for determining the cluster.

Topological Data Analysis MATH 800 Fall Topological Data Analysis (TDA) An ε-chain is a finite sequence of points x 1,..., x n such that |x i –

Cluster Analysis: Basic Concepts and Algorithms

Introduction to Hierarchical Clustering Analysis Pengyu Hong 09/16/2005.

Introduction to Bioinformatics - Tutorial no. 12

© University of Minnesota Data Mining for the Discovery of Ocean Climate Indices 1 CSci 8980: Data Mining (Fall 2002) Vipin Kumar Army High Performance.

Ulf Schmitz, Pattern recognition - Clustering1 Bioinformatics Pattern recognition - Clustering Ulf Schmitz

Chapter 3: Cluster Analysis  3.1 Basic Concepts of Clustering  3.2 Partitioning Methods  3.3 Hierarchical Methods The Principle Agglomerative.

Clustering Unsupervised learning Generating “classes”

Image Segmentation Seminar III Xiaofeng Fan. Today ’ s Presentation Problem Definition Problem Definition Approach Approach Segmentation Methods Segmentation.

Hierarchical Clustering

Towards Theoretical Foundations of Clustering Margareta Ackerman Caltech Joint work with Shai Ben-David and David Loker.

A Clustering Algorithm based on Graph Connectivity Balakrishna Thiagarajan Computer Science and Engineering State University of New York at Buffalo.

Clustering Algorithms k-means Hierarchic Agglomerative Clustering (HAC) …. BIRCH Association Rule Hypergraph Partitioning (ARHP) Categorical clustering.

1. Clustering is one of the most widely used tools for exploratory data analysis. Social Sciences Biology Astronomy Computer Science …. All apply clustering.

Chapter 14 – Cluster Analysis © Galit Shmueli and Peter Bruce 2010 Data Mining for Business Intelligence Shmueli, Patel & Bruce.

CSE5334 DATA MINING CSE4334/5334 Data Mining, Fall 2014 Department of Computer Science and Engineering, University of Texas at Arlington Chengkai Li (Slides.

Prepared by: Mahmoud Rafeek Al-Farra

K-Means Algorithm Each cluster is represented by the mean value of the objects in the cluster Input: set of objects (n), no of clusters (k) Output:

Theoretical Foundations of Clustering – CS497 Talk Shai Ben-David February 2007.

Formal Foundations of Clustering Margareta Ackerman Work with Shai Ben-David, Simina Branzei, and David Loker.

Reza Bosagh Zadeh (Carnegie Mellon) Shai Ben-David (Waterloo) UAI 09, Montréal, June 2009 A UNIQUENESS THEOREM FOR CLUSTERING.

Hierarchical Clustering Produces a set of nested clusters organized as a hierarchical tree Can be visualized as a dendrogram – A tree like diagram that.

Clustering Algorithm CS 157B JIA HUANG. Definition Data clustering is a method in which we make cluster of objects that are somehow similar in characteristics.

Information Retrieval and Organisation Chapter 17 Hierarchical Clustering Dell Zhang Birkbeck, University of London.

Unsupervised Learning

Hierarchical Clustering

What is a metric embedding?Embedding ultrametrics into R d An embedding of an input metric space into a host metric space is a mapping that sends each.

Machine Learning Lunch - 29 Sep 2009 – ClusteringTheory.org John Oliver from “The Daily Show” Supporting worthy causes at the G20 Pittsburgh Summit: “Bayesians.

Data Mining Cluster Analysis: Basic Concepts and Algorithms Lecture Notes Introduction to Data Mining by Tan, Steinbach, Kumar © Tan,Steinbach, Kumar Introduction.

Hierarchical Clustering

Clustering Patrice Koehl Department of Biological Sciences

CSE 4705 Artificial Intelligence

Chapter 15 – Cluster Analysis

Hierarchical Clustering

K-means and Hierarchical Clustering

John Nicholas Owen Sarah Smith

CSE572, CBS598: Data Mining by H. Liu

Clustering and Multidimensional Scaling

Cluster Analysis in Bioinformatics

Data Mining – Chapter 4 Cluster Analysis Part 2

Hierarchical Clustering

Birch presented by : Bahare hajihashemi Atefeh Rahimi

Cluster Analysis.

Hierarchical Clustering

SEEM4630 Tutorial 3 – Clustering.

CSE572: Data Mining by H. Liu

Hierarchical Clustering

Presentation transcript:

Linked Based Clustering and Its Theoretical Foundations Paper written by Margareta Ackerman and Shai Ben-David Yan T. Yang Presented by Yan T. Yang

Outline Hierarchical clustering Linkage based clustering Theoretical foundation of clustering Characterization of linkage based clustering

Outline Hierarchical clustering Linkage based clustering Theoretical foundation of clustering Characterization of linkage based clustering

Hierarchical clustering Agglomerative Divisive

Hierarchical clustering A D E B C ABCDE Raw Data Dendrogram F F

Agglomerative clustering Raw Data Dendrogram ABCDEF A D E B C F

Agglomerative clustering Raw Data Dendrogram ABCDEF A D E B C F

Agglomerative clustering A D E B C Raw Data Dendrogram F ABCDEF

Agglomerative clustering A D E B C Raw Data Dendrogram F ABCDEF

Agglomerative clustering Raw Data Dendrogram ABCDEF A D E B C F

Agglomerative clustering Raw Data Dendrogram ABCDEF A D E B C F General Complexity: O(n 3 )

Agglomerative clustering Raw Data Dendrogram General Complexity: O(n 3 ) Linkage Based: O(n 2 )

Linkage Based Clustering General Complexity: O(n 3 ) Linkage Based: O(n 2 ) Whenever a pair of points share the same cluster, they are connected via a tight chain of points. A D E B C F

Linkage Based Clustering General Complexity: O(n 3 ) Linkage Based: O(n 2 ) Whenever a pair of points share the same cluster, they are connected via a tight chain of points. A D E B C F 3. Repeat until only k clusters remain. Steps: 1. Start: singletons 2. At each step, merge the closest pair of clusters

Linkage-based clustering Linkage Based: O(n 2 ) Dissimilarity MeasureLinkage Criteria A measure of distance between pair of observations. A measure of distance between sets of observations as function of dissimilarity measure.

Linkage-based clustering Linkage Based: O(n 2 ) Euclidean distance Minkowski distance (generalized Euclidean) Mahalahobis distance (scaled Euclidean) City block distance ( |*| version of Euclidean) Dissimilarity MeasureLinkage Criteria

Linkage-based clustering Linkage Based: O(n 2 ) Dissimilarity MeasureLinkage Criteria Between two sets A and B

Linkage-based clustering Linkage Based: O(n 2 ) Linkage Criteria Between two sets A and B Minimum Spanning Tree Successive Cliques UPGMA tree

Linkage-based clustering Linkage Based: O(n 2 ) Linkage Criteria Between two sets A and B SLINK [Defays] CLINK [Sibson] UPGMA [Murtagh]

Linkage-based clustering Add to cluster as long as there is a link

Linkage-based clustering Add to cluster as long as there is a link

Linkage-based clustering Add to cluster as long as there is a link

Linkage-based clustering Add to cluster as long as there is a link

Linkage-based clustering Add to cluster as long as there is a link

Linkage-based clustering Add to cluster as long as there is a link

Linkage-based clustering Add to cluster when a clique is formed

Linkage-based clustering Add to cluster when a clique is formed

Linkage-based clustering Add to cluster when a clique is formed

Linkage-based clustering Add to cluster when a clique is formed

Linkage-based clustering Add to cluster when a clique is formed

Linkage-based clustering Add to cluster when a clique is formed

Linkage-based clustering Add to cluster when a clique is formed

Linkage-based clustering Add to cluster when a clique is formed

Linkage-based clustering Add to cluster when a clique is formed

Theoretical Foundation Many different clustering techniques: which one to use ? Identify properties that separate input-output behavior of different clustering paradigms: [Ackerman, Ben-David, Loker] 1) Be intuitive and meaningful 2) Differentiate algorithms The properties should

Theoretical Foundation [Kleinberg, 2002]: abstract properties OR axioms. [Zadeh, Ben-David 2009]: properties to characterize single linkage. [Ackerman, Ben-David, Loker] : properties to characterize all linkage.

Theoretical Foundation Properties Extended richness Outer consistency Locality Definition Refinement Hierarchical Clustering function Hierarchical

Theoretical Foundation Properties Extended richness Outer consistency Locality Definition A clustering C is a refinement of clustering C’ if every cluster in C’ is a union of some clusters in C. C C’ Refinement Hierarchical Clustering function Hierarchical

Theoretical Foundation Properties Extended richness Outer consistency Locality Definition Refinement Hierarchical A clustering function F(X,d,k): Clustering function Hierarchical 1. Take input X, all data points 2. Distance measure, d 3. Number of clusters desired, k. 4. Output the clusters

Theoretical Foundation Properties Extended richness Outer consistency Locality Definition Refinement Hierarchical A clustering function is hierarchical if for all X and all d and every 1<= k <= k’ <= |X| F(X,d,k’) is a refinement of F(X,d,k) Clustering function ABCDEF Hierarchical

Theoretical Foundation Properties Extended richness Outer consistency Locality Definition Refinement Hierarchical Clustering function C Hierarchical

Theoretical Foundation Properties Extended richness Outer consistency Locality Definition Refinement Hierarchical Clustering function dd’ Hierarchical

Theoretical Foundation Properties Extended richness Outer consistency Locality Definition Refinement Hierarchical Clustering function Locality and outer consistency Not axioms Spectral clustering with Ratio region Normalized cut does not satisfy these two. Hierarchical

Theoretical Foundation Properties Extended richness Outer consistency Locality Definition Refinement Hierarchical Clustering function Hierarchical

Theoretical Foundation Properties Extended richness Outer consistency Locality Theorem: The clustering function is linkage based If and only if All of the properties are satisfied. Hierarchical

Theoretical Foundation Theorem: The clustering function is linkage based If and only if All of the properties are satisfied. properties are satisfied. Properties Extended richness Outer consistency Locality Hierarchical linkage based function

Theoretical Foundation Theorem: The clustering function is linkage based If and only if All of the properties are satisfied. linkage based functionproperties are satisfied. Straightforward Properties Extended richness Outer consistency Locality Hierarchical

Theoretical Foundation Theorem: The clustering function is linkage based If and only if All of the properties are satisfied. properties are satisfied. Properties Extended richness Outer consistency Locality Hierarchical linkage based function

Theoretical Foundation Properties Extended richness Outer consistency Locality Hierarchical linkage based function A linkage function is a function l : {(X 1,X 2,d) : d is a dissimilarity function over X 1 U X 2 } → R + that satisfies the following: - Representation independent - Monotonic - Any pair of clusters can be made arbitrarily distant

Theoretical Foundation Properties Extended richness Outer consistency Locality Hierarchical linkage based function Goal Given a clustering function F that satisfies the properties, define a linkage function l so that the linkage-based clustering based on l coincides with F (for every X, d and k). F Linkage-based

Theoretical Foundation Given a clustering function F that satisfies the properties, define a linkage function l so that the linkage-based clustering based on l coincides with F (for every X, d and k). F Linkage-based Define an operator < F : (A,B,d 1 ) < F (C,D,d 2 ) if there exists d that extends d 1 and d 2 such that when we run F on (A U B U C U D), A and B are merged before C and D. A B C D F(AUBUCUD,d,4) A B C D F(AUBUCUD,d,3)

Theoretical Foundation Given a clustering function F that satisfies the properties, define a linkage function l so that the linkage-based clustering based on l coincides with F (for every X, d and k). F Linkage-based Define an operator < F : (A,B,d 1 ) < F (C,D,d 2 ) if there exists d that extends d 1 and d 2 such that when we run F on (A U B U C U D), A and B are merged before C and D. A B C D F(AUBUCUD,d,4) A B C D F(AUBUCUD,d,3)

Theoretical Foundation - Prove that < F can be extended to a partial ordering - Use the ordering to define l Lemma (< F is cycle-free) Given a function F that is hierarchical, local, outer-consistent and satisfies extended richness, there are no (A 1,B 1,d 1 ), (A 2,B 2,d 2 ),… (A n,B n,d n ) so that (A 1,B 1,d 1 ) < F (A 2,B 2,d 2 ) … < F (A n,B n,d n ) and (A 1,B 1,d 1 ) = (A n,B n,d n ) By the above Lemma, the transitive closure of < F is a partial ordering. There exists an order preserving function l that maps pairs of data sets to R (since < F is defined over a countable set) It can be shown that l satisfies the properties of a linkage function.