1. Segmentation and Clustering

2. Déjà Vu? Q: Haven’t we already seen this with snakes?Q: Haven’t we already seen this with snakes? A: no, that was about boundaries, this is about interiorsA: no, that was about boundaries, this is about interiors – Want regions of similar contents

3. Segmentation via Clustering Segmentation: Divide image into regions of similar contentsSegmentation: Divide image into regions of similar contents Clustering: Aggregate pixels into regions of similar contents

4. But Wait! We speak of segmenting foreground from backgroundWe speak of segmenting foreground from background Segmenting out skin colorsSegmenting out skin colors Segmenting out the moving personSegmenting out the moving person How do these relate to “similar regions”?How do these relate to “similar regions”?

5. Segmentation and Clustering Defining regionsDefining regions – Should they be compact? Smooth boundary? Defining similarityDefining similarity – Color, texture, motion, … Defining similarity of regionsDefining similarity of regions – Minimum distance, mean, maximum

6. Segmentation and Clustering Applications Semantics “Intelligent scissors” Finding skin-colored regions Foreground / background segmentation Finding the moving objects Finding the cars in a video sequence

7. Themes Energy minimizationEnergy minimization – Snakes: discretize, greedy minimization – Snakes, shape from shading: differential eqn – Stereo with smoothness: min-cost graph cuts – Today: k-means clustering StatisticsStatistics TemplatesTemplates Hot topic in vision research: combining these Hot topic in vision research: combining these

8. Segmentation and Clustering Applications “Intelligent scissors” Finding skin-colored regions Foreground / background segmentation Finding the moving objects Finding the cars in a video sequence StatisticsTemplates

9. Clustering Based on Color Let’s make a few concrete choices:Let’s make a few concrete choices: – Arbitrary regions – Similarity based on color only – Similarity of regions = distance between mean colors

10. Simple Agglomerative Clustering Start with each pixel in its own clusterStart with each pixel in its own cluster Iterate:Iterate: – Find pair of clusters with smallest inter-cluster distance – Merge Stopping thresholdStopping threshold

11. Simple Divisive Clustering Start with whole image in one clusterStart with whole image in one cluster Iterate:Iterate: – Find cluster with largest intra-cluster variation – Split into two pieces that yield largest inter- cluster distance Stopping thresholdStopping threshold

12. Difficulties with Simple Clustering Many possibilities at each iterationMany possibilities at each iteration Computing distance between clusters or optimal split expensiveComputing distance between clusters or optimal split expensive Heuristics to speed this up:Heuristics to speed this up: – For agglomerative clustering, approximate each cluster by average for distance computations – For divisive clustering, use summary (histogram) of a region to compute split

13. k-means Clustering 1.Pick number of clusters k 2.Randomly scatter k “cluster centers” in color space 3.Repeat: a.Assign each data point to its closest cluster center b.Move each cluster center to the mean of the points assigned to it Instead of merging or splitting, start out with the clusters and move them aroundInstead of merging or splitting, start out with the clusters and move them around

14. k-means Clustering

22. This process always converges (to something)This process always converges (to something) – Not necessarily globally-best assignment Informal proof: look at energy minimizationInformal proof: look at energy minimization – Reclassifying points reduces (or maintains) energy – Recomputing centers reduces (or maintains) energy – Can’t reduce energy forever

23. Results of Clustering Original Image k-means, k=5 k-means, k=11

24. Results of Clustering Sample clusters with k-means clustering based on color

25. Other Distance Measures Suppose we want to have compact regionsSuppose we want to have compact regions New feature space: 5D (2 spatial coordinates, 3 color components)New feature space: 5D (2 spatial coordinates, 3 color components) Points close in this space are close both in color and in actual proximityPoints close in this space are close both in color and in actual proximity

26. Results of Clustering Sample clusters with k-means clustering based on color and distance

27. Other Distance Measures Problem with simple Euclidean distance: what if coordinates range from 0-1000 but colors only range from 0-255?Problem with simple Euclidean distance: what if coordinates range from 0-1000 but colors only range from 0-255? – Depending on how things are scaled, gives different weight to different kinds of data Weighted Euclidean distance: adjust weights to emphasize different dimensionsWeighted Euclidean distance: adjust weights to emphasize different dimensions

28. Mahalanobis Distance Automatically assign weights based on actual variation in the data where C is covariance matrix of all pointsAutomatically assign weights based on actual variation in the data where C is covariance matrix of all points Gives each dimension “equal” weightGives each dimension “equal” weight Also accounts for correlations between different dimensionsAlso accounts for correlations between different dimensions