1. CS 376b Introduction to Computer Vision 04 / 08 / 2008 Instructor: Michael Eckmann

2. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Today’s Topics Comments/Questions Segmentation algorithms –region growing single linkage centroid linkage –clustering using graph algorithms –representing regions

3. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation Haralick and Shapiro (1985)‏ –considered a region to be a set of connected pixels with same mean and variance –mean and scatter of a region is defined –when a new pixel is added into a region, the new mean and scatter are given

4. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation Haralick and Shapiro (1985) region growing algorithm –a T statistic is also defined which is used to determine if a test pixel is part of the same distribution of a region (should be added into the region) up to some confidence level –if T is too high, y (the test pixel) is likely to not be part of the same population of pixels in the region –if T is too high for y and all neighboring regions of y, then y starts as the first pixel in a new region –besides being close to the mean of a region's distribution (the T test above), could also require that a neighboring pixel have a close enough value to y (which is the main criteria in single-linkage region growers)‏

5. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation Haralick and Shapiro (1985) region growing algorithm –T is considered for some fraction called alpha (e.g. 1/20)‏ this alpha is the probability that a T with N-1 d.o.f. exceeds T N-1 (alpha)‏ –T N-1 (alpha) for a given N-1 and alpha, is a defined quantity if our computed T does exceed this T N-1 (alpha) then it is determined that the pixel is significantly different than the population of the region if the pixel really did belong to the same population as the region, then the probability that the test we did was incorrect is alpha (e.g. 1/20)

6. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation Haralick and Shapiro (1985) region growing algorithm –the d.o.f. N-1 T N-1 (alpha) is higher for smaller N-1 and lower for larger N-1 the d.o.f. is determined from the number of pixels in the region --- the more pixels already determined as part of the population, the larger the d.o.f. (and vice-versa)‏ what does this mean for us? –the larger number of pixels a region has, the closer a pixel's value has to be to that mean to join the region (and the smaller a region is...)‏

7. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation Other researchers took a graph theoretic approach to segmentation. –Let G=(V,E) where V is the set of points in the measurement space and E is the set of weighted edges where the weight represents the similarity between the two vertices –Goal partition the graph into disjoint subsets of V so that similarity within sets is high and similarity between sets is low To partition a graph into two disjoint graphs G a =(A,E a ) and G b =(B,E b ) whereV = A union B we remove any edges from E that connect v's in A with v's in B

8. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation To partition a graph into two disjoint graphs G a =(A,E a ) and G b =(B,E b ) whereV = A union B we remove any edges from E that connect v's in A with v's in B –the degree of dissimilarity between the two disjoint sets A and B can be computed as the sum of the weights of the edges removed --- this sum of the weights of the edges removed is called a cut so, cut(A,B) = Sum of weights of edges connecting v's in A with v's in B

9. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation Wu & Leahy (1993) used minimum cut to segment regions –does anyone know any problems with minimum cut?

10. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation Wu & Leahy (1993) used minimum cut to segment regions –problem with minimum cut is it tends to prefer cutting off small regions which isn't great for image segmentation (example from paper)‏ Others came up with other formulations on what to minimize. What is really wanted is –each segment should have v's that are highly similar (have large weights) and between segments should be highly dissimilar (have low weights) without a bias towards small regions being cut off

11. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation Shi and Malik –developed a new graph partitioning method –problem with minimum cut --- favors small sets of isolated vertices --- not useful for segmenting into large uniform (in some measurement) regions –proposed normalized cut (Ncut) which is: Ncut(A,B) = cut(A,B)/assoc(A,V) + cut(A,B)/assoc(B,V)‏ where assoc(A,V) = sum of all weights of edges from vertices in A to vertices in V (V is all the vertices in G)‏

12. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation Shi and Malik Ncut(A,B) is a measure of cut cost as a fraction of the total edge connections to all the nodes in the graph the cut that partitions out small isolated points will not have a small Ncut example: consider a cut that isolates 1 vertex what is Ncut's value? consider a case where a cut would divide the graph into two large groups by cutting only one small weight edge vs. a cut in the same graph that would cut a different edge of the same small weight but would divide the graph into two groups with: 1 vertex and N-1 vertices

13. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation A segmentation algorithm based on the Ncut would do something like this: –set up a weighted graph with v's the pixels and edges the weigths between the v's. weight of edge e ij should represent the similarity of pixel i to pixel j based on color or texture or intensity (and possibly in addition to distance within the image)‏ –find the minimum Ncut for the whole graph and partition the pixels into two segments –recursively do the same thing but with each partition created above –do this until each partition should no longer be subdivided

14. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation If you wish to read more about the Shi, Malik scheme, see “Normalized Cuts and Image Segmentation”, Shi, Malik in IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) August 2000, volume 22, number 8, pages 888-905

15. Michael Eckmann - Skidmore College - CS 376b - Spring 2008 Segmentation All algorithms discussed so far generate segments which need to be represented in some fashion. The most common representation is probably a labelled image, where an image of the same size as the original is created with labels as pixel values. All pixels that are part of the same segment have the same label and different segments have different labels. Connected components algorithms typically produce something like this.

16. Segmentation Other possibilities include –overlays (line drawings over the top of the original image to show the segments)‏ –boundary coding linear list of the border pixels chain code –store coords of one border pixel and then a list of “directions” quadtrees –multiresolution scheme which encodes the whole region not just the border property tables –each row is a region and each column is some property of interest

17. Segmentation chain code (part of figure 10.12 in Shapiro and Stockman)‏ Michael Eckmann - Skidmore College - CS 376b - Spring 2008

18. Segmentation quadtrees (figure 10.13 in Shapiro and Stockman)‏ Michael Eckmann - Skidmore College - CS 376b - Spring 2008

19. Segmentation property tables –each row is a region and each column is some property of interest area, main colors, texture measures, and even the chain code or quadtree representation of the region Exercise 10.4 Consider an image region represented by a) a labelled image and b) a chain code. Create 4 algorithms and give their running times: –compute the area of region using labelled image –compute the perimeter of region using labelled image –compute the area of region using the chain code –compute the perimeter of region using the chain code

20. Segmentation Boundary extraction –a process by which we can determine and encode the boundaries of all the segments or connected components in an image –a simple procedure is as follows: given a labelled image and assume background has some predetermined label make a list of the first border pixel for each connected-component by scanning through the whole image once for each first border pixel, start there and follow the border in a clockwise fashion.