1. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 1 Segmentation and Labeling Outline: Edge detection Chain coding of curves Segmentation into Regions Connected components labeling using the UNION-FIND abstract data type Blocks world scene analysis using Guzman’s labeling method

2. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 2 Edge Detection Rationale: In order to find out where the objects are and what their properties are, locate their boundaries by finding “edges”. An edge is a transition from a region of one predominant color or texture to another predominant color or texture. Approaches: 1. local difference operators. E.g., Sobel, Roberts. * 2. model fitting. E.g., Hueckel, Canny. 3. vector space transformation. E.g., Frei & Chen. 3. contour finding via search. 4. snakes.

3. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 3 Edge Detection (cont.) Local Difference Operators (Oriented) Horizontal differences: E h [i, j] = F[i, j] – F[i, j-1] Vertical differences: E v [i, j] = F[i, j] – F[i-1, j]

4. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 4 Edge Detection (cont.) Combining directions Strength[i,j] = sqrt( E h 2 + E v 2 ) Direction[i,j] = arctan(E v / E h ) (note: the latter has a bug when E h = 0. This can be fixed using alternative cases.)

5. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 5 Edge Detection (cont.) Local Difference Operators (Omnidirectional) Roberts Cross operator: E[i, j] = sqrt( (F[i, j] – F[i+1, j+1]) 2 + (F[i, j+1] – F[i+1, j]) 2 ) Sobel operator: E[i, j] = sqrt( [(c+2f+i)-(a+2d+g)] 2 + [(g+2h+i)-(a+2b+c)] 2 ) a b c b e f c h i

6. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 6 Chain Coding of Contours A chain code is a system for representing elementary vectors by small integers.  = 0  = 1  = 2  = 3 To represent a curve, first approximate it as a sequence of elementary vectors, and then write down the code number for each vector in a sequence.

7. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 7 Segmentation into Regions Technique 1: Thresholding Technique 2: Region growing Example: Connected-components labeling with the UNION-FIND abstract data type.

8. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 8 Thresholding Assumption: We have a monochrome image each of whose pixels belong either to the figure(s) or to the background. Perform a thresholding transformation: T[i,j] = 0 if F[i,j] <  1 otherwise The value  is called a threshold.

9. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 9 Threshold Selection Choosing a suitable can be challenging. If is too low, many pixels may be misclassified as figure, whereas, if it is too high, many pixels may be misclassified as background. The valley method: If the histogram of the input image is bimodal, choose as the threshold, the value between the modes where the histogram is lowest.

10. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 10 Valley Method Example 2 3 5 1 1 7 6 2 5 5 6 1 1 5 1 2 X X X X X X X X X 1 2 3 4 5 6 7 Modes at 1 and 5. Valley at  = 4.

11. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 11 Valley Method Example Result of thresholding with = 4 0 0 1 0 0 1 1 0 1 1 1 0 0 1 0 0

12. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 12 Segmentation into Regions: A segmentation into regions means dividing up all the pixels of an image into groups. Each group of pixels must be connected to each other rather than separated in different parts of the image. Each group must have the same brightness, color, or other local property. It should not be possible to merge two groups and have the new group be both connected and uniform in property.

13. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 13 Segmentation into Regions: Formal Definitions “Adjacency” a b c d e f g h i Pixel e is 4-adjacent to each of b, d, f, and h. a b c d e f g h i Pixel e is 8-adjacent to each of a, b, c, d, f, g, h, i.

14. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 14 Segmentation into Regions: Connectivity Let I = {p 0, p 1, …, p N-1 } be an image. Let R be a nonempty subset of I. Suppose R has at least two pixels. Let p s1 and p sn be two pixels in R. Pixels p s1 and p sn are 4-connected in R provided there is a sequence of pixels (p s1, p s2, … p sn ) such that each successive pair of pixels is 4-adjacent, and all these pixels are in R.

15. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 15 Segmentation into Regions: Definition of Region A region R of I is a 4-connected nonempty subset of I, such that every pair of pixels in R is 4-connected in R.

16. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 16 Region Adjacency Let R i and R j be disjoint regions. If p i is a pixel in R i and p j is a pixel in R j, and p i is 4-adjacent to p j, then regions R i and R j are adjacent to one another. Then we write R i adj R j.

17. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 17 Segmentation Definition Let I = {p 0, p 1, …, p N-1 } be an image. Let S = {R 1, R 2, …, R k } be a set of regions of I. Let P be a uniformity predicate where P(R) is true provided that all the pixels in R have a uniform appearance. For example, P(R) could be defined to be true whenever all pixels in R have the same value. S is a segmentation of I provided that: (1) The regions cover I. That is, R 1  R 2  …  R k = I. (2) The regions are disjoint. i  j  R i  R j = . (3) Each region is uniform.  j, P(R j ). (4) Segments are maximal. If R i adj R j then  P(R i  R j ). This means no regions can be merged and still maintain uniformity within the resulting region.

18. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 18 Computing a Segmentation with the UNION-FIND ADT Initialize a set of regions by putting each pixel in its own region. Scan the image in raster-scan order. At each pixel p i, check its neighbor p j to the west (if it has one) as follows: If the two pixels have the same pixel value, do a FIND on each one getting c i and c j. If c i  c j, then perform UNION(c i, c j ). Then check its neighbor p q to the north (if it has one) in the same way. Perform a second scan of the image and at each pixel p, replace the pixel value by FIND(p)

19. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 19 Vision: Scene Labeling Structure in the world allows us to interpret complex situations by propagating constraints. This is conveniently demonstrated in computer vision, using a technique known as Guzman labeling.

20. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 20 Blocks-World Vision Problem Given: A 2-D line drawing representing a collection of polyhedral blocks on a table. Determine: Which faces (bounded regions in the drawing) go together as parts of the same objects. Which edges (line segments) are internal to objects and which are “occluding” edges.

21. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 21 Example Scene Analysis Problem

22. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 22 Guzman’s Vertex Labels EllArrowFork Tee Arrow: There is an angle of more than 180 deg. Fork: Each of the 3 angles is of less than 180 deg. Tee: There is one angle of exactly 180 deg., and 2 smaller ones.

23. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 23 Guzman’s Labelling Technique 1. Classify each vertex into one of the categories: ell, arrow, fork, tee, or other. 2. At each vertex, mark each incoming edge according to the role it plays at the vertex (e.g., shank of an arrow, stem of a tee). 3. Create single links between neighboring regions each time: a. they are divided by a line segment at a fork, b. they are divided by the shank of an arrow, c. they are in corresponding positions of a configuration of two opposing tees with colinear stems. 4. Create a “node” for each region. Whenever two regions are doubly linked, connect their nodes with a “same object” link.

24. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 24 Example with Vertex Labels A A A A A F F A L L L L T T T L L T T

25. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 25 Example with Region Links

26. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 26 Example with Object Links

27. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 27 Example with Internal and Occluding Edges Identified occluding internal

28. CSE 415 -- (c) S. Tanimoto, 2007 Segmentation and Labeling 28 Discussion The work of Guzman was refined by D. Huffman, M. Clowes, and D. Waltz to take into account additional kinds of edges, including crack edges, oriented occluding edges, and concave and convex internal edges. A key point of this research was to show that constraints imposed by the real world could make seemingly intractable combinatorial interpretation problems actually solvable.