1. The University of Ontario CS 4487/9587 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz

2. The University of Ontario CS 4487/9587 Algorithms for Image Analysis Segmentation (2D) n Blobs Need for blobs Extracting blobs –Thresholding, region growing, mean-shift –Minimum spanning trees n Object extraction Intelligent scissors (also known as live-wire) Snakes Region+boundary methods n Local, greedy, and global processing Extra Reading: Sonka at.al. Ch. 5 Gonzalez and Woods, Ch. 10

3. The University of Ontario HW assignment 1 n 2D image segmentation Live-wire (based on Dijkstra algorithm for shortest paths on a graph) Implement and experiment on many images Compare with local/greedy methods Some useful code libraries and sample images will be posted within a couple of days

4. The University of Ontario Goal: Extract “Blobs” n What are “blobs”? Regions of an image that are somehow coherent n Why? Object extraction, object removal, compositing, etc. …but are “blobs” objects? No, not in general

5. The University of Ontario Blob’s coherence n Simplest way to define blob coherence is as similarity in brightness or color: The tools become blobs The house, grass, and sky make different blobs

6. The University of Ontario Why is this useful? AIBO RoboSoccer (VelosoLab)

7. The University of Ontario Ideal Segmentation

8. The University of Ontario Result of Segmentation

9. The University of Ontario Thresholding n Basic segmentation operation: mask(x,y) = 1 if im(x,y) > T mask(x,y) = 0 if im(x,y) < T n T is threshold User-defined Or automatic n Same as histogram partitioning:

10. The University of Ontario Sometimes works well… What are potential Problems?

11. The University of Ontario …but more often not Adaptive thresholding

12. The University of Ontario Region growing n Is this same as global threshold? Start with initial set of pixels K (initial seed(s)) Add to pixels p in K their neighbors q if |Ip-Iq| < T Repeat until nothing changes

13. The University of Ontario Region growing

14. The University of Ontario What can go wrong with region growing ? Region growth may “leak” through a single week spot in the boundary

15. The University of Ontario Region growing See region leaks into sky due to a weak boundary between them

16. The University of Ontario Due to Pedro Felzenszwalb and Daniel Huttenlocher Motivating example This image has three perceptually distinct regions Difference along border between A and B is less then differences within C A CB Q: Where would image thresholding fail? Q: Where would region growing fail? A: Region A would be divided in two sets and region C will be split into a large number of arbitrary small subsets A: Either A and B are merged or region C is split into many small subsets Also, B and C are merged

17. The University of Ontario Color-Based Blob Segmentation n Automatic Histogram Partitioning Given image with N colors, choose K Each of the K colors defines a region –not necessarily contiguous Performed by computing color histogram, looking for modes

18. The University of Ontario Finding Modes in a Histogram n How Many Modes Are There? Easy to see, hard to compute

19. The University of Ontario Mean Shift [ Comaniciu & Meer] 1.Initialize random seed, and fixed window 2.Calculate center of gravity ‘x’ of the window (the“mean”) 3.Translate the search window to the mean 4.Repeat Step 2 until convergence Iterative Mode Search x o x x mode

20. The University of Ontario Mean-Shift

21. The University of Ontario Mean-shift results More ExamplesMore Examples: http://www.caip.rutgers.edu/~comanici/segm_images.html

22. The University of Ontario Issues: n Although often useful, all these approaches work only some of the time, and are considered rather “hacky”. n Can’t even handle our tiger: Problem is that blobs != objects!

23. The University of Ontario Extracting objects n How could this be done?

24. The University of Ontario Extracting objects n Many approaches proposed color cues region cues contour cues n We will consider a few of these n Today: Intelligent Scissors (contour-based) –E. N. Mortensen and W. A. Barrett, Intelligent Scissors for Image Composition, in ACM Computer Graphics (SIGGRAPH `95), pp. 191-198, 1995Intelligent Scissors for Image Composition

25. The University of Ontario Intelligent Scissors

26. The University of Ontario Intelligent Scissors n Approach answers a basic question Q: how to find a path from seed to mouse that follows object boundary as closely as possible? A: define a path that stays as close as possible to edges

27. The University of Ontario Intelligent Scissors n Basic Idea Define edge score for each pixel –edge pixels have low cost Find lowest cost path from seed to mouse seed mouse Questions How to define costs? How to find the path?

28. The University of Ontario Path Search (basic idea) n Graph Search Algorithm Computes minimum cost path from seed to all other pixels Note: diagonal “paths” are scaled by a factor. … Why? 3 sets of nodes Free Active Done

29. The University of Ontario Segmentation should be Invariant to Image Rotation After object rotation L L Path’s cost along the top boundaryPath’s cost along the top-left boundary After diagonal links are adjusted by 2 2 2 2 2 2 2 2 2 2 2 2 image gradient scores

30. The University of Ontario Instead of nodes, image gradient scores can be assigned directly to graph edges n Graph node for every pixel p link between every adjacent pair of pixels e=(p,q) cost w(e) for each link n Note: each link has a cost this is a little different than the figure before where each pixel (graph node) had a cost p q n Treat the image as a graph

31. The University of Ontario Defining edge costs n Treat the image as a graph n Want to hug image edges: how to define cost of a link? p q the link should follow the intensity edge –want intensity to change rapidly to the link the cost of edge should be small when the difference of intensity to the link is large T T

32. The University of Ontario Defining edge costs p q n First, estimate image derivative in the direction orthogonal to the edge n Use finite differences (or cross-correlation filters) 1 11

33. The University of Ontario Defining edge costs p q n Second, use some penalty function g( ) assigning low penalty to large directional derivatives and large penalty to small derivatives

34. The University of Ontario Defining edge costs p q n Finally, cost of en edge e should be local contrast score adjusted by the edge length Why?

35. The University of Ontario Defining edge costs n When computing the shortest path we approximate a contour C minimizing continuous geometrically meaningful functional Cost of one edge Cost of a PATH Integral of contrast penalty along the contour

36. The University of Ontario (see Cormen et.al. “Introduction to Algorithms”, p.595) Dijkstra’s shortest path algorithm 0 5 31 33 4 9 2 link cost 1.init node costs to , set p = seed point, cost(p) = 0 2.expand p as follows: for each of p’s neighbors q that are not expanded set cost(q) = min( cost(p) + c pq, cost(q) ) ALGORITHM 3 sets of nodes Free Active Done

37. The University of Ontario Dijkstra’s shortest path algorithm 4 10 5 3 323 9 5 31 33 4 9 2 11 1.init node costs to , set p = seed point, cost(p) = 0 2.expand p as follows: for each of p’s neighbors q that are not expanded set cost(q) = min( cost(p) + c pq, cost(q) ) –if q’s cost changed, make q point back to p put q on the ACTIVE list (if not already there) ALGORITHM 3 sets of nodes Free Active Done

38. The University of Ontario Dijkstra’s shortest path algorithm 4 10 5 3 323 9 5 31 33 4 9 2 1 5 2 33 3 2 4 1.init node costs to , set p = seed point, cost(p) = 0 2.expand p as follows: for each of p’s neighbors q that are not expanded set cost(q) = min( cost(p) + c pq, cost(q) ) –if q’s cost changed, make q point back to p put q on the ACTIVE list (if not already there) 3.set r = node with minimum cost on the ACTIVE list 4.repeat Step 2 for p = r ALGORITHM 3 sets of nodes Free Active Done

39. The University of Ontario Dijkstra’s shortest path algorithm 3 10 5 3 323 6 5 31 33 4 9 2 4 31 4 5 2 33 3 2 4 1.init node costs to , set p = seed point, cost(p) = 0 2.expand p as follows: for each of p’s neighbors q that are not expanded set cost(q) = min( cost(p) + c pq, cost(q) ) –if q’s cost changed, make q point back to p put q on the ACTIVE list (if not already there) 3.set r = node with minimum cost on the ACTIVE list 4.repeat Step 2 for p = r ALGORITHM 3 sets of nodes Free Active Done

40. The University of Ontario Dijkstra’s shortest path algorithm 3 10 5 3 323 6 5 31 33 4 9 2 4 31 4 5 2 33 3 2 4 2 1.init node costs to , set p = seed point, cost(p) = 0 2.expand p as follows: for each of p’s neighbors q that are not expanded set cost(q) = min( cost(p) + c pq, cost(q) ) –if q’s cost changed, make q point back to p put q on the ACTIVE list (if not already there) 3.set r = node with minimum cost on the ACTIVE list 4.repeat Step 2 for p = r ALGORITHM 3 sets of nodes Free Active Done

41. The University of Ontario Path Search (basic idea) A B Dijkstra algorithm - processed nodes (distance to A is known) - active nodes (front) - active node with the smallest distance value

42. The University of Ontario Dijkstra’s shortest path algorithm n Properties It computes the minimum cost path from the seed to every node in the graph. This set of minimum paths is represented as a tree Running time, with N pixels: –O(N 2 ) time if you use an active list –O(N log N) if you use an active priority queue (heap) –takes < second for a typical (640x480) image Once this tree is computed once, we can extract the optimal path from any point to the seed in O(N) time. –it runs in real time as the mouse moves What happens when the user specifies a new seed?

43. The University of Ontario Livewire extensions n Directed graphs n Restricted search space Restricted domain (e.g. near a priori model) Restricted backward search n Different edge weight functions Image-Edge strength Image-Edge Curvature Proximity to known approximate model/boundary n Multi-resolution processing

44. The University of Ontario Results http://www.cs.washington.edu/education/courses/455/03wi/projects/project1/artifacts/index.html