Today: Image Segmentation Image Segmentation Techniques Snakes Scissors Graph Cuts Mean Shift Wednesday (2/28) Texture analysis and synthesis Multiple hypothesis tracking Final Project Presentations Q: one two-hour slot or two one-hour slots? March 12 or 14?

From Images to Objects What Defines an Object? Subjective problem, but has been well-studied Gestalt Laws seek to formalize this –proximity, similarity, continuation, closure, common fate –see notes by Steve Joordens, U. Torontonotes

Image Segmentation Many Approaches Proposed color cues region cues contour cues Today: Three Approaches Mean Shift (color-based) –D. Comaniciu, P. Meer, Robust Analysis of Feature Spaces: Color Image Segmentation, Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR'97), 1997, Robust Analysis of Feature Spaces: Color Image Segmentation Normalized Cuts (region-based) –J. Shi and J. Malik, Normalized Cuts and Image Segmentation, IEEE Conf. Computer Vision and Pattern Recognition(CVPR), 1997Normalized Cuts and Image Segmentation Snakes and Scissors (contour-based) –M. Kass, A. Witkin, and D. Terzopoulos, Snakes: Active contour models. International Journal of Computer Vision. v. 1, n. 4, pp , 1987.Snakes: Active contour models –E. N. Mortensen and W. A. Barrett, Intelligent Scissors for Image Composition, in ACM Computer Graphics (SIGGRAPH `95), pp , 1995Intelligent Scissors for Image Composition

Color-Based Segmentation Segmentation by Histogram Processing Given image with N colors, choose K Each of the K colors defines a region –not necessarily contiguous Performed by computing color histogram, looking for modes This is what happens when you downsample image color range, for instance in Photoshop

Finding Modes in a Histogram How Many Modes Are There? Easy to see, hard to compute

Mean Shift [ Comaniciu & Meer] Iterative Mode Search 1.Initialize random seed, and fixed window 2.Calculate center of gravity of the window (the “mean”) 3.Translate the search window to the mean 4.Repeat Step 2 until convergence

Mean-Shift

More ExamplesMore Examples:

Region-Based Segmentation We Want Regions why not build this in as a constraint?

q Images as Graphs [Shi & Malik] Graph G = (V, E, W) node for every pixel edge between every pair of pixels, p,q weight w pq for each edge –w pq measures similarity »similarity: difference in color and position p w pq w

Segmentation by Graph Cuts Break Graph into Segments Delete edges that cross between segments Easiest to break edges that have low weight: –similar pixels should be in the same segments –dissimilar pixels should be in different segments w ABC

Cuts in a graph Edge Cut set of edges whose removal makes a graph disconnected cost of a cut: A B Normalized Cut a cut penalizes large segments fix by normalizing for size of segments Vol(A)Vol(B)

The Normalized Cut (NCut) criterion NP-Hard! Given a Graph G = (V, E, W) Find A ½ V that minimizes

Normalize Cut in Matrix Form

Eigenvalue Problem After lot’s of math, we get: This is a Rayleigh Quotient Solution given by “generalized” eigenvalue problem: Solved by converting to standard eigenvalue problem: Subtleties optimal solution is second smallest eigenvector gives real result—must convert into discrete values of y

Interpretation as a Dynamical System Weights are Springs eigenvectors correspond to vibration modes

Interpretation as a Dynamical System Weights are Springs eigenvectors correspond to vibration modes

Color Image Segmentation

Graph Cuts Formulations We’ve seen graph cuts before Stereo (Y. Boykov, O. Veksler, and Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts)Fast Approximate Energy Minimization via Graph Cuts The formulation is different however –In Boykov approach, each pixel was connected to a label node –Boykov approach gives better optimality guarantees, but requires discrete labels

Segmentation Using Edges Edges  Segments Spurious edges Missing edges How could user-interaction help?

Snakes [Kass et al.]

Snakes: Energy Minimization Ingredients: A contour An energy function Try to make the contour snap to edges minimize E snake Issues contour representation energy function minimization algorithm

Snakes Contour representation splines [Kass et al., IJCV 89, original formulation] discrete set of points [Williams & Shah, CVGIP 55(1), 92] implicit function [Malladi et al., 95 (see readings] Energy functional Variations on edge term (gradient magnitude, laplacian) Minor variations in smoothness terms (first/second order) Volume expansion (balloon) or contraction Deformable templates [Yuille, IJCV 92] Learned shape models [Isard & Blake, others] Minimization algorithm Numerical Euler integration [e.g., Kass] Greedy algorithms [e.g., Williams] Level set evolution [e.g., Malladi]

Issues: Scale Bandpass Images Lowpass Images Pyramid snakes for coarse-to-fine processing

Problem with Snakes You can’t control the shape evolution* * to be fair: Kass et al described how you can add forces to push and pull the snake as desired, but still kind of kludgy

Intelligent Scissors [Mortensen & Barrett]

Intelligent Scissors Approach answers a basic question Q: how to find a path from seed to mouse that follows object boundary as closely as possible? A: define a path that stays as close as possible to edges

Intelligent Scissors Basic Idea Define edge score for each pixel –edge pixels have low cost Find lowest cost path from seed to mouse seed mouse Questions How to define costs? How to find the path?

Defining Costs Link Costs Costs are assigned to a pair of adjacent pixels (a link) f z (q) = 0 if Laplacian is 0 at q f z (q) = 1 otherwise f d (q) measures difference in gradient direction p q link

Path Search Graph Search Algorithm Computes minimum cost path from seed to all other pixels

Results