Snakes - Active Contour Lecturer: Hagit Hel-Or By: Saar Arbel Lecturer: Hagit Hel-Or

What is our objective? But Wait... To find a contour that best approximates the perimeter of an object Subdividing or partitioning an image into its constituent regions or objects But Wait... Its just like segmentation and many other methods...

Problem with other methods Not effective in presence of noise and sampling artifacts (e.g. medical images). This is done with segmentation on a noisy image

This is done with snakes on a noisy image

What is a snake? A framework for drawing an object outline from a possibly noisy 2D image. An energy-minimizing curve guided by external constraint forces and influenced by image forces that pull it towards features (lines, edges). Represents an object boundary or some other salient image feature as a parametric curve

So... Why snakes? Snakes are autonomous and self-adapting in their search for a minimal energy state They can be easily manipulated using external image forces They can be used to track dynamic objects in temporal as well as the spatial dimensions

Applications Object tracking Shape recognition Segmentation Segmentation example Shape recognition Segmentation Edge detection

Aspects we need to consider Representation of the contours Defining the energy functions - Internal - External Minimizing the energy function

How is a snake represented? A Snake is represented as a set of points (x,y) that defines a function V(s).

Every snake include: External Energy Function Internal Energy Function A set of k points (in the discreet world) or a continuous  function that will represent the points

The Energy functions The Snake's energy functions measure the appropriateness of the contour Good solutions correspond to the minimum of the energy functions Getting the snake closer to the target In charge of the snake curvature and smoothness Our goal is to minimize this function with respect to the contour parameters

A brief example:

The Energy functions Our goal is to find a function V that will minimize E and give us a perfect match

Intuitive tip You can think about the energy function like a heuristic function, they are similar because they both lead to a solution in a way that lets the programmer decide what is important and what’s not.

Internal Energy Econt Ecurv Forces the contour to be continuous Forces the contour to be smooth Decide if the snake wants to shrink/expand Econt Ecurv

Econt Forces the contour to be continuous Minimizing the first derivative has two effects: The snake wants to shrink The squaring of  the differences causes the differences to be as similar as possible (the points are at equal distances from each other)

On the continuous world: On the discrete world: The contour is approximated by N points P1, P2, . .., Pn and the first derivative is approximated by a finite difference:

Example The points are equi-distances The points are not equi-distances

Ecurv Forces the contour to be smooth. By taking the second derivative which represent the difference of the differences  between 2 consecutive differences. This 2 consecutive differences represent the curvature and we want to penalize if the curvature is too high.

On the continuous world: On the discrete world: The curvature can be approximated by the following finite difference:

Example: compare curvature That’s better!

α,β meaning Reminder: We can play with α,β and to control the relation between the first derivative and the second derivative and get different results

Small α Big α Small alpha make the energy function insensitive to the amount of stretch Increase the internal energy of the snake as it stretches more and more

Small β Big β Small beta causes snake to allow large curvature so that snake will curve into bends in the contour When beta is large then high price for curvature so snake prefers to be smooth and not curving

Internal energy - Summary On the continuous world: On the discrete world:

Discrete representation Represent the snake with a set of n points (X0 , Y0) (X3 , Y3) (X14 , Y14)

Open and close snake closed snake open snake Dealing with open snake is easier and our algorithm will run faster (soon…)

External Energy – The Energy map An energy map is a function f (x, y) that we extract from the image – I(x, y) The energy map’s task is to emphasize the areas of the contour and to weaken the other areas of the image by creating a new image. By doing this we can attract our snake near the “emphasized” pixels which represent the contour of the image.

How do we build an Energy map? By given an image – I(x, y) , we can build an energy map – f(x, y), that will attract our snake to edges on our image. For each pixel we will do: (color image) (black-white image) By doing this we can easily locate the edges on the images because the gradient there is big.

Energy map Image

External Energy Attracts the contour towards the closest image edge with dependence on the energy map. determines whether the snake feels attracted to object boundaries

Intuitive example Initial state Final state

Eexternal In order to make snakes useful for early vision we need energy functions that attract them to salient features in images. I will present you 2 different energy functions which attract a snake to lines & edges.

Eline Depending on the sign of Wline the snake will be attracted either to bright lines or dark lines.

Eedge Reminder: Gradient is a generalization of the usual concept of derivative to the functions of several variables

Clarification Note: As long as the points on the snake are closer to the edge (where the gradient is bigger) and then the Eedge is smaller, so that’s a good thing because that's how the external energy works! this gets the snake closer and closer to the target contour.

Examle Image External force Zoomed in

External energy - Summary On the continuous world: On the discrete world:

Example – External energy

How to find the optimal function V? Several algorithms have been proposed to fit deformable contours. We’ll look at two: -  Greedy search -  Dynamic programming (for 2d snakes)

Greedy algorithm A greedy algorithm makes locally optimal choices, hoping that the final solution will be globally optimal. For each point of the snake, we search a window around it (window size: m) and move to where the energy function is minimal. We will stop after enough points have not changed in the last iteration 

Points are calculates by an iterative process: Energy function for each point in the local neighborhood is calculated The point is moved to the next point with lowest energy function This process is repeated for every point Iteration is done until termination condition met Defined number of iterations Complexity:

But... Convergence not guaranteed!! Need decent initialization Works very well as far as the initial snake is not too far from the desired solution

Viterbi algorithm Dynamic programming algorithm We need to make the snake discrete in order to use the algorithm The discretization of the external energy depends on the particular energy term but the internal energy is always the same and can be discretized as follows:

Our problem is to finding the set of vertex positions V that minimizes:

(The internal energy is a local property of neighboring vertices) We will now make the assumption that the snake is open and that β=0 In most cases, snake energy can be rewritten as a sum of pair-wise interaction potential: (The internal energy is a local property of neighboring vertices)

This means that each vertex position Vi influences the total energy only through the terms and Each vertex has m possible moves

We now define n variables (i=2,3,…,n) Each contains the lowest total energy for the first k-1 vertices of the snake for a given value of . Thus the minimum energy E of the whole snake is equal to

The globally best position for the snake is therefore computed by first computing all the , so we determine at each node the optimal position of its predecessor for each possible location of the node. When we have computed we can find the optimal position for by minimizing the expression Once we know the position of the last node we look up the optimal position for the second last node and so on until we have determined the optimal position for all the nodes.

states 1 2 … m sites Complexity:

Now, β≠0 If β≠0 then the decomposition of the energy function into local terms becomes: This means that each vertex position Vi influences the total energy only through the terms and

And now we will define again n variables (i=2,3,…,n) Note that we have to compute a table of values at each node instead of only m values per node for the simpler snakes with β=0. Complexity:

Now, the snake is closed Solution: 1. Can use Viterbi to optimize snake energy in case v1 = c is fixed. (in this case the energy above effectively has no loop) 2. Use Viterbi to optimize snake for all possible values of c and choose the best of the obtained m solutions. Complexity:

Snakes - Pros Vs. Cons Pros: Snakes optimize globally. Snakes are 1-dimensional which means that we can reduce 2-dimensional optimization problems to 1-dimensional optimization problems. Active contour models provide a unified solution to several image processing problems such as the detection of light and dark lines and edges. Useful to track and fit non-rigid shapes. Flexibility in how energy function is defined.

Cons: Must have decent initialization near true boundary, may get stuck in local minimum. Parameters of energy function must be set well based on prior information.

Extensions We will review two main applications of snakes: -  Snake splitting -  Snake expending

Snake Splitting Problem: Snakes including multiple objects can not always extract each of them individually (when initialize the contour wrong) Solution: Using the split-and-merge contour model based on detecting self- and mutual-crossings of the contour. The method: There is a single contour which surrounding all objects and then iteratively splits into multiple contours by cutting at self-crossing points for extracting multiple objects individually

Snake Expanding There is some cases when our snakes would need to expand in order to get close to the contour.

Problems: If there is no image force (F=0), the curve shrinks on itself and vanishes to a point. Often, due to noise, some isolated points are gradient maxima and can stop the curve when it passes by. We can get stuck inside a shape (if our image force=0) So we need the define new force for this cases:

Solution: we add another force which makes the contour have a more dynamic behavior. You can think about our curve as a “Balloon” Force which inflate our snake The new Force

Cool demonstrations

Summary Snakes include: - Two energy functions (Internal and External) - N points who represents a function V Main problem: Finding an optimal function V that minimize the Energy function Solved by: - Greedy algorithm - Viterbi algorithm

References M. Kass, A. Witkin, and D. Terzopoulos, "Snakes: Active contour models.“, International Journal of Computer Vision. v. 1, n. 4, pp. 321-331, 1987. 1. http://www.computing.edu.au/~jim/thesis/ivins02.pdf 3. Wikipedia – Active contour models (Snakes) 2. http://www.cse.unr.edu/~bebis/CS791E/Notes/Def ormableContours.pdf 4. http://www.cs.utexas.edu/~grauman/courses/378/ handouts/snakes.pdf 5. http://cs.gmu.edu/~kosecka/cs682/lect-snakes.pdf 6.

Thank you