1. Snakes with Some Math

2. Acknowledgements: University of Western Ontario, University of Manchester, demos from Visual Dynamics Group (University of Oxford),

3. Given: initial contour (model) near desirable object
Snakes
Snakes, active contours [Kass, Witkin, Terzopoulos 1987]
In general, deformable models are widely used
Given: initial contour (model) near desirable object

4. Snakes Snakes, active contours [Kass, Witkin, Terzopoulos 1987]
In general, deformable models are widely used
Given: initial contour (model) near desirable object
Goal: evolve the contour to fit exact object boundary

5. Tracking via deformable models
Use final contour/model extracted at frame t as an initial solution for frame t+1
Evolve initial contour to fit exact object boundary at frame t+1
Repeat steps 1 and 2 for t ‘= t+1

6. Tracking via deformable models
Tracking Heart Ventricles

7. “Snakes” A smooth 2D curve which matches to image data
Initialized near target, iteratively refined
Can restore missing data
intermediate
final
initial

8. Parametric Curve Representation (continuous case)
A curve can be represented parametrically
open curve
closed curve
Note: in computer vision and medical image analysis communities the term “snake”
is normally associated with such parametric representation of contours.

9. Internal Energy The bending energy of a continuous curve is
Elasticity
Stiffness
The more the curve bends, the larger this value

10. External energy
The external energy describes how well the curve matches the image data locally
Numerous forms can be used, attracting the curve toward different image features

11. Simple Edge Strength Suppose we have an image I(x,y)
Generate gradient images &
External energy based on edge strength at a point is then
An external energy term for a snake is
thus
(Negative in order that minimizing it forces the curve toward large edges)

12. Snake Energy (continuous form)
The total energy of the snake is defined as
e.g. bending energy
e.g. total edge strength
under curve

13. Discrete approach discrete snake representation discrete optimization
(dynamic programming)
discrete image

14. Discrete and Continuous approaches
Models in
Image Analysis
Discrete
Models in
Image Analysis
The relationship is somewhat analogous to relation between continuous and discrete models in physics, e.g.
Continuous Model of Light
Electro-Magnetic Waves
Discrete Model of Light
Photons

15. Parametric Curve Representation (discrete case)
Represent the curve with a set of n points

16. Discrete Representation
If the curve is represented by n points
Elasticity
Stiffness

17. Simple Elastic Curve (example)
For a curve represented as a set of points a simple elastic energy term is
This encourages the closed curve to shrink to a
point (like a very small elastic band)

18. Encouraging point spacing
To stop the curve from shrinking to a point, add a term such as [v-v’]^2 where v’ is average distance between snaxels.

19. Simple Edge Strength
An external energy term for a (discrete) snake based on image edge
(Negative in order that minimizing it forces the curve toward large edges)

20. Simple Elastic Snake A simple elastic snake is thus defined by
A set of n points,
An internal elastic energy term
An external edge based energy term
To use this to locate the outline of an object
Initialize in the vicinity of the object
Modify the points to minimize the total energy

21. Simple Elastic Snake In this case we have
where
Optimization problem (2n variables)
note that we can easily compute the derivatives, which allows efficient optimization (via gradient descent) converging to a local minima
more robust option: Dynamic Programming

22. Synthetic example
(1)
(2)
(3)
(4)

23. Dealing with missing data
The smoothness constraint can deal with missing data (sometimes maybe wrong!):

24. Relative weighting
Notice that the strength of the internal elastic component can be controlled by a parameter,
Increasing this increases stiffness of curve
large
medium
small
Note: values of energy are normalized from 0 to 1, so if 0.1 times 10 becomes .01, etc.,
Then re-scale it. Basically this weight makes it more sensitive to internal energy.

25. Simple shape prior
If object is some smooth variation on a known shape, use
where give points of the basic shape

26. Alternative External (image) Energies
Directed gradient measures
Where is the unit normal to the boundary at contour point
This gives a good response when the boundary has the same direction as the edge, but weaker responses when it does not

27. Open and Closed Curves
closed curve
open curve
When using an open curve we can impose constraints on the end points (e.g. end points may have fixed position)

28. Additional Constraints
Snakes originally developed for interactive image segmentation
• Initial snake result can be nudged where it goes wrong
(correct the incorrect ones)
• Simply modify the external energy term to
– Pull nearby points toward cursor, or
– Push nearby points away from cursor

29. Interactive (external) forces
Pull points towards cursor:
Nearby points get pulled hardest
Negative sign gives better energy for positions near p

30. Interactive (external) forces
Push points from cursor:
Nearby points get pushed hardest
Positive sign gives better energy for positions far from p

31. Dynamic snakes
Adding motion parameters into hidden variables (for each snake node)
Introduce energy terms for motion consistency
Example: use optic flow to get candidate
points for the snaxels in the next frame.

32. Discrete Snakes Optimization
At each iteration we compute a new snake position within proximity to the previous snake
New snake energy should be smaller than the previous one
Stop when the energy can not be decreased within local neighborhood of the snake (local energy minima)
Optimization Methods
Gradient Descent
Dynamic Programming

33. Gradient Descent Example: minimization of functions of 2 variables
negative gradient at point (x,y) gives direction of the steepest descent towards lower values of function E

34. Stop at a local minima where
Gradient Descent
Example: minimization of functions of 2 variables
Stop at a local minima where

35. High sensitivity wrt. the initialisation !!
Gradient Descent
Example: minimization of functions of 2 variables
High sensitivity wrt. the initialisation !!

36. Gradient Descent for Snakes
simple elastic snake energy
energy as a function 2n variables
Force is a negative gradient of energy function (2n-dimentional vector)
We can break force/gradient into components corresponding to individual snake nodes

37. Discrete Snakes: “Gradient Flow” evolution
Contour evolution via
“Gradient flow”
Update equation for each node
Stopping criteria:
at a local minima of energy

38. Difficulties with Gradient Descent
Very difficult to obtain accurate estimates of high-order derivatives on images (discretization errors)
E.g., estimating requires computation of
Gradient descent is not trivial even for one-dimensional functions. Robust numerical performance for 2n-dimensional function is problematic.
Choice of parameter is non-trivial
Small , the algorithm may be too slow
Large , the algorithm may never converge
Even if “converged” to a good local minima, the snake is likely to oscillate near it

39. Alternative solution for 2D snakes: Dynamic Programming
In most cases, snake energy can be rewritten as a sum of pair-wise interaction potential
More generally, it can be written as a sum of triple-interaction potentials.

40. Snake energy: pair-wise interactions
Example: simple elastic snake energy
where
Q: give an example of snake with triple-interaction potentials?

41. DP Snakes [Amini, Weymouth, Jain, 1990] control points
First-order interactions
Energy E is minimized via Dynamic Programming

42. DP Snakes [Amini, Weymouth, Jain, 1990] control points
First-order interactions
Energy E is minimized via Dynamic Programming
Iterate until optimal position for each point is the center of the box,
i.e. the snake is optimal in the local search space constrained by boxes

43. Dynamic Programming (DP) Viterbi Algorithm
Here we will concentrate on first-order interactions
states 1 2 … m
sites
This is iterative, as search nbd changes
each time.
Complexity:

44. Dynamic Programming for a closed snake?
Clearly, DP can be applied to optimize an open ended snake
Can we use DP for a “looped” energy in case of a closed snake?

45. Dynamic Programming for a closed snake
Unfortunately, DP can not be directly applied in case of a “loop”.
However, some (approximation) tricks are often used in practice…
Use DP to optimize snake energy with fixed (according to a given initial snake position).
Use DP to optimize snake energy again. This time fix position of an intermediate node where is an optimal position obtained in step 1.

46. Problems with snakes Depends on number and spacing of control points
Snake may oversmooth the boundary
Not trivial to prevent curve self intersecting
Can not follow topological changes of objects
(having a model helps!)

47. Problems with snakes May be sensitive to initialization
may get stuck in a local energy minimum near initial contour
Numerical stability can be an issue for gradient descent and variational methods (continuous formulation)
E.g. requires computing second order derivatives
The general concept of snakes (deformable models) does generalize to 3D (deformable mesh), but many robust optimization methods suitable for 2D snakes do not apply in 3D
E.g.: dynamic programming only works for 2D snakes

48. Problems with snakes
External energy: may need to diffuse image gradients, otherwise the snake does not really “see” object boundaries in the image unless it gets very close to it.
image gradients
are large only directly on the boundary

49. Diffusing Image Gradients
image gradients diffused via Gradient Vector Flow (GVF)
Chenyang Xu and Jerry Prince, 98

50. Alternative Way to Improve External Energy
Use instead of where D() is
Distance Transform (for detected binary image features, e.g. edges)
binary image features
(edges)
Distance Transform
Distance Transform can be visualized as a gray-scale image
Generalized Distance Transform (directly for image gradients)

51. Distance Transform
Image features (2D) 3 4 2 5 1
Distance Transform
Distance Transform is a function that for each image pixel p assigns a non-negative number corresponding to distance from p to the nearest feature in the image I

52. Some more Variations..

53. Internal Energy Controls the contour smoothness and continuity
We use the discrete contour representation. For each snake element vi, its internal energy has two terms. The first term is defined as one minus costheta where theta is the angle between the vector vi minus one vi and vector vi, vi plus one. Minimization of this term will force theta toward zero so the first term controls the contour smoothness at vi. The second term of the internal energy is defined as the difference between the distance from vi to its neighbor point and the average distance between any two neighbor points, this constrains that points along the contour are equally distributed during the minimization process.

54. External Energy Limitations of the constraint of intensity homogeneity
Cannot be applied to open contour
Does not work for band-shaped object
Ultrasound image
Key-chain ring image
But the constraint of intensity homogeneity still has some limitations. For example, it can not be used to extract open contour. The left figure is an ultrasound image of the human tongue. In the ultrasound image, only part of the tongue is captured. The bright white band in the image is the air reflection at the upper surface of the tongue. The lower edge of the band is the tongue and the upper edge of the band is unrelated. There is no enclosed region where the intensity homogeneity constraint can apply.
Even for closed edge detection, the intensity homogeneity constraint has problem. For example, in the key-chain ring image, the ring is band-shaped. if the outer-edge of the ring is of interest, the intensity homogeneity constraint will fail since the region enclosed by the inner edge is more homogeneous than the outer edge.

55. Band Energy For the contour element , we can define its tangent as:
Two regions and can be defined for contour element
To solve these problems, we proposed a band energy.
To calculate the the band energy at point vi, we first define the tangent of vi as the direction of the line connecting its two
neighbor points, normal of vi can be obtained by rotating the tangent 90 degrees in clockwise or counter-clockwise direction, depending on the edge orientation. Then we can define two regions R and R’ for vi. R is in the normal direction and R’ is in the opposite of the normal direction. Depth of R or R’ is defined as 2 or 3 pixels which is the typical depth of the edge in an image.

56. Band Energy Normalized mean intensity difference
Band energy (given ring example is opposite)
Combination of intensity and gradient:
Penalty
We then can calculate the NMID between R and R’ and define the band energy Eband is defined according the mean intensity different. When the NMID is negative, we give Eband a big value as the penalty, otherwise it is defined as 1 minus the NMID. The final external energy in our model is the combination of Eband and the general external energy which has the gradient information. With the band energy, the upper edge of the white band in the ultrasound image and the inner edge of the key-chain ring will have a big penalty from Eband and can not attract the snake any more. And since Eband is calculated over regions, it is robust to speckle noise

57. Performance Without band energy With band energy
Image Initialization Without band energy With band energy
Here is an example to show the performance of our snake model. If the out…
For the ultrasound tongue image in the left, if the snake is initialized as shown in the middle-left, the middle-right is the result without the band energy. Some snake elements are attracted to the unrelated upper edge. But with the band energy, the tongue surface which is the lower edge the white band is correctly recovered.
To detect…, we can switch the role of region R and R’ by reversing the snake orientation and our snake model will work at the same way.

58. 3D reconstruction Experiments
Input
We also tested our snake model on CT lung images. There are 238 lung images which are on the horizontal plane of the body,
Cut from the top to bottom. Some images and results are shown here. By stacking all detected 2D contours togheter, we reconstructed the 3D lung and here are two views of the 3D lung.
Results