1. Level set method and image segmentation
Lecture 4
Level set method and image segmentation

2. Overview
What is image segmentation? Separation of image domain based on contents.
General image segmentation:

3. Overview
What is image segmentation? Separation of image domain based on contents.
Object recognition (computer vision)

4. Overview
What is image segmentation? Separation of image domain based on contents.
Object recognition (computer vision)

5. Overview
What is image segmentation? Separation of image domain based on contents.
Medical image segmentation

6. Overview
What is image segmentation? Separation of image domain based on contents.
Medical image segmentation

7. Overview
What is image segmentation? Separation of image domain based on contents.
Medical image segmentation
Computer-aided diagnosis

8. Overview
What is image segmentation? Separation of image domain based on contents.
Image segmentation in biology

9. Overview What is important for image segmentation? Main approaches:
Edges which can be extracted by differentiation (lower level segmentation)
Content classification (higher level segmentation)
Main approaches:
Energy method (Mumford-Shah)
Curve evolution (snake, level set method)

10. Energy Method
Mumford-Shah (MS) Model

11. Mumford-Shah Functional
Mumford-Shah (MS) functional:
Open questions:
Observed Image
Set of Discontinuities

12. Finite Difference Approximation of MS Model: A. Chambolle, 1995
Finite difference approximation (1D)
Finite difference approximation (2D)
where and
Set of discontinuities of u
Hard to solve!

13. Curve Evolution
Snake, Geodesic Active-Contour, Chan-Vese Model, Level Set Method

14. Overview Objective: automatically detect contours of objects.
Questions:
How contours (curves in 2D or surfaces in 3D) are represented?
Explicit representation (parametric)
Implicit representation (level set)
How the locations of contours are determined?

15. Snakes: Active contour models
(Kass, Witkin and Terzopoulos, 1987)
where
Edge Indicator Function
Makes curves act membrane like.
Makes curves act thin plate like.
Makes curves be stuck at edges.

16. Snakes: Active contour models
Gradient flow:
Drawbacks of “snakes”:
Curves’ representation is not intrinsic. We could obtain different solutions by changing the parametrization while preserving the same initial curve.
Because of the regularity constraint, the model does not handle changes of topology.
We can reach only a local minimum, we have to choose initial curve close enough to the object to be detected.
The choice of a set of marker points for discretizing the parametrized evolving curve may need to be constantly updated.
where

17. The Geodesic Active Contours Model
Dropping the second order term of “snakes”
Geodesic active contours model
Intrinsic! Let
Idea: weight defines a new Riemannian metric for which we search for geodesics.
What’s their relations?

18. Curvature: Elements of Differential Geometry
Parametric curves:
Note:
Curvature:
and

19. Curvature: Elements of Differential Geometry
Parametric curves:
Note:
Curvature (if the curve is parameterized by arc length):

20. Curvature: Elements of Differential Geometry
Curves as level set of a function :
Differentiating the equation
Suppose , then
Differentiating one more time

21. Link Between Snakes and Geodesic Active Contours
Let and
Calculus of We have
Assuming :
Integration by parts

22. Link Between Snakes and Geodesic Active Contours
Denote arc length by , then
Thus
Decompose in tangential and normal directions
where T and N are normalized tangent and normal vectors.

23. Link Between Snakes and Geodesic Active Contours
Then
By Cauchy-Schwartz inequality, the flow leads to most rapid decrease of the energy functional is

24. Link Between Snakes and Geodesic Active Contours
Calculus of We have
Then,
Integration by part w.r.t. q

25. Link Between Snakes and Geodesic Active Contours
Then
( by )
Flow of most rapidly decrease

26. Note: Mean Curvature Flow
If g is constant, we obtain the mean curvature flow:

27. Link Between Snakes and Geodesic Active Contours
Under suitable conditions, the models of snakes and geodesic active contours are equivalent in the following sense:
Why is this true?
Plug
in and respectively

28. Caselles et al.’s Modification
Drawbacks of original geodesic active contours: hard to detect nonconvex objects.
Improved evolution equation
Note: the above flow does not correspond to any energy functional unless g is a constant.
Implement the flow by level set method

29. Level Set Method
Level set method is an efficient and effective method in solving curve evolution equations:
Observation: A curve can be seen as the zero-level of a function in higher dimension.
Consider such that
Differentiating w.r.t. t

30. Level Set Method
If the level set function is negative inward and positive outward of the curve, then the unit inward normal vector is

31. Level Set Method Level set equation
Level set formulation of geodesic active contours
where

32. Level Set Method Discretization Mean curvature motion
Reinitialization every n steps (Section 4.3.4)
Central differencing

33. Level Set Method What does reinitialization do? How can this happen?
Solution of mean curvature flow with initial curve a unite circle
0-level set
Becomes flat as t approaches 1/2

34. Level Set Method
Discretization
Mean curvature motion
t increases

35. Level Set Method Discretization Scalar speed evolution where
Non-oscillotary upwind discretization

36. Level Set Method
Discretization
Scalar speed evolution
c=1
c=-1

37. Level Set Method Discretization Pure advection equation
Motion vector field
One-side upwind discretization

38. Level Set Method
Discretization
where

39. Level Set Method
Numerical results

40. Brain Aneurysm Segmentation
B. Dong et al., Level set based brain aneurysm capturing in 3D, Inverse Problems and Imaging, 4(2), ,

41. Brain Aneurysm Segmentation
Raw CT data

42. Brain Aneurysm Segmentation
Vascular tree segmentation (e.g. using CV model)

43. Brain Aneurysm Segmentation
Objective: separate aneurysm from vascular tree

44. Brain Aneurysm Segmentation
What was commonly done by doctors
Problems:
Robustness: hard to perform for complicated vessels.
Consistency: hard to unify the cut cross subjects.

45. Brain Aneurysm Segmentation
Understanding of the problem: illusory contours

46. Brain Aneurysm Segmentation
Zhu and Chan, 2003:

47. Brain Aneurysm Segmentation
From 2D to 3D: choice of curvature

48. Brain Aneurysm Segmentation
Zhu&Chan
Ours
Initial
Final Result

49. Energy Method - revisited
Approximations of Mumford-Shah Model

50. Binary Approximation of Mumford-Shah Model
Active Contours without Edges, T.F. Chan and L. A. Vese, (CV model)

51. Binary Approximation of Mumford-Shah Model
Level set representation of curves
Level set formulation of curve length and area inside C

52. Binary Approximation of Mumford-Shah Model
Level set formulation of CV model
Solution of piecewise constant MS model

53. Binary Approximation of Mumford-Shah Model
Solving CV model
Solving for the constants c

54. Binary Approximation of Mumford-Shah Model
Solving CV model
Solving for the level set function

55. Binary Approximation of Mumford-Shah Model
Solving CV model
Solving for the level set function
H2 has global support while H1 is local.

56. Binary Approximation of Mumford-Shah Model
Solving CV model
Solving for the level set function

57. Binary Approximation of Mumford-Shah Model
Solving CV model
Solving for the level set function
Reinitialize after every n iterations

58. Binary Approximation of Mumford-Shah Model
Advantages of CV model
Do not rely on image gradient, robust to noise

59. Binary Approximation of Mumford-Shah Model
Advantages of CV model
Do not rely on image gradient, robust to noise
Find boundaries without edge information

60. Convexified Binary MS Model
Binary Mumford-Shah
CV model
Gradient flow of CV model

61. Convexified Binary MS Model
When a non-compactly supported smooth Heaviside function is chosen, the PDE has the same stationary solution as
Corresponding gradient flow
Convexified segmentation model

62. Convexified Binary MS Model
The non-convex MS model can be solved by a convex relaxation model!
Further development:
Full convexification and generalization (UCLA CAM 10-43, )
Piecewise polynomial (UCLA CAM 13-50)
Generalization on graphs (UCLA CAM 12-03, 14-79)

63. Homework (Due April 20th 11:59pm)
Implement the geodesic active contours model using level set formulation (ppt page 38).
Use images of your own selection
Observe:
Segmentation results for different types of images
Effects of noise and blur on the results
Note: codes of reinitialization will be provided.