1. Image Segmentation and Registration Rachel Jiang Department of Computer Science Ryerson University 2006

2. Rachel Jiang, 2006 2 Rigid registration Intensity-based methods –Have been very successful Monomodal Multimodal –Assuming a global relationship between the … images to register –Deriving a suitable similarity measure Correlation coefficient Correlation ratio Mutual information Block matching …

3. Rachel Jiang, 2006 3 Methods for fusing images There are three general methods for fusing images from different (or the same) image modalities: –landmark matching include external fiducial landmarks or anatomic landmarks. –surface matching uses an algorithm that matches different images of the same patient surface. –intensity matching.

4. Rachel Jiang, 2006 4 intensity matching uses mutual intensity information to co- register different images. The matched intensities may come from the same scanner –two different MRI scans acquired on different days – from different modalities such as MRI and PET.

5. Rachel Jiang, 2006 5 Rigid Body Model Most constraint model for medical imaging asserts that the distance and internal angles within images can not be changed during the registration

6. Rachel Jiang, 2006 6 Non rigid model can detect and correct discrepancies of small spatial extent, by deforming one of the images (source) to match the other (reference). Spatial deformation model can be based on different physical properties like elasticity or viscosity, or their generalizations and simplifications. Deformation is driven by external forces, which tend to minimize image differences, measured by image similarity measures.

7. Rachel Jiang, 2006 7 Mutual information The mutual information of two images is the amount of information that one image contains about the other or vice versa. Transforming one image with respect to the other such that their mutual information is maximized. The images are assumed to be registered.

8. Rachel Jiang, 2006 8 Mutual information A function of transformation between the images –an algorithm that searches maximum value for a function that gives the alignment information between images –different transformation estimates are evaluated (those transformation estimates will result in varying degrees of overlap between the two images) –(MI as a registration criteria is not invariant to the overlap between images)

9. Rachel Jiang, 2006 9 MI

10. 10 MI in 2D/3D space –In 3D space We attempt to find the registration by maximizing the information that one volumetric image provides about the other. –In 2D space Two curve that are to be matched as the reference curve and the current active evolving curve. We seek an estimate of the transformation that registers the reference curve and the active curve by maximizing their mutual information

11. Rachel Jiang, 2006 11 Degree of freedom Six degree of freedom charactering the rigid movements –3 describe the Rotation –3 describe Translation

12. Rachel Jiang, 2006 12 Similarity measures Common registration methods can be grouped as –Feature based techniques Rely on the presence and identification of natural landmarks or fiducial marks in the input dataset to determine the best alignment –Intensity-based measures Operate on the pixel/voxel intensities directly –Varies statistics are calculated by using the raw intensity values

13. Rachel Jiang, 2006 13 Taking geometric constraints Special land marks on human body Manually embed land marks Spacial correlations …

14. Rachel Jiang, 2006 14 Correlation ratio Given two images I and J, the basic principle of the CR method is to search spatial transformation T and an intensity mapping f such that, by displacing J and remapping its intensities, the resulting image f(JoT) be as similar as possible to I.

15. Rachel Jiang, 2006 15 Example of Image Segmentation Bone fracture detection

16. Rachel Jiang, 2006 16 Image Segmentation Techniques threshold techniques –make decisions based on local pixel information edge-based methods –Weakness: broken contour lines causes failure region-based techniques –partitioning the image into connected regions by grouping neighbouring pixels of similar intensity levels. –Adjacent regions are then merged under certain criterion. Criteria create fragmentation or overlook blurred boundaries and overmerge. Active contour models

17. Rachel Jiang, 2006 17 Deformable models –Snakes/Balloons/Deformable Templates provide a curve as a compromise between regularity of the curve and high gradient values among the curve points. –(Kass et al., 1988; Cohen, 1991; Terzopoulos, 1992) –Level set methods Level Set Methods are numerical techniques which can follow the evolution of interfaces. These interfaces can develop sharp corners, break apart, and merge together. –(Osher and Sethian, 1988; Sethian, 2001) –Geodesic Active Contour take the advantages of both Snake and Level set methods –(Caselles et al., 1995; Malladi et al, 1995; Sapiro, 2001)

18. Rachel Jiang, 2006 18 The Snake formula

19. Rachel Jiang, 2006 19 Snake: image force

20. Rachel Jiang, 2006 20 Snake: example

21. Rachel Jiang, 2006 21 Level Set Method Level Set Methods –provide formulation of propagating interfaces, a mathematical formulation and numerical algorithm for tracking the motion of curve and surfaces (Osher and Sethian, 1988; Sethian, 2001) –For segmenting several objects simultaneously or an objects with holes, it is possible to model the contour as a level set of a surface, allow it to change its topology in a nature way (Cohen, 1997).

22. Rachel Jiang, 2006 22 Level Set

23. Rachel Jiang, 2006 23 Formula of Level Set method Curve evolving formula:

24. Rachel Jiang, 2006 24 LSM: example

25. Rachel Jiang, 2006 25 LSM: More Example

26. Rachel Jiang, 2006 26 Geodesic Active Contour (GAC) –presents some nice properties the initialization step does not impose any significant constraint can deal successfully with topological changes, finding the global minimum of energy minimizing curve can be solved by mapping the boundary detection problem into a single minimum problem. The new model mathematically inherit –the way handling the topological changes from the Level Set – the minimizing deformation energy function with ‘internal’ and ‘external’ energies along its boundary from the traditional Snake. –This model inherit the advantages of LS and ‘Snakes’ by transform mathematical formulation of Snake – Lagrenge formula with PDEs –The theory behind the GAC is the use of partial differential equations and curvature-driven flows. (Caselles et al., 1995; Malladi et al, 1995; Sapiro, 2001)

27. Rachel Jiang, 2006 27 Segmentation result using ACM

28. Rachel Jiang, 2006 28 Surface reconstruction

29. Rachel Jiang, 2006 29 Mathematical Morphology –provides the foundation for measuring topological shape, size, location. The theory behind mathematical morphology is defining computing operations by primitive shapes –Georges Matheron, Jean Serra and their colleagues of Centre de Morphologie Mathematique –G.Matheron 1975, Serra, 1982, Vicent, 1990… –offer several robust theories and algorithms to implement on digital images to extract complex features –uses ‘Set Theory’ as the foundation for its functions. The simplest functions to implement are ‘Dilation’ and ‘Erosion’.

30. Rachel Jiang, 2006 30 Erosion and Dilation (1D)

31. Rachel Jiang, 2006 31 Erosion and Dilation (2D)

32. Rachel Jiang, 2006 32 Erosion & Dilation example

33. Rachel Jiang, 2006 33 Opening & Closing

34. Rachel Jiang, 2006 34 Shape Operators  Shapes are usually combined by means of : Set Intersection (occluded objects): Set Union (overlapping objects):

35. Rachel Jiang, 2006 35 Dilation B A

36. Rachel Jiang, 2006 36 Dilation

37. Rachel Jiang, 2006 37 Extensitivity B A

38. Rachel Jiang, 2006 38 Erosion B A

39. Rachel Jiang, 2006 39 Erosion

40. Rachel Jiang, 2006 40 Erosion

41. Rachel Jiang, 2006 41 Opening and Closing  Opening and closing are iteratively applied dilation and erosion Opening Closing

42. Rachel Jiang, 2006 42 Opening and Closing

43. Rachel Jiang, 2006 43 Opening and Closing  They are idempotent. Their reapplication has not further effects to the previously transformed result

44. Rachel Jiang, 2006 44

45. Rachel Jiang, 2006 45 Watershed

46. Rachel Jiang, 2006 46 Capturing the shape prior the curve C and the transformation S, R, T is calculated such that the curve C new = SRC + T and C * are perfectly aligned. The minimization problem now can be solved by finding steady state solutions to the following system

47. Rachel Jiang, 2006 47 Minimization processing system

48. Rachel Jiang, 2006 48 Distance measure d(x,y) = d(C *,(x,y)) is the distance of the point (x,y) from C* The function d is evaluated at SRC(p) + T

49. Rachel Jiang, 2006 49 Minimizing Energy function