1. Segmentation of Medical Images under Topology Constraints Florent Ségonne Computer Science and Artificial Intelligence Laboratory, MIT Athinoula Martinos Center for Biomedical Imaging, MGH, Boston

2. Most macroscopic brain structures have the topology of a sphere.

3. Cortical surface C can be considered to have the topology of a sphere S GOAL: Achieve Accurate and Topologically Correct Segmentations of Anatomical Structures from Medical Images

4. Definition of Topological Defects Background TOPOLOGICAL DEFECT: deviation from spherical topology Handles or holes Cavities Disconnected components

5. Importance of Accurate, Topologically-Correct Segmentations Local functional organization of the cortex is largely 2-dimensional From (Sereno et al, 1995, Science)

6. Cortical parcellation Analysis of functional activitySpherical AtlasVisualization Shape Analysis Presurgical Planning Statistical analysis of morphometric properties Aging Neurodegenerative diseases Longitudinal studies of structural changes Hemispheric asymmetry

7. Image artifacts - Partial voluming a single voxel may contain more than one tissue type. - Bias field effective flip angle or sensitivity of receive coil may vary across space. - Tissue inhomogeneities even within tissue type (e.g. cortical gray matter), intrinsic properties such as T1, PD can vary (up to 20%). Topological notions are difficult to integrate into a discrete numerical framework Difficulties of Achieving Accurate, Topologically Correct Segmentations

8. Assigning tissue classes to voxels can be difficult Partial volume effect is often the cause for an incorrect topology

9. Previous Work Essentially two types of approaches: 1) Segmentation under topological constraint 2) Retrospective topology correction of segmentations

10. Previous Work Local decision to preserve topology might lead to large geometrical errors 1) Segmentation under Topological Constraints Active contours: Triangulations: self-intersection Dale et al.-99 Davatzikos-Bryan-96 MacDonald-et-al.-00 Level sets and Topologically constrained level-sets Zeng-Staib-Schultz-Duncan-99 Han-Xu-Prince-03 Homotopic digital deformations Mangin-et-al.-95 Poupon-et-al.-98 Bazin-Pham-05 Segmentation by registration and vector fields Karacali-Davatzikos-04 Christensen-et-al.-97

11. Final surfaceDeformationInitial condition Correct segmentation Incorrect segmentation Sensitivity to the initialization Incorrect segmentation Strict topology preservation is restrictive Previous Work

12. 1) Limitations of Segmentations under Topological Constraints Sensitivity to initialization Strict topology preservation is restrictive cannot distinguish handles from disconnected components or cavities  Section I

13. Location of the topological defects is hard to control Correction of the defects might not be optimal Previous Work 2) Retrospective Topology Correction of Segmentations Digital binary images Shattuck-Leahy-01 Han-Xu-Braga-Neto-Prince-02 Kriegeskorte-Goeble-01 Triangulations Guskov-Wood-01 Fischl-Liu-Dale-01

14. Topological defectInaccurate correctionAccurate correction Necessity to integrate additional information Solution is not necessarily obvious Previous Work Difficulty of finding the correct solution! Topological defect Sagital view Corrected defect Sagital view

15. 1) Limitations of Retrospective Topology Correction Methods Do not use all the available information corrections based on the size of the defect only Do not generate several potential solutions to find the best one cutting a handle or filling the corresponding hole may not generate the valid correction  Section II

16. Motivation 1) Existing methods are limited - segmentation under strict topology preservation is too restrictive. - retrospective topology correction methods do not use of additional information and cannot guarantee optimal topological corrections. 2) Discrete topological tools are restricted

17. Introduction Background Section I - Novel Approaches in Digital Topology Section III - Manifold Surgery Road Map

18. Background Topology in Discrete Imaging General Notions of Topology Topology in Discrete Imaging

19. Topology: study of shape properties preserved through deformations, twistings, and stretchings, but no tearings. [Massey 1967] = Background General Notions of Topology

20. Continuous theory Different levels of equivalence Background Difficult to adapt into a discrete framework Intrinsic Topology :- properties preserved by homeomorphisms - ignore the embedding space Homotopy type :- continuous transformations in the embedding space

21. Euler-characteristic , genus g of a surface S with V: # vertices E: # edges F: # faces in any polyhedral decomposition of S Background  = V-E+F Very useful to check the topology type of a triangulation  = 8-12+6=2  = 16-32+16=0 Euler-characteristic  is a topological invariant But no localization of the topological defects Related to the genus of a surface  = 2-2g Genus of a surface is equivalent to the number of handles

22. Adapt concept of continuity to a discrete framework Surfaces and 3D images - Tessellations - 3D digital images Background Topology in Discrete Imaging use the notion of connectivity instead Euler-characteristic Theory of Digital topology

23. Theory of Digital Topology Replace the notion of continuity by one of connectivity  Choice of connectivity n=8 or n=4 NEED TO WORK WITH A PAIR OF CONNECTIVITIES TO AVOID TOPOLOGICAL PARADOXES. Intersecting curves or not? Background One connectivity for the object X and one compatible connectivity for the inverse object

24. Background Þ 4 different pairs of compatible connectivities for the foreground F and Background B FB 626 6 6+18 6+ n=6 n=18 n=26 3D digital images: 3 different types of connectivity How to characterize the topology type of a given voxel ? How to detect topology changes ?

25. Characterize the topology type of a given voxel Simple point: point that can be added or removed from a digital object without changing its topology. They are characterized by: Local computations using the 3D neighborhood only  Fast Topological numbers (see Bertrand, 1994, Pat. Rec. Let.) Background

26. Homotopic deformation: addition or deletion of simple points. Background simple point in red – non simple point in yellow Problem: concept of simple point cannot distinguish different topological changes (formation of handles ≠ formation of cavities ≠ formation of disconnected components) Homotopic deformations are often too restrictive

27. Section I Novel Approaches in Digital Topology On the Characterization of Multisimple Points Genus Preserving Level Sets

28. Strict preservation of the topology is a strong constraint Notion of simple point is too restrictive Cannot distinguish different topological changes. Difficult to use topology preserving methods Sensitivity to initialization, noise, unexpected cavities, … Interested in handles Handles are hard to detect and retrospectively correct. Disconnected components, cavities are less problematic. Requires an extension of the concept of simple point On the Characterization of Multisimple Points

29. Goal: Develop digital techniques to prevent formation of handles but not the generation of cavities or of disconnected components. Definition: a multisimple point is a point that can be added or removed from a digital object X without changing the number of holes in X or the complement. On the Characterization of Multisimple Points  Need to find a characterization of multisimple points.

30. On the Characterization of Multisimple Points Can we use the topological numbers to characterize different topological changes ? Topology type 0Isolated point 0Interior point 11Border point ( = Simple Point ) 21Curve point >21Curves junction 12Surface point 1>2Surfaces junction >1 Surface(s)-curve(s) junction Voxel topology type and topological numbers

31. Topological numbers are not able to characterize multisimple points Topological numbers are computed locally: no information about the global connectivity of the digital object  extension of the concept of Topological Numbers

32. We introduce the set of neighboring components of a point x: We define two Extended Topological Numbers: On the Characterization of Multisimple Points How to integrate information about the global connectivity of the digital object ?

33. On the Characterization of Multisimple Points A point is multisimple if and only if: Necessary and Sufficient Condition for a Point to be Multisimple ? Simple characterization of multisimple points. ‘Low’ computational complexity.  deformations that do not generate/suppress any handles in the digital objects

34. On the Characterization of Multisimple Points Example

35. Computational complexity - Implementation using a grid of labels  assign a label to each component - local computations using the grid of labels  merging = assigning the same label  splitting = assigning different labels (using region growing algorithms) New sets of transformations under topology control - Preservation of the genus  Components can split/merge/appear/disappear without generating any handles On the Characterization of Multisimple Points Topology constrained Topology controlled with one single initial component Topology controlled with several initial components simple point non simple point Multisimple point

36. 1)Multisimple Points preserve the genus of a digital object 2) Level sets require an underlying digital image to be implemented  The negative/positive grid points of a level set define set of connected components of a digital object Apply the concept of multisimple point to level sets in order to design a Genus Preserving Level Set Framework Genus Preserving Level Sets

37. Level set evolution equation with outward normal speed β: What is the level set method ? Representing a manifold implicitly as the zero level set of a higher-dimensional function (Osher-Sethian, 1988)

38. + No parameterization + Easy computation of intrinsic geometric properties (e.g. normal, curvature) + Mathematical proofs and numerical stability + Topology changes handled automatically + No self-intersection in the resulting mesh - Computationally expensive  narrow band algorithm - Need for an isocontour extraction step  marching cubes algorithm Level sets vs. explicit representations Genus Preserving Level Sets Why using the level set methods ?

39. Genus Preserving Level Sets How to preserve the topology of level sets ? Topology Preserving Level Sets (Han, Xu, Prince, 2003) Algorithm: for each grid point at each time step, - Compute the new value of the level set - Sign is about to change? No: accept the new value Yes: check if the point is simple SIMPLE: accept the new value COMPLEX: set +/-ε as the new level set value Initial condition Traditional level sets Topological level sets + Local computations only  FAST  Strict Preservation of the topology  RESTRICTIVE

40. Algorithm: for each grid point at each time step, - Compute the new value of the level set - Sign is about to change? No: accept the new value Yes: check if the point is simple - SIMPLE: accept the new value - COMPLEX: set +/-ε as the new level set value Genus Preserving Level Sets Strict Topology Preservation Algorithm: for each grid point at each time step, - Compute the new value of the level set - Sign is about to change? No: accept the new value Yes: check if the point is simple - SIMPLE: accept the new value - COMPLEX: set +/-ε as the new level set value - Compute the set of neighboring connected components from the grid of labels L - Check if the point is multisimple MULTISIMPLE: accept the new value update the grid of labels L COMPLEX: set +/-ε as the new level set value

41. Genus Preserving Level Sets Experiments: Simple Segmentation Tasks withI(x) : intensity in the image at location x I thres :global intensity threshold H(x,t) : mean curvature of active contour at location x N(x,t) : normal of active contour at location x  : mixing parameter Data term Smoother surfaces

42. Genus Preserving Level Sets Traditional level setsTopology preserving level setsGenus preserving level sets Segmentation of a synthetic ‘C’ shape ÞTrade-off between traditional level-sets and topology preserving level-sets Þ Less sensitive to initial conditions

43. Genus Preserving Level Sets Segmentation of blood vessels from MRA Segmentation of blood vessels from MRA under two different initial conditions movie

44. Genus Preserving Level Sets Topology preserving level setsGenus preserving level sets OTHER EXAMPLES: square with cavitiessquare with cavities cortical segmentation initialized with 100 seeds

45. Advantages of Genus Preserving Level Sets Genus Preserving Level Sets Preserve the number of handles in the volume Components can be created or destroyed, can merge or split Less sensitive to initial conditions Less sensitive to noise/unexpected structures in the image ‘Low’ computational complexity (except for splits)

46. Section I: Contributions Novel Approaches in Digital Topology Concept of Multisimple Point New sets of digital transformations Genus Preserving Level Sets

47. Section II Manifold Surgery Topology Correction of Cortical Surfaces

48. Surfaces with the same Euler-characteristic are homeomorphic Manifold Surgery : Position of the problem Segmentation under topological constraints is a difficult problem: partial voluming effect, noise, bias field, image inhomogeneity, … many topological defects (handles) in the resulting cortical surface

49. GOAL: given a cortical surface C, detect and optimally correct the topological defects D

50. Manifold Surgery Approach Location of topological defects Optimal topology correction

51. Shrink-Wrap Methods cannot reach deep folds Quasi-Homeomorphic mapping to locate and correct the defects Manifold Surgery Manifold Surgery : Approach

52. Homeomorphic mapping M:C  S –Transformation that is continuous, one-to-one with continuous inverse M -1 –Jacobian is positive definite: J M > 0 –Jacobian is related to the areal distortion –For triangulations, the areal distortion is an approximation of the Jacobian Manifold Surgery Definition: Homeomorphism

53. Manifold Surgery Approach Location of topological defects Optimal topology correction

54. Cortical surface C with correct topology is homeomorphic to the sphere S : - continuous, one-to-one, continuous inverse - strictly positive Jacobian In the presence of topological defects (i.e. handles), no such mapping exists. Search for a mapping that minimizes the regions with negative Jacobian Quasi-homeomorphic mapping = maximally homeomorphic Manifold Surgery

55. Try to find a mapping that is maximally homeomorphic The areal distortion is an approximation of the Jacobian Quasi-Homeomorphic mapping Generate a mapping that minimizes the regions with negative area 1) Initialize the mapping by inflation and projection 2) Minimize E M by gradient descent Manifold Surgery

56. Location of topological defects different steps - inflation - projection - minimization of negative regions - defect = set of overlapping faces - back-projection Manifold Surgery

57. Location of topological defects Cortical surface with incorrect topology Topological defect Spherical projection Manifold Surgery

58. Approach Location of topological defects Optimal topology correction

59. The retessellation problem defect in original cortical surface corresponding spherical projection of defect quasi- homeomorphic mapping 1) Discard all faces and edges in each defect D 2) Find a valid retessellation T D for each defect D Manifold Surgery

60. Finding a valid retessellation T D for each defect D - Geometrically Accurate, i.e. smoothness, location, self-intersection Use Spherical Representation S Use Cortical Representation C - Geometrically Accurate, i.e. smoothness, location, self-intersection Manifold Surgery The retessellation problem

61. Constraining the Topology on S Discard edges and facesRetessellate Find a retessellation with no self- intersecting edges on the sphere  D = 1 Manifold Surgery

62. Many potential retessellations exist! S edges ={ …} ={e 0,e 1,…,e N } set of all potential edges where N=n(n-1)/2 and n is the number of vertices in the defect One candidate retessellation corresponds to an ordering O D of the set S edges Retessellating = iteratively adding edges Constraining the Topology on S Manifold Surgery

63. Retessellating = Knitting on the sphere Edge ordering O D ={ …} Different edge orderings (permutations) will generate different configurations in the original cortical space Constraining the Topology on S Manifold Surgery

64. Measuring the accuracy of T D on C No self-intersection Smoothness of T D MRI intensity I profile inside C - and outside C + Bayesian Framework Manifold Surgery

65. Prior = smoothness Likelihood = MRI intensity profile volume surface Measuring the accuracy of T D on C Manifold Surgery

66. Huge space to be searched! Discrete space  No gradient information Search : Genetic Algorithm defect with n vertices  N=n(n-1)/2 potential edges For a patch, v-e+f=1  # of used edges  3n Size of Space of potential retessellations  Manifold Surgery Use of a Genetic Algorithm to explore the space of potential retessellations O D

67. introduced in the 60s by John Holland as a way to import the mechanism of natural adaptation into computer algorithm and numerical optimization candidate solution = chromosome Fitness function = measure the evolutionary viability of each chromosome Genetic operations = mutations, crossovers  representation = edge ordering O D  posterior probability Search : Genetic Algorithm Manifold Surgery

68. Mutations : swap intersecting edges in O D with probability p mut Crossovers : randomly combine two orderings by iteratively adding edges from each of them Parameters : population = 20 chromosomes elite population = 40% - mutation = 30% - crossovers = 30% p mut = 0.1 – stop after 10 populations without any change Genetic Operations Manifold Surgery

69. Results: real MRI datasets applied to hundred of brains (part of FreeSurfer) typical brain segmentation = 40 defects (~50 vertices) average time = ~3 hours (~ 5 minutes / defect) validation on 35 brains (70 hemispheres) comparison with expert  Hausdorff distance < 0.2mm Manifold Surgery

70. Topological defect Corrected defect Results: real MRI datasets Manifold Surgery

71. Topological defect Corrected defect Initial cortical surface Sagital viewCoronal view Results: real MRI datasets Manifold Surgery

72. Topological defect Corrected defect Results: real MRI datasets Manifold Surgery

73. Convergence Genetic Search versus Random Search - Boost up average fitness of population -Speed up convergence by a log factor of 2 -Converge in few iterations Manifold Surgery

74. Section II: Contributions Manifold Surgery Method for optimally correcting the topology of cortical surfaces Genetic algorithm for solving the retessellation problem Manifold Surgery

75. CONCLUSION TO BE DONE… IT”S GREAT WORK !

77. Manifold Surgery Approach: retrospective topology correction of the cortical surface C orig MRIsegmentationtessellate surfacelocate defectscorrect defects skull strip spherical mapping

78. Discarding vertices on S Spherical mapping C S All vertices are included in the retessellation  jagged surfaces Solution : use the iterative retessellation to eliminate late vertices During the retessellation, vertices that are included inside temporary triangles are discarded from the tessellation

79. The space of all potential retessellations depends on the spherical mapping Limitations: the mapping problem! Spherical mapping C S Discard faces and edges impossible Retessellate

80. The mapping problem! Solution: Generate several mappings corresponding to different configuration - identify defects with large handles (geodesic distances) - cluster proximal vertices - generate several mappings using spring term force

81. The mapping problem! Example Topological defect Spherical representation Sagital view of topological defect

82. The mapping problem Example Topological defectCorrected defect Spherical representation

83. The mapping problem! Solution: Generate several mappings corresponding to different configuration - spherical location is not appropriate   circle in 2D

84. The mapping problem! - cluster proximal vertices in original space - generate different mappings using a spring term force

85. The mapping problem Original defectFinal configuration Example

86. Results: synthetic data