1. Towards Robust Medical Image Segmentation Shaoting Zhang CBIM Center Computer Science Department Rutgers University

2. Introduction Background Segmentation (finding 2D/3D region-of-interest) is a fundamental problem and bottleneck in many areas. We focus on learning-based deformable models with shape priors (2D contour or 3D mesh). Chest X-ray Lung CAD Liver in whole- body CT (PET-CT) Rat brain structures in MR Microscopy 2

3. Introduction Background End-to-end, automatic, accurate, efficient. Robustness – Handle weak or misleading appearance cues. – Handle diseased cases (e.g., with tumor/cancer). – Leverage shape priors to improve the robustness (Active Shape Model, T. Cootes, CVIU95; 3D ASM for cardiac segmentation, Y. Zheng, TMI08) Segmentation system 3 Liver shape variations [Springer images]

4. Introduction Research Void Limitations of existing shape prior methods: – Assume Gaussian errors Sensitive to outliers – Assume unimodal distribution of shapes Cannot handle large shape variations, e.g., multimodal – Only keep major variation Lose local shape detail 4

5. Introduction Research Void Handling gross errors or outliers. – RANSAC + ASM [M. Rogers, ECCV02] – Robust Point Matching [J. Nahed, MICCAI06] Handling multimodal distribution of shapes. – Mixture of Gaussians [T. F. Cootes, IVC97] – Manifold learning for shape prior [Etyngier, ICCV07] – Patient-specific shape [Y. Zhu, TMI10] Preserving local shape details. – Sparse PCA [K. Sjostrand, TMI07] – Hierarchical ASM [D. Shen, TMI03] 5 Need to solve all three challenges simultaneously in practice

6. Methods Segmentation framework 6 Landmark Detectors Offline Learning Boundary Detectors Runtime Segmentation New Image Model Initialize Iterative Deformation and Shape Refinement Final Results Shape Priors via Sparse Shape Representation Image Data and Manually Labeled Ground Truths Critical anatomical landmarks.

7. Methods Segmentation framework Initialization: Landmark detectors + shape prior – Automatic, fast (AdaBoosting/PBT/random forest + 3D Haar). Deformation: Boundary detectors + shape prior – Extend the landmark detectors to locate the sub-surfaces. Critical anatomical landmarks. 7 Lung Heart Rib Colon Abdomen 7

8. Methods Shape prior using sparse shape representation Our shape prior is based on two observations: – An input shape can be approximately represented by a sparse linear combination of training shapes. – The given shape information may contain gross errors, but such errors are often sparse.... 8

9. Methods Shape prior using sparse shape representation Formulation: – Sparse linear combination: –...... Dense x Sparse x Aligned shape data matrix D Weight x Input y Global transformation parameter Number of nonzero elements Global transformation operator 9

10. Methods Shape prior using sparse shape representation Non-Gaussian errors: – 10

11. Methods Shape prior using sparse shape representation Why it works? – Robust: Explicitly modeling e with L0 norm constraint. Thus it can detect gross (sparse) errors, i.e., non-Gaussian – General: No assumption of any parametric distribution model (e.g., a unimodal distribution assumption in ASM). Thus it can model large shape variations. – Lossless: It uses all training shapes. Thus it is able to recover detail information even if the detail is not statistically significant in training data. 11

12. Methods Optimization Convex relaxation [Candes, Tao, T-IT06] : Tuning parameters (λ 1 and λ 2 ) are not difficult to choose, compared to k 1 and k 2. – λ 1 x 1, λ 1 controls the sparsity of x. – λ 2 e 1, λ 2 controls the sparsity of e. One group of parameters work well for all images. Efficient convex optimization: fast proximal gradient methods (e.g., FISTA) 12Shaoting Zhang

13. Methods Scalability 13Shaoting Zhang...... Dictionary size = 64#Coefficients = 1000#Input samples = 1000 1. To efficiently handle many training samples - dictionary learning: 2. To efficiently handle thousands of vertices - mesh partitioning:

14. Applications – Part I 2D lung localization in X-ray (Lung computer-aided diagnosis system, Siemens) Handling gross errors Detection PA ASM RASM NN TPS Sparse1 Sparse2 Procrustes analysis Active Shape Model Robust ASMNearest Neighbors Thin-plate-spline Without modeling e Proposed method 14

15. Applications – Part I 2D lung localization in X-ray Multimodal shape distribution Detection PA ASM/RASM NN TPS Sparse1 Sparse2 15

16. Applications – Part I 2D lung localization in X-ray Recover local detail information Detection PA ASM/RASM NN TPS Sparse1 Sparse2 16

17. Applications – Part I 2D lung localization in X-ray Sparse shape components ASM modes: 17 0.5760 ++ 0.21560.0982

18. Applications – Part I 2D lung localization in X-ray Mean values and standard deviations. ~1,000 cases. Left lung Right lung 1)PA, 2)ASM, 3)RASM, 4)NN, 5)TPS, 6)Sparse1, 7)Sparse2 µ σ 18

19. Applications – Part II 3D liver segmentation in low-dose CT 19 Use case: Improved interpretation of PET-CT, with Siemens – PET: Measure functional process (metabolic activity), oncology – CT: Provide anatomical information, co-registered with PET – Issues: High variations of activities across different organs – Solution: Segment organs in CT for organ-specific interpret. – Challenge: Low dose results in low contrast & fuzzy boundary

20. Applications – Part II 3D liver segmentation in low-dose CT Procrustes analysis Sparse shapeGround truth Initialization Deformation 20 Same landmarks + different shape priors Same deformation module

21. Applications – Part II 3D liver segmentation in low-dose CT Procrustes analysis Sparse shape 21

22. Applications – Part II 3D liver segmentation in low-dose CT Procrustes analysis Sparse shape 22

23. Applications – Part II 3D liver segmentation in low-dose CT Procrustes analysis Sparse shape 23

24. Applications – Part II 3D liver segmentation in low-dose CT Procrustes analysis Sparse shape 24

25. Applications – Part II 3D liver segmentation in low-dose CT ASM-type [Zhan09] Sparse shapeGround truth Shape refinement during segmentation 25

26. Applications – Part II 3D liver segmentation in low-dose CT Quantitative results: surface distances. ~80 cases Mean value and standard deviation (voxel) 26

27. Applications – Part III Segmentation of Multiple Brain Structures in MRI Rat brain structures Drug addiction and alcoholism (with Brookhaven National Lab) Human brain, 34 structures Alzheimer's disease (with GE Global Research) 27

28. Summary of Robust Segmentation Robustly handle abnormal cases, such as diseased cases (liver tumor). Critical to healthcare applications such as computational diagnosis systems. Patent with Siemens. Used in several clinical applications. Key contribution for our awarded NSF-MRI grant (12-16). Relevant publications: – First author papers MICCAI 2012, 2011 (MICCAI Young Scientist Award Finalist) CVPR 2011 Two journal papers in Medical Image Analysis – Second author paper Medical Physics 2013 (with my co-mentored student, G. Wang) ISBI 2013, oral (with my co-mentored student, Z. Yan) 28