ProbExplorer: Uncertainty-guided Exploration and Editing of Probabilistic Medical Image Segmentation Ahmed Saad 1,2, Torsten Möller 1, and Ghassan Hamarneh 2 1 Graphics, Usability, and Visualization (GrUVi) Lab, 2 Medical Image Analysis Lab (MIAL), School of Computing Science, Simon Fraser University, Canada

Ahmed Saad ProbExplorer Outline Medical image segmentation challenges ProbExplorer framework Case studies – Highlight suspicious regions (e.g. tumors) – Correct misclassification results Uncertainty visualization using shape and appearance prior information Conclusion and future work 2

Ahmed Saad ProbExplorer Medical image segmentation Partitioning the image into disjoint regions of homogeneous properties Useful for statistical analysis, diagnosis, and treatment evaluation Medical Image Segmentation 3

Ahmed Saad ProbExplorer Segmentation challenges Low signal-to-noise ratio Partial volume effect Anatomical shape variability Multi-dimensional data Magnetic Resonance Imaging Positron Emission Tomography 4

Ahmed Saad ProbExplorer Segmentation challenges Low signal-to-noise ratio Partial volume effect Anatomical shape variability Multi-dimensional data 5

Ahmed Saad ProbExplorer Segmentation challenges Low signal-to-noise ratio Partial volume effect Anatomical shape variability Multi-dimensional data 6 Patient 1 Patient 2Patient 3Patient 4

Ahmed Saad ProbExplorer Segmentation challenges Low signal-to-noise ratio Partial volume effect Anatomical shape variability Multi-dimensional data 4D CTdPET DTMRI 7

Ahmed Saad ProbExplorer Segmentation output CrispProbabilistic (Fuzzy) 70% 20% 10% Putamen White matter Grey matter Putamen 8 Max

Ahmed Saad ProbExplorer Outline Medical image segmentation challenges ProbExplorer framework Case studies – Highlight suspicious regions (e.g. tumors) – Correct misclassification results Uncertainty visualization using shape and appearance prior information Conclusion and future work 9

Ahmed Saad ProbExplorer Goal Given probabilistic segmentation results, we will allow expert users to visually examine regions of segmentation uncertainty to – Highlight suspicious regions (e.g. tumors) – Correct misclassification results without re- running the segmentation 10

Ahmed Saad ProbExplorer Preprocessing Selecting voxels Editing Probabilistic segmentation Change selection Commit an editing action 11

Ahmed Saad ProbExplorer Preprocessing Selecting voxels Editing Probabilistic segmentation Change selection Commit an editing action 12

Ahmed Saad ProbExplorer Preprocessing Selecting voxels Editing Probabilistic segmentation Change selection Commit an editing action 13

Ahmed Saad ProbExplorer Preprocessing Selecting voxels Editing Probabilistic segmentation Change selection Commit an editing action 14 BeforeAfter

Ahmed Saad ProbExplorer Preprocessing A probabilistic vector field Sort 15

Ahmed Saad ProbExplorer Outline Medical image segmentation challenges ProbExplorer framework Case studies – Highlight suspicious regions (e.g. tumors) – Correct misclassification results Uncertainty visualization using shape and appearance prior information Conclusion and future work 16

Ahmed Saad ProbExplorer Renal dynamic SPECT 4D image of size 64 x 64 x 32 with 48 time steps with an isotropic voxel size of (2 mm) 3 17 Raw data Crisp segmentation

Ahmed Saad ProbExplorer Uncertainty interaction overview widget 18   ?

Ahmed Saad ProbExplorer Selection of normal behavior 19

Ahmed Saad ProbExplorer Selection of abnormal behavior 20

Ahmed Saad ProbExplorer Outline Medical image segmentation challenges ProbExplorer framework Case studies – Highlight suspicious regions (e.g. tumors) – Correct misclassification results Uncertainty visualization using shape and appearance prior information Conclusion and future work 21

Ahmed Saad ProbExplorer Uncertainty-based segmentation editing 22 Ground truthOverestimationUnderestimation

Ahmed Saad ProbExplorer Synthetic example 23 No noise no PVE Ground truth Observed = noise + PVE Current segmentation

Ahmed Saad ProbExplorer Synthetic example: push action Push action Source set Destination set 24

Ahmed Saad ProbExplorer Synthetic example: push action 25 is the first best guess Swap

Ahmed Saad ProbExplorer Dynamic PET brain 26

Ahmed Saad ProbExplorer Overestimated putamen 27 Ground truth Overestimated Putamen

Ahmed Saad ProbExplorer Uncertainty interaction overview widget 28

Ahmed Saad ProbExplorer Dynamic PET brain 29

Ahmed Saad ProbExplorer Dynamic PET brain Push action Putamen Background Skull Grey matter Cerebellum Source set Destination set 30

Ahmed Saad ProbExplorer Dynamic PET brain 31 After 2 editing actions

Ahmed Saad ProbExplorer More (Saad et al., EuroVis10) 32 Selection

Ahmed Saad ProbExplorer Outline Medical image segmentation challenges ProbExplorer framework Case studies – Highlight suspicious regions (e.g. tumors) – Correct misclassification results Uncertainty visualization using shape and appearance prior information Conclusion and future work 33

Ahmed Saad ProbExplorer Bayesian perspective LikelihoodPriorPosterior 34

Ahmed Saad ProbExplorer Framework Atlas construction Shape prior Likelihood Appearance prior Likelihood Images Expert binary segmentations Probabilistic shape prior Probabilistic appearance prior Population representative image New image New probabilistic segmentation Image-to-Image registration Aligned likelihood 35

Ahmed Saad ProbExplorer Mathematical notations 36

Ahmed Saad ProbExplorer Algorithm demonstration using synthetic example Piecewise constantBlurring Noise 100 noise realizations and random translations 37

Ahmed Saad ProbExplorer Atlas construction: Shape prior modeling 38

Ahmed Saad ProbExplorer Atlas construction: Shape prior modeling 39

Ahmed Saad ProbExplorer Atlas construction: Shape prior modeling 40

Ahmed Saad ProbExplorer Atlas construction: Appearance prior modeling 41

Ahmed Saad ProbExplorer Mixture of Gaussians Other probabilistic segmentation techniques can be used, e.g. Random walker, Probabilistic SVM, etc. Likelihood 42

Ahmed Saad ProbExplorer Abnormal cases 43

Ahmed Saad ProbExplorer Abnormal shape DataMaximum likelihood Selection 44

Ahmed Saad ProbExplorer Abnormal shape Data Selection Maximum likelihood 45

Ahmed Saad ProbExplorer Abnormal appearance Data Selection Maximum likelihood 46

Ahmed Saad ProbExplorer Abnormal shape and appearance Data Selection Maximum likelihood 47

Ahmed Saad ProbExplorer Misclassification correction Dice: 0.32Dice:

Ahmed Saad ProbExplorer More (Saad et al., IEEEVis10) 49

Ahmed Saad ProbExplorer User evaluation Our clinical collaborators showed how ProbExplorer can be used to achieve highly accurate segmentation from a very noisy dSPECT renal study (Humphries et al. IEEE Nuclear Science Symposium/Medical Image Conference 2009) 50

Ahmed Saad ProbExplorer Conclusion ProbExplorer: a framework for the analysis and visualization of probabilistic segmentation results We provided a number of visual data analysis widgets to reveal the different class interactions that are usually hidden by a simple crisp visualization 51

Ahmed Saad ProbExplorer Future work Spatial dependency between voxels during interactive editing Investigate the behavior of the resulting probabilistic results from different segmentation techniques Multi-structure atlas Registration uncertainty visualization 52

Ahmed Saad ProbExplorer Acknowledgements Natural Sciences and Engineering Research Council of Canada (NSERC) Prof. Vesna Sossi, Prof. Anna Celler, Thomas Humphries, and Prof. Manfred Trummer 53

54 Ahmed Saad