Sketch-Based Interactive Segmentation and Segmentation Editing for Oncological Therapy Monitoring Frank Heckel March 17, 2015 BVM-Award 2015 – PhD Thesis –

2 / 22 Medical Background  Change in tumor size is an important criterion for assessing the success of a chemotherapy  RECIST 1 1.1: Sum of maximum diameters of target lesions  Relative change  Volume is a more accurate measure  Many tumors grow/shrink irregularly in 3D  Requires appropriate segmentation Oncological Therapy Response Monitoring 1 RECIST: Response Evaluation Criteria In Solid Tumors

3 / 22 The Segmentation Problem  Ultimate Goal: Automatic segmentation  Reproducible results with no effort for the user  Solutions for specific purposes  Might fail (low contrast, noise, biological variability)  Unsolved or insufficient for many real-world problems  Solutions:  Manual segmentation  Interactive tools  Automatic segmentation + manual correction  Drawbacks:  Higher effort  Lower reproducibility

4 / 22 Interactive Segmentation  Based on common 2D user interaction: drawing contours  Segmentation as an object reconstruction problem  Energy-minimizing surface reconstruction from a point cloud based on RBFs  3D surface based on contours from a few slices in arbitrary orientations Variational Interpolation

5 / 22 Interactive Segmentation  Computation time optimization  Shape preserving constraint reduction  Parallelization  Robustness improvement  Approximation instead of interpolation for resolving contradictions  Detection and consideration of self-intersection points Main Challenges

6 / 22 Interactive Segmentation  Computation time: Speedup ≈80  Evaluation:  Data: 15 liver metastases, 1 liver  Participants: 2 experienced radiology technicians Results 1 CLAPACK, 1 thread, no reduction 2 MKL, 4/8 threads, reduction by ≈80%

7 / 22 Segmentation Editing  Most existing methods are low-level and unintuitive in 3D  High-level correction has not received much attention in research

8 / 22 Segmentation Editing Sketch-Based Editing in 2D add remove add + remove replace

9 / 22 Segmentation Editing The Correction Depth

10 / 22 Segmentation Editing  Sample user contour into reference points  Move reference points to next slice using a block matching  Connect seed points using a shortest-path algorithm Image-Based 3D Extrapolation

11 / 22 Segmentation Editing  Utilizes the RBF-based interpolation approach  Reconstruct the new segmentation with contours in the edited slice and a start / end slice given by the correction depth  Restrict the new segmentation to the edited region Image-Independent 3D Extrapolation

12 / 22 Evaluation of Editing Tools  131 representative tumor segmentations in CT (lung nodules, liver metastases, lymph nodes)  5 radiologists with different level of experience  Editing rating score: Qualitative Evaluation

13 / 22 Evaluation of Editing Tools Quantitative Evaluation

14 / 22 Evaluation of Editing Tools  Problem: High effort and bad reproducibility of user studies  Idea: Replace user by a simulation  Benefits:  Objective and reproducible validation  Objective comparison  Improved regression testing  Better parameter tuning Simulation-Based Evaluation

15 / 22 Evaluation of Editing Tools  Step 1: Find most probably corrected 3D error  Step 2: Select slice and view where the error is most probably corrected  Step 3: Generate user-input for sketching  Step 4: Apply editing algorithm Simulation-Based Evaluation

16 / 22 Evaluation of Editing Tools Simulation-Based Evaluation

17 / 22 Partial Volume Correction  Smoothing effect caused by limited spatial resolution (of CT)  Ill-defined border between tumor and healthy tissue, making segmentation an ill-defined problem  Could cause significant differences in size measurements The Partial Volume Effect 28.4 ml (-27.5%) 39.2 ml 56.8 ml (+44.9%)

18 / 22 Partial Volume Correction Method ml 70.8 ml

19 / 22 Partial Volume Correction Software Phantom Results

20 / 22 Partial Volume Correction Hardware Phantom Results

21 / 22 Partial Volume Correction Multi-Reader Data Results

22 / 22 Summary  Contributions:  General image-independent interactive segmentation method  Efficient and intuitive segmentation editing tools + methodologies for their evaluation  Fast algorithm for compensation of partial volume effects  Future Work:  Improve algorithms for irregular and large objects  Combine image-based and image-independent editing  Make editing simulation more realistic  HCI aspects in editing  4D and multi-label segmentations  Establish volumetric measurements in clinical routine

Acknowledgement Thanks to all colleagues at (Fraunhofer) MEVIS, particularly Dr. Jan Moltz, Lars Bornemann, Dr. Hans Meine, Dr. Stefan Braunewell, Dr. Markus Lang, Michael Schwier, Dr. Volker Dicken, Dr. Benjamin Geisler, Olaf Konrad, Wolf Spindler and Prof. Horst Hahn. Special thanks to Dr. Christian Tietjen, Dr. Grzegorz Soza, Andreas Wimmer, Dr. Ola Friman, Prof. Bernhard Preim, Prof. Andreas Nüchter, all clinical partners and the Visual Computing in Biology and Medicine community. An finally, my wife and my children!

Thank you! Bei Herausforderungen geht es nicht ums Gewinnen, sondern darum, herauszufinden, was für ein Mensch man ist.