1. © Fraunhofer MEVIS Meeting of the Working Group VCBM, 3. September 2013, Vienna Frank Heckel Fraunhofer MEVIS, Bremen, Germany | Innovation Center Computer Assisted Surgery, Leipzig, Germany Sketch-Based Segmentation Editing for Oncological Therapy Monitoring

2. © Fraunhofer MEVIS 1 / 22 Why do we need efficient segmentation editing tools? Segmentation is one of the essential tasks in medical image analysis Many sophisticated automatic segmentation algorithms exist … … which might fail in some cases (low contrast, noise, biological variability) What to do? Manual segmentation?  Takes too long Different algorithm?  Might fail as well Locally correct the error!

3. © Fraunhofer MEVIS 2 / 22 The Segmentation Editing Process

4. © Fraunhofer MEVIS 3 / 22 What makes segmentation editing a difficult problem? Requirements: Intuitive interaction in 2D – estimate the user’s intention in 3D Local modifications Real-time feedback Provide a general tool (for different objects and modalities) Be independent of the preceding automatic algorithm The user expects the tool to allow him or her to correct all errors With only a few steps! The segmentation problems are typically hard

5. © Fraunhofer MEVIS 4 / 22 Editing Approaches Parameter tuning Direct interaction Moderate performance requirement High reproducibility Incorporate user information Indirect interaction Medium performance requirement Medium reproducibility Low-level manual correction Direct interaction Real-time performance requirement Low reproducibility High-level manual correction Indirect interaction Real-time performance requirement Medium reproducibility Guide initial algorithmDedicated editing tools

6. © Fraunhofer MEVIS 5 / 22 Sketch-Based Editing in 2D User inputCorrection resultEdited region Part containing the center of gravity

7. © Fraunhofer MEVIS 6 / 22 Sketch-Based Editing in 2D Possible Interactions add remove add + remove replace

8. © Fraunhofer MEVIS 7 / 22 Sketch-Based Editing in 3D The Correction Depth Add/Remove:Replace:

9. © Fraunhofer MEVIS 8 / 22 Sketch-Based Editing in 3D Image-Based Extrapolation

10. © Fraunhofer MEVIS 9 / 22 Sketch-Based Editing in 3D Image-Based Extrapolation s

11. © Fraunhofer MEVIS 10 / 22 Sketch-Based Editing in 3D Image-Independent Extrapolation

12. © Fraunhofer MEVIS 11 / 22 Sketch-Based Editing in 3D Previously performed corrections should be part of the new surface Keep all user-inputs and use them for reconstruction Major issue: Contradictory inputs Ignore constraints from previous contours within a specific range: Image-Independent Extrapolation 1 st input2 nd input

13. © Fraunhofer MEVIS 12 / 22 Sketch-Based Editing in 3D Image-Independent Extrapolation

14. © Fraunhofer MEVIS 13 / 22 Qualitative Evaluation Editing is a dynamic, user- driven process User studies are the most important tool for evaluation Quality of editing tools is highly subjective Subjective quality suffers from bad intermediate results (Vague) criteria: Accuracy Efficiency Repeatability visually performed by the user intended result that the user only has in mind

15. © Fraunhofer MEVIS 14 / 22 Qualitative Evaluation Rating Scheme for Accuracy and Efficiency sufficient insufficient

16. © Fraunhofer MEVIS 15 / 22 Qualitative Evaluation Results Editing rating score: 131 representative tumor segmentations in CT (lung nodules, liver metastases, lymph nodes) 5 radiologists with different level of experience

17. © Fraunhofer MEVIS 16 / 22 Quantitative Evaluation

18. © Fraunhofer MEVIS 17 / 22 Quantitative Evaluation Results

19. © Fraunhofer MEVIS 18 / 22 Simulation-Based Evaluation Problem: High effort and bad reproducibility of user studies Idea: Replace user by a simulation Benefits: Objective and reproducible validation Objective comparison Improved regression testing Better parameter tuning

20. © Fraunhofer MEVIS 19 / 22 Simulation-Based Evaluation Simulation Step 1: Find most probably corrected 3D error Step 2: Select slice and view where the error is most probably corrected Step 3: Generate user-input for sketching Step 4: Apply editing algorithm

21. © Fraunhofer MEVIS 20 / 22 Simulation-Based Evaluation Results

22. © Fraunhofer MEVIS 21 / 22 Takeaway Message The segmentation problem is not solved yet Segmentation editing is an indispensable step in the segmentation process Particularly for clinical routine, editing is a mandatory feature and not only nice to have Efficient editing in 3D is challenging and not much work has been done in this field so far Sketching has shown to be an intuitive interface in 2D The presented methods provide efficient 3D editing tools for tumor segmentation

23. © Fraunhofer MEVIS 22 / 22 What's next? Editing for complex objects with irregular shapes Support additional user inputs Combine image-based and image-independent approaches Further improve the evaluation (measures, simulation) Find the most efficient human-computer interfaces for editing

24. © Fraunhofer MEVIS frank.heckel@mevis.fraunhofer.de Thank you!