1. Anatomical Sketch Understanding Peter Haddawy Computer Science & Information Management Asian Institute of Technology

2. Anatomical Sketching Sketching is ubiquitous in medicine –Patient records –Communication with patients –Consultation –Medical education, particularly PBL

3. COMET: Collaborative Medical PBL Tutor Suebnukarn & Haddawy, Clinical-Reasoning Skill Acquisition through Intelligent Group Tutoring, IJCAI-05.

4. Scenario Unconsciousness due to car accident It’s certainly possible to have a contusion in the frontal lobe, but what about damage to the brain stem? Discussion board Recognize what anatomical structure has been drawn and from what perspective Parietal view of the brain Identify anatomical parts of the structure, even if not explicitly drawn Frontal lobe of the brain Understand annotations on the sketch Arrow pointing from the label “contusion” to the frontal lobe Effectively use the sketch as a medium of communication in a dialog Recognize what anatomical structure has been drawn and from what perspective Identify anatomical parts of the structure, even if not explicitly drawn

5. Why sketch understanding? The informalness of sketches is important: –Conveys that the ideas are still rough and thus invites collaboration and modification. –Clean, precise-looking diagrams create the impression of more precision than was intended and lead to a feeling of commitment to the sketch as originally drawn. Sketching in medical PBL gives students practice in recalling anatomical structure.

6. The challenge of anatomical sketches Previous work (e.g. ASSIST, SILK) starts by recognizing primitive components –circles, lines, rectangles This approach works for assemblies of components but not for anatomical sketches Anatomical sketches are complex structures that may be drawn with greatly varying degrees of detail.

7. Sketch Recognition By convention, 2-D depictions of anatomy are drawn from one of eight standard views. Five external views, corresponding to the sides of a cube: –Anterior –Posterior –Superior –Inferior –Lateral (2 sides) Three internal views, corresponding to the cutting planes: –Sagittal –Coronal –Axial

8. Assumptions Sketch drawn from one of the eight standard views Sketch contains a single anatomical structure with no extraneous parts Sketch is complete: no major parts left out

9. UNAS Sketch Recognition Algorithm Sketch recognition becomes a classification problem: –Identify the anatomical structure and view Heart: anterior view Naïve Bayes classifier –Features: similarity between sketch and each of a set of templates –Similarity computed using Shape Context [Belongie, et al 2002]

10. Recognition Process Anatomical Structure & View … Template 1 Template m Naïve Bayes Classifier Lung External User Sketch Template 1 Template m … Shape Context Matching

11. Data Collection Collected sample sketches from 3 rd – 6 th year medical students 70 sketches of 6 anatomical structures, 2-3 views per structure: 1050 sketches

12. Constructing the Classifier 70% of the data for training –49 sketches per view Treat each sketch as a potential template Calculate 15 conditional Gaussians for each sketch –Distribution of shape context match over the other sketches conditional on the class (anatomical structure & view) Use the Wrapper approach [Kohavi, 1997] to select a small, accurate subset of templates –32 templates

13. Evaluation 30% of the data for testing –21 sketches per view Asked ten 3 rd - 6 th year medical students to classify each sketch Compared: Student accuracy vs UNAS accuracy

14. Evaluation: Accuracy BrainHeartLung ParietalSagittalBasalAnteriorPosteriorInteriorExternalInternal Students92.9 97.687.180.593.893.391.0 UNAS86.067.057.081.052.057.076.048.0 SkullStomachKidney Overall Accuracy (%) AnteriorParietalBasalExternalInternalExternalInternal Students99.098.697.695.793.397.196.793.81 UNAS76.081.052.076.067.052.067.066.30 Baseline random classification accuracy = 6.7%

15. Brain

16. Heart

17. Between internal & external viewsBetween internal & external views Between similarly shaped structuresBetween similarly shaped structures UNAS Common Errors Kidney ExternalKidney Internal Skull BasalBrain Basal

18. Sketch Segmentation Task: Identify explicit and implicit anatomical parts of the sketch Algorithm: –Find the best correspondence between the sketch and a pre-labeled canonical template (Shape Context) –Transfer labeled points to the sketch –Connect the points to form the parts

19. Matching sketch and template Simple Matching Sketch Template

20. Transformation Transform the template to fit the sketch –Thin-plate splines Then project labels from transformed template onto sketch Curve reconstruction (TSP [Giesen, 1999])

21. Evaluation Selected five qualitatively different sketches from two views of each of four structures –Brain : Basal, Parietal –Lung : External, Internal –Heart : Anterior, Posterior –Skull : Anterior, Parietal Asked a physician to segment the sketches Compared: Physician segmentations vs UNAS segmentations

22. Evaluation Segmentations of the system and those of the expert are very similar. Segmentations of the system have no missing or overlapping regions sketchsystemphysiciansketchsystemphysician

23. Evaluation: Disagreements The system is less tolerant of inaccurate proportions than the physician sketch systemphysician

24. Conclusions First application of sketch-based interfaces to intelligent tutoring First in a medical domain other than image retrieval Can recognize 15 views of 6 anatomical structures –Accuracy 66% vs baseline random classification accuracy of 6.7% System’s segmentations qualitatively similar to those of the physician

25. Ongoing & Future Work Improve recognition accuracy and speed Improve segmentation quality and speed Integrate with annotation understanding component Incorporate into COMET tutoring system (using UMLS) Sketches with partial and multiple anatomical structures Demos gladly given on request

26. Shape Context Represent a shape as a set points sampled from the shape’s contours Each point represented as a coarse histogram of the points surrounding it.

27. Shape Context Uniformly randomly sample points from the sketch and template. Compute the shape context descriptor for each point. Normalize by the mean squared distance between points to make scale invariant. The similarity between two shape context descriptors is just the χ 2 statistic. Find the least cost match between all points in the sketch and template. –Weighted bipartite graph matching –Hungarian method: O(N 3 )

28. Evaluation: Medical Student Accuracy

29. Evaluation: UNAS Accuracy

30. Skull

31. Lung

32. Evaluation: Disagreements The system’s segmentations are occasionally effected by excessive and unimportant contours in the sketch sketchsystemexpert