1. Surgical Planning Laboratory Brigham and Women’s Hospital Boston, Massachusetts USA a teaching affiliate of Harvard Medical School User Interfaces and Human- Computer Interaction Randy Ellis, Ph.D

2. ©2006 Surgical Planning Laboratory, ARR Slide 2 Control Loop: Classical View Command (control input) Sensor (state of WORLD) WORLD: an unknown dynamical system Action (effect on WORLD) Decision (difference between command and sensor)

3. ©2006 Surgical Planning Laboratory, ARR Slide 3 Control Loop: Surgeon’s View Command (control input) Sensor (screen update) INTERNALS: an unknown system (to the user) Action (updates internals) Computer: Integrates plans, sensors, etc.

4. ©2006 Surgical Planning Laboratory, ARR Slide 4 Driven by Requirements A requirement is a property that, if not present, means that the software is useless Functional requirements include: – Features, speed, order of actions, etc. Nonfunctional requirements include: – Appearance, documentation, etc. Technical constraints include: – Hardware, operating system, etc.

5. ©2006 Surgical Planning Laboratory, ARR Slide 5 Application-Oriented Architecture Base (common) software MRI Biopsy app.Neurosurgery app. Trans-Gastric app.

6. ©2006 Surgical Planning Laboratory, ARR Slide 6 Storyboards for Applications Define the basic approach to the interventional procedure Prototype screen appearances (paper cartoons or PowerPoint are useful) Specify screen-to-screen actions Implement “happy-day” scenario Elaborate alternative workflows (e.g., error handling, new image acquisitions) Result: fast development of well tested software Engineer and clinicians work together:

7. ©2006 Surgical Planning Laboratory, ARR Slide 7 Technical Overview Requirements drive the process Application-specific: – Rapid specification of ideal workflow – Catch design problems early – Elaborate necessary extensions only, not all possible variants Test-oriented: – The specification produces tests before code is written – Repeat all tests whenever changes are made Result: robust, trustable systems

8. ©2006 Surgical Planning Laboratory, ARR Slide 8 Example Application: CT-Guided Brain Tumor Biopsy

9. ©2006 Surgical Planning Laboratory, ARR Slide 9 Clinical Problem: Biopsy in CT Scanner Idea: Perform brain biopsy using 2D slices Technique: Attach fiducials to head holder Scan holder and patient Select slices showing structures at risk Superimpose needle trajectory on slices Verify biopsy with new CT acquisition(s)

10. ©2006 Surgical Planning Laboratory, ARR Slide 10 CT Biopsy Phase 1: Procedure Workflow 1.Prepare patient (including fiducials) 2.Transfer patient to CT scanner 3.Acquire the CT scan (?inject?, image) 4.Load scan into Navigator 5.Software calculates registration 6.Software displays needle trajectory in CT 7.Perform the biopsy 8.Close the patient

11. ©2006 Surgical Planning Laboratory, ARR Slide 11 CT Biopsy Phase 2: Software Actions 1.Load instrument descriptions 2.Load CT scan 3.Detect fiducials 4.Calculate registration 5.Select key CT slices 6.Track the needle guide 7.Overlay needle trajectory on slices

12. ©2006 Surgical Planning Laboratory, ARR Slide 12 CT Biopsy Phase 3: Procedure Steps 1.Screen to select scan from disk 2.Screen to give registration progress 3.Screen to select CT slices 4.Screen to display navigation Additional Requirements: – Forward/backward buttons – Exit button – Tool status display (e.g., visibility)

13. ©2006 Surgical Planning Laboratory, ARR Slide 13 CT Biopsy Phase 4: State Transitions File Screen CT scan found & loaded Registration Screen Fiducials detected & computed Selection Screen Slices selected by clinician Navigation Screen

14. ©2006 Surgical Planning Laboratory, ARR Slide 14 LOGO GOES HERE EXIT Tool Status Display CT Biopsy Storyboard: File Selection All we need here is a simple dialog box

15. ©2006 Surgical Planning Laboratory, ARR Slide 15 LOGO GOES HERE EXIT Tool Status Display CT Biopsy Storyboard : Registration Need a progress bar during registration, then report RMS error after registration

16. ©2006 Surgical Planning Laboratory, ARR Slide 16 LOGO GOES HERE EXIT Tool Status Display CT Biopsy Storyboard: Selection Idea #1 Show original slices in here SELECT SLICE 1/2/3/4  Scroll bar 

17. ©2006 Surgical Planning Laboratory, ARR Slide 17 LOGO GOES HERE EXIT Tool Status Display CT Biopsy Storyboard: Selection Idea #2 Show original slices in here Drag/Drop into here  Scroll bar 

18. ©2006 Surgical Planning Laboratory, ARR Slide 18 LOGO GOES HERE EXIT Tool Status Display CT Biopsy Storyboard: 2D Navigation Slice #1 Slice #3 Slice #2 Slice #4 Show the needle trajectory on top of each slice

19. ©2006 Surgical Planning Laboratory, ARR Slide 19 CT Biopsy Phase 5: Implement and Test 1.Create scans with/without fiducials (to test registration algorithms) 2.Build phantoms to measure navigation accuracy 3.Power cycle computer to test system robustness Unplug/plug tracking system, etc.

20. ©2006 Surgical Planning Laboratory, ARR Slide 20 CT Biopsy Phase 6: Next Iteration Next round, we may need to: Calibrate the biopsy needle intraoperatively Navigate the biopsy needle Use needle-oriented reslicing Migrate system to use MRI scans (new application: software re-use) Control MRI scanner for in-scanner biopsy (new application: software re-use)

21. ©2006 Surgical Planning Laboratory, ARR Slide 21 Example Application: Ventriculoscopy for Brain Tumor Biopsy

22. ©2006 Surgical Planning Laboratory, ARR Slide 22 Clinical Problem: Deep Brain Biopsy Idea: Use flexible scope to biopsy from within a brain ventricle Technique: Same setup as for CT biopsy navigation Show 3D location of scope inside volumetric image Needs calibration of optical image to tracking device

23. ©2006 Surgical Planning Laboratory, ARR Slide 23 Deep-Biopsy Phase 1: Procedure Workflow 1.Prepare patient (including fiducials) 2.Transfer patient to CT or MRI scanner 3.Acquire the 3D scan (inject & image) 4.Load scan into Navigator 5.Software calculates registration 6.Calibrate ventriculoscope 7.Software displays scope in volumetric scan 8.Perform the biopsy 9.Close the patient

24. ©2006 Surgical Planning Laboratory, ARR Slide 24 Deep-Biopsy Phase 2: Software Actions 1.Load instrument descriptions 2.Load volumetric (MRI/PET/CET) scan 3.Detect fiducials 4.Calculate registration 5.Calibrate ventriculoscope to tracker device 6.Confirm calibration using a sharp probe 7.Track the ventriculoscope 8.Show scope in volumetric (MRI/PET/CET) scan

25. ©2006 Surgical Planning Laboratory, ARR Slide 25 Deep-Biopsy Phase 3: Procedure Steps 1.Screen to select scan from disk 2.Screen to give registration progess 3.Screen to calibrate ventriculoscope 4.Screen to verify calibration 5.Screen to display navigation Additional Requirements: – Forward/backward/exit buttons – “Tabs” to other tasks  label “forward”

26. ©2006 Surgical Planning Laboratory, ARR Slide 26 Deep-Biopsy Phase 4: State Transitions File Screen MRI/PET/CT scan found & loaded Registration Screen Fiducials detected & computed Calibration Screen Calibration accepted by surgeon Verification Screen Calibration verified by surgeon Navigation Screen

27. ©2006 Surgical Planning Laboratory, ARR Slide 27 LOADCALIBRATEVERIFYNAVIGATEREGISTER LOGO GOES HERE EXIT Deep-Biopsy Storyboard: File Selection All we need here is a simple dialog box ACCEPT

28. ©2006 Surgical Planning Laboratory, ARR Slide 28 LOADCALIBRATEVERIFYNAVIGATEREGISTER LOGO GOES HERE EXIT Deep-Biopsy Storyboard: Registration Need a progress bar during registration, then report RMS error after registration ACCEPT

29. ©2006 Surgical Planning Laboratory, ARR Slide 29 LOADCALIBRATEVERIFYNAVIGATEREGISTER LOGO GOES HERE EXIT Deep-Biopsy Storyboard: Calibrate Vscope Show live video feed in here CALIBRATE

30. ©2006 Surgical Planning Laboratory, ARR Slide 30 LOADCALIBRATEVERIFYNAVIGATEREGISTER LOGO GOES HERE EXIT Deep-Biopsy Storyboard: Verification 3D probe model on top of live video feed here ACCEPT

31. ©2006 Surgical Planning Laboratory, ARR Slide 31 LOADCALIBRATEVERIFYNAVIGATEREGISTER LOGO GOES HERE EXIT Deep-Biopsy Storyboard: 3D Navigation Live video feed from scope Scope field-of- view cone inside volume Forward rendering from scope

32. ©2006 Surgical Planning Laboratory, ARR Slide 32 Deep-Biopsy Phase 5: Implement and Test 1.Create scans with/without fiducials (to test registration algorithms) 2.Calibrate optics to tracker device 3.Verify calibration with sharp probe 4.Build phantoms to measure navigation accuracy 5.Power cycle computer to test system robustness Unplug/plug tracking system, etc.

33. ©2006 Surgical Planning Laboratory, ARR Slide 33 Deep-Biopsy Phase 6: Next Iteration Next round, we may need to: Calibrate the biopsy needle Navigate the biopsy needle Use needle-oriented reslicing Acquire/integrate intraoperative MRI (new application: software re-use) Migrate to laparoscopic ultrasound (new application: software re-use)