1. Image-guided therapy and medical robotics tutorial using a LEGO Mindstorms NXT robot and 3D Slicer Danielle F. Pace, B.CmpH Ron Kikinis, M.D. Nobuhiko Hata, Ph.D Surgical Planning Laboratory, Brigham and Women’s Hospital and Harvard Medical School

2. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -2- Funding Information This tutorial was made possible by grant numbers 5U41RR019703, 5P01CA067165 and 5U54EB005149 from the National Institutes of Health (NIH). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. This tutorial was also in part supported by National Science Foundation (NSF) grant number 9731748 and the Center for Integration of Medicine and Innovative Technology (CIMIT).

3. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -3- Thanks to Terry Peters, Ph.D (Robarts Research Institute, University of Western Ontario) Steven Canvin (The LEGO Group) Cory Walker (NXT++ Developer) G. Wade Johnson (Device::USB Developer) Steve Pieper, Ph.D (Surgical Planning Laboratory) Haiying Liu (Surgical Planning Laboratory) Junichi Tokuda, Ph.D (Surgical Planning Laboratory) Christoph Ruetz (Surgical Planning Laboratory) Philip Mewes (Surgical Planning Laboratory) The entire LEGO Mindstorms community

4. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -4- Goals of the LEGO IGT and medical robotics tutorial To emphasize Slicer3’s support for image- guided therapy (IGT) using medical robots To demonstrate the typical steps of an image- guided therapy or medical robotics procedure in a hands-on manner The example procedure that we will use to do this is a needle biopsy.

5. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -5- An Example of IGT in Action: Needle Biopsy In suspected cancer cases (such as for prostate or breast cancer) a needle biopsy is often performed as part of the diagnostic process. A needle is used to extract small pieces of tissue for analysis. Goal: Hit the tumour so that it can be detected! (image from R. Alterovitz, K. Goldberg and A. Okamura, Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, April 2005, pp. 1652-1657)

6. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -6- Medical Robotics Medical robots are increasingly popular for procedures such as needle biopsy in prostate cancer In medical robotics, we use the same steps as in IGT: imaging, planning, registration, tracking and navigation To the right is a prostate biopsy robot that operates within an MRI machine (being tested on a melon!) (JHU/SPL MRI prostate robot)

7. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -7- Tutorial Supplies One LEGO Mindstorms NXT robotics kit One LEGO Deluxe Brick Box Two pom-poms CT volume of the phantom 3D Slicer LEGO tutorial module A Linux computer with root access Two sheets of white paper and tape Assembly instructions for the robot and phantom Phantom placement guide

8. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -8- IGT Workflow z x y Step 1: Imaging Step 2: Planning Step 3: Registration Step 4: Tracking and Navigation (-22, 84, 41) z x y ICS PCS (image modified from S. Haker et al., Top Magn. Reson. Imaging 16(5), October 2005, pp. 355-368) (image from http://www.ncigt.org/ ourwork/core_mrguidedtherapy.html)

9. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -9- 1. Set up the Robot and Phantom Position the phantom relative to the LEGO robot such that: 1.The needle can reach both pom-pom targets 2.The registration pillars will be in view of the LEGO robot’s ultrasonic sensor

10. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -10- 2. Imaging - Load the CT Volume of the Phantom

11. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -11- 3. Connect Slicer to the robot Slicer3 connects to the robot via a USB cable

12. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -12- 4. Registration The position of the phantom relative to the LEGO robot is not known. We must determine the relationship between the image coordinate system (ICS) and the patient (robot) coordinate system (PCS). We must perform registration!

13. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -13- The registration procedure 1.Find the eight fiducials in the patient (robot) coordinate system (PCS) 2.Find the eight fiducials in the image coordinate system (ICS) 3.Use a landmark registration algorithm to determine the rigid transformation from the image coordinate system to the patient (robot) coordinate system: PCS 3x1 = R 3x3 ICS 3x1 + t 3x1

14. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -14- The eight fiducials 1 2 3 4 5 6 7 8 back front (towards robot)

15. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -15- Finding the fiducials in the PCS Two views of the patient (robot) coordinate system: z y x z y x

16. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -16- Finding the fiducials in the PCS The LEGO robot’s ultrasonic sensor detects distance to the nearest obstacle. We will find the phantom by using the robot’s ultrasonic sensor to find the “registration pillars” on the phantom. four “registration pillars”

17. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -17- Finding the fiducials in the PCS y x Use the ultrasonic sensor to find the registration pillars: the 4 angles shown Average the 4 angles to find  We know d because the robot’s needle can hit both pom-poms Use , d and knowledge of the phantom’s structure to find the eight fiducials on the phantom PCS  d

18. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -18- Finding the fiducials in the PCS y The robotic arm moves from left to right while polling the ultrasonic sensor for distance and the motor’s rotation sensor for the number of degrees This occurs twice - once for the “top” pillars and once for the “bottom” pillars.

19. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -19- Finding the fiducials in the PCS y Profiles of the number of degrees that the motor has turned vs. distance are generated for the top and the bottom swipes The local minima represent the registration pillars and are automatically detected

20. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -20- Finding the fiducials in the ICS Select the corresponding fiducials in the image coordinate system by clicking on the points on the CT volume.

21. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -21- 5. Preoperative Planning - select the target coordinate The target coordinate is the location where you would like the tip of the robot’s needle to be at the end of the “biopsy”. Let’s try to have the robot biopsy the red pom-pom. We need to find the center of the red pom- pom on the CT volume.

22. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -22- 5. Planning - select the target coordinate Click on the target coordinate: the center of the red pom- pom.

23. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -23- 6) Execute the biopsy The target in the patient (robot) coordinate system Transformation from the ICS to the PCS The final needle position in the patient (robot) coordinate system Tracking is done through targeting during the robot’s movement The display of the final needle position in the PCS provides Navigation

24. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -24- After completing this tutorial: You have seen an example of medical robotics within Slicer3 You have learned about the following steps in IGT and medical robotics in a hands-on manner: Imaging Preoperative planning Targeting and tracking Navigation Registration Tutorial website: http://wiki.na-mic.org/Wiki/index.php/LEGO_IGT_and_ Medical_Robotics_Tutorial http://wiki.na-mic.org/Wiki/index.php/LEGO_IGT_and_ Medical_Robotics_Tutorial

25. Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s Hospital August 2007 -25- Additional references Learn about image registration by reading: B. Zitová and J. Flusser. Image Registration Methods: A Survey. Image and Vision Computing, 21:977-1000, 2003. Brainstorm about the components of the total error that occur from the selection of the target to the final position of the robot needle tip. How could you measure those errors? Start by learning about error in registration by reading: J.B. West, M.J. Fitzpatrick, S.A. Toms, C.R. Maurer, and R.J. Maciunas. Fiducial Point Placement and the Accuracy of Point-based, Rigid Body Registration. Neurosurgery, 48(4):810-817, 2001.