1. Planning in surgery and surgical simulation Comp 790-058 course presentation Mert Sedef

2. Planning in robotic radiosurgery R. Tombropoulos, J.R. Adler, and J.C. Latombe. CARABEAMER: A Treatment Planner for a Robotic Radiosurgical System with General Kinematics. Accepted for publication in Medical Image Analysis, Oxford University Press, 1998. Rhea Tombropoulos et al., Treatment Planning for Image-Guided Robotic Radiosurgery, Computer Vision, Virtual Reality and Robotics in Medicine, 1997 R.Z. Tombropoulos, J.C. Latombe, and J.R. Adler. A General Algorithm for Beam Selection in Radiosurgery. In Preprints of the IARP Workshop on Medical Robotics, 91--98, Vienna, Austria, 1996. A. Schweikard, J.R. Adler, and J.C. Latombe. Motion Planning in Stereotaxic Radiosurgery. IEEE Tr. on Robotics and Automation, 9(6):764--774, 1993

3. Radiosurgery Non-invasive procedure Moving beam of radiation to ablate (destroy) brain tumors The problem is delivering Enough dose of radiation to the tumor to destroy the tumor Enough dose of radiation to the tumor to destroy the tumor Minimum dose of radiation to the healthy and dose-sensitive tissue (e.g., brain stem and optic nerves) not to destroy them Minimum dose of radiation to the healthy and dose-sensitive tissue (e.g., brain stem and optic nerves) not to destroy them The solution is Crossfiring at the tumor: several weaker beams from different directions Crossfiring at the tumor: several weaker beams from different directions Tombropoulos, Adler, and Latombe, 1998

4. Treatment planning in radiosurgery Determination of a series of beam configuration (position and orientation) Constraints: The beams should intersect to form a region of high-dose on the tumor The beams should intersect to form a region of high-dose on the tumor The dose distribution should match the shape of the tumor The dose distribution should match the shape of the tumor Healthy or critical tissues should get minimum or no radiation Healthy or critical tissues should get minimum or no radiation

5. A treatment planning system 6-dof robotic manipulator arm Positions the radiation source Positions the radiation source Real-time imaging system Monitors patient’s motion continuously Monitors patient’s motion continuously A treatment planning algorithm Allows the surgeon to specify particular region of interest (e.g., tumors, dose-sensitive tissue) and range of dose Allows the surgeon to specify particular region of interest (e.g., tumors, dose-sensitive tissue) and range of dose Uses linear programming to optimize the plans and satisfy constraints Uses linear programming to optimize the plans and satisfy constraints Tombropoulos, Adler, and Latombe, 1998

6. Steps of the treatment planning system - 1 The surgeon specifies regions of interest on the CTs (e.g., the tumor and critical structures) the system makes a 3D reconstruction of the geometry the system makes a 3D reconstruction of the geometry and imposes constraints on the amount of radiation that these regions should receive. Eg., Tumor should get 2000 rads min and brain stem should get 500 rads max Eg., Tumor should get 2000 rads min and brain stem should get 500 rads max Tombropoulos, Adler, and Latombe, 1998

7. Beam selection Target point selection: Evenly space targets on the surface of the 3D tumor model coming from the CT Target point selection: Evenly space targets on the surface of the 3D tumor model coming from the CT Source point selection: Select source points making use of pre-recorded robot configurations. Record the target point and robot configuration. Source point selection: Select source points making use of pre-recorded robot configurations. Record the target point and robot configuration. Path generation: Connect all beam configurations into a path such that the robot traverses in a collision-free path in the environment. Path generation: Connect all beam configurations into a path such that the robot traverses in a collision-free path in the environment. Steps of the treatment planning system - 2

8. Plan refinement Problem! Beam selection does not consider the location of critical tissues and does not guarantee a highly homogeneous dose distribution on the tumor Problem! Beam selection does not consider the location of critical tissues and does not guarantee a highly homogeneous dose distribution on the tumor Given these constraints, Linear Programming adjusts and finds the optimal values of the dose and diameter of individual beams. Given these constraints, Linear Programming adjusts and finds the optimal values of the dose and diameter of individual beams. Steps of the treatment planning system - 3

9. Plan Evaluation The surgeon is provided with the results of planning The surgeon is provided with the results of planning 3D iso-dose surfaces, dose-volume histograms, etc. If the surgeon is not satisfied, planning is restarted from the desired step If the surgeon is not satisfied, planning is restarted from the desired step Steps of the treatment planning system - 4

10. Motion planning in maxillofacial robotic surgery Burghart et al., On-line motion planning for medical applications, Proceedings of the 24th Annual Conference of the IEEE, 1998

11. Maxillofacial robotic surgery Maxillofacial surgery: Surgery in the maxilla and face area Motion of the surgical robot should be planned for Bone cutting Bone cutting Planned motion should be safe and adequate be safe and adequate Have online capabilities to react dynamical changes (i.e. movements of the patient and surgical instruments) Have online capabilities to react dynamical changes (i.e. movements of the patient and surgical instruments)

12. Planner – Overall concept A volume and surface model of the patient data is constructed beforehand Surgery setup: 6-dof surgical robot for 6-dof surgical robot for Bone cutting, hole creating in patient’s skull Infrared navigation system for Infrared navigation system for Detecting and monitoring the positions of Patient’s skull, robot’s tools, surgeons instruments Patient’s skull, robot’s tools, surgeons instruments Environment modeling 3D modeling of the whole environment including patient data and surgical tools and screws attached to the skull 3D modeling of the whole environment including patient data and surgical tools and screws attached to the skull Online collision-free motion planning for the 6-dof robot The planner reacts according to the current state of the environment The planner reacts according to the current state of the environment Burghart et al., 1999

13. Planner – Environment modeling Convex hull of the surgical wound, patient, and the hooks are generated at different levels dynamically Burghart et al., 1999

14. Planner – Robot motion planning The planner searches a solution in the implicit robot joint-value space (c-space) and checks for collisions in the workspace (environment space) C-space: A* search algorithm A*: a graph-tree search algorithm. Example of best- search algorithm A*: a graph-tree search algorithm. Example of best- search algorithm Eg. Depth-first, breath-first, djkstra Eg. Depth-first, breath-first, djkstra Collision-detection by distance computation in the workspace Burghart et al., 1999

15. Motion planning for performance assessment in Minimally Invasive Surgery Haniffa et al., Motion Planning System for Minimally Invasive Surgery, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07), pp. 609-610, 2007

16. Minimally invasive surgery – laparoscopic surgery Basdogan, Ho, and Sirinivasan, 2001

17. A box-training system mimicking MIS settings An enclosure (box) with openings for surgical instruments Surgical tasks are performed within the box Surgical tasks are performed within the box Surgical instruments are mounted with motion sensors The maneuvers of the trainee are recorded during the performance The maneuvers of the trainee are recorded during the performance Given the task, the optimal traverse of the surgical tools calculated The optimal traverse is compared with the maneuvers of the trainee for performance assessment The optimal traverse is compared with the maneuvers of the trainee for performance assessment

18. Motion planning for replacement of a rubber band across two hooks Potential field method Instrument tips = point robots in c-space Instrument tips = point robots in c-space Artificial potential field: attractive towards the goal state and repulsive towards forbidden regions Artificial potential field: attractive towards the goal state and repulsive towards forbidden regions Haniffa et al., 2007

19. Performance Assessment Steepest descent path Movement towards steepest descent earns credit Movement towards steepest descent earns credit Movement towards obstacles imposes penalties Movement towards obstacles imposes penaltiesAlso Completion time Completion time # of collisions with obstacles and instruments # of collisions with obstacles and instruments # of c-space violations # of c-space violations Haniffa et al., 2007

20. Planning for needle insertion Alterovitz, R et al., “Sensorless planning for medical needle insertion procedures,” IEEE/RSJ International Conference on Intelligent Robots and Systems, volume 4, pp. 3337 – 3343, 2003. R. Alterovitz, K. Goldberg, and A. Okamura, "Planning for steerable bevel-tip needle insertion through 2D soft tissue with obstacles," in Proc. IEEE Int. Conf. on Robotics and Automation, Apr. 2005, pp. 1652--1657 Alterovitz, R et al., “Steering flexible needles under Markov motion uncertainty,” IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1570- 1575, 2005. R. Alterovitz, M. Branicky, and K. Goldberg, "Constant-Curvature Motion Planning Under Uncertainty with Applications in Image- Guided Medical Needle Steering," in Proc. Workshop on the Algorithmic Foundations of Robotics, July 2006.

21. Needle insertion Medical applications: Brachytheraphy (seed implantation): radiotherapy in which the source of radiation is placed (as by implantation) in or close to the area being treated Brachytheraphy (seed implantation): radiotherapy in which the source of radiation is placed (as by implantation) in or close to the area being treated Biopsies: the removal and examination of tissue, cells, or fluids from the living body Biopsies: the removal and examination of tissue, cells, or fluids from the living body Treatment injections: inserting a needle to a specific target location inside the body to inject a drug Treatment injections: inserting a needle to a specific target location inside the body to inject a drug

22. Needle insertion Aim needle tip should be as close as possible to an internal target when the procedure is performed needle tip should be as close as possible to an internal target when the procedure is performedChallenge needle insertion causes soft tissues to displace and deform needle insertion causes soft tissues to displace and deform difficult or impossible to obtain precise imaging data during insertion difficult or impossible to obtain precise imaging data during insertion Incorrect placement of a radioactive seed cannot treat tumor and can damage healthy tissue Incorrect placement of a radioactive seed cannot treat tumor and can damage healthy tissue Alterovitz et al., 2003 Ultrasound images of human prostate

23. Sensorless planning algorithm for radioactive seed implantation – rigid needle The system computes needle insertion offsets that compensate for tissue deformations. It uses 2D FEM model (simulation) of the soft tissues surrounding the target implant location (hence sensorless!) uses 2D FEM model (simulation) of the soft tissues surrounding the target implant location (hence sensorless!) performs dynamic simulation of needle insertion to compute tissue deformations performs dynamic simulation of needle insertion to compute tissue deformations iteratively tests different insertion locations and depths to compute the optimal needle offset iteratively tests different insertion locations and depths to compute the optimal needle offset Alterovitz et al., 2003

24. Problem definition Deformable body in 2D plane, attached target at Pt Insert needle from a specified height until a specified depth, Pr Release seed at that depth = Pr Retract needle Final actual seed position = Pa Note that Pa≠Pr due to tissue defrmation Error = ||Pa-Pt|| Alterovitz et al., 2003

25. Planning For a target, define a set of insertion heights and insertion depths. A virtual 2D grid consisting of different (height, depth) couples Inset needle with constant velocity from a chosen height up to a chosen depth Deform 2D model with FEM Release seed, retract needle, wait for the model to stop its dynamic deformations Calculate error between the final position of the seed and target for that (height, depth) couple (height, depth) couple with minimum error is the solution Alterovitz et al., 2003

26. Steering Flexible Needles Under Markov Motion Uncertainty Unlike rigid needles, flexible bevel tip needles can be steered around obstacles. Flexible needles with bevel tips follow a path of constant curvature in the direction of the bevel. Controlling 2 DOF at the needle base (rotation or bevel direction and insertion distance), the needle can be steered around obstacles to reach targets inaccessible to rigid needles. Planning motion for such a needle is difficult due to uncertainty constraints, i.e. uncertainty in tissue properties tissue properties needle mechanics needle mechanics interaction forces. interaction forces. Alterovitz, Goldberg, Okamura 2005 Alterovitz et al., 2005

27. Motion planning for flexible bevel tip needles in uncertainty Motion planning problem as a Markov Decision Process (MDP) based on Dynamic Programming The planner Computes discrete control sequence of insertion & direction changes in needle Computes discrete control sequence of insertion & direction changes in needle Minimizes expected cost (tissue deformation & damage) due to Minimizes expected cost (tissue deformation & damage) due to insertion distance direction changes obstacle collisions Considers Deterministic case (needle response to controls known) Deterministic case (needle response to controls known) Uncertain case (probability distribution of needle response known) Uncertain case (probability distribution of needle response known)

28. Problem definition Needle Flexible & bevel tip Flexible & bevel tip Stiff soft tissue relative to needle Stiff soft tissue relative to needle Rotation, i.e. bevel direction (right or left, 180 degree turn) Rotation, i.e. bevel direction (right or left, 180 degree turn) Insertion only (no retraction) Insertion only (no retraction) 2D rectangular workspace Specified by segmenting 2D cross-section of patient anatomy via MRI or ultrasound Specified by segmenting 2D cross-section of patient anatomy via MRI or ultrasound Two actions for needle: Insert a distance (constant velocity) Insert a distance (constant velocity) Change direction (180 degree turn, no insertion) and insert a distance (constant velocity) Change direction (180 degree turn, no insertion) and insert a distance (constant velocity) Needle movement damages tissue Cost for both insertion and rotation (occurs as long as needle moves) Cost for both insertion and rotation (occurs as long as needle moves) Prohibitive cost for obstacle colliision Prohibitive cost for obstacle colliision Planning goal: Find a set of discrete needle controls to reach a target from a starting point with minimum cost Alterovitz et al., 2005

29. Problem formulation Dynamic programming requires discrete state Discrete state space: 2D workspace discretized as a grid 2D workspace discretized as a grid Needle movement is discretized based on Needle movement is discretized based on Needle tip position Rotation Control circle variables These two states are merged These two states are merged State transitions: Deterministic: Next state calculated using current state values. P = 1 Deterministic: Next state calculated using current state values. P = 1 Uncertain: Uncertainty due to tissue inhomogeneity Uncertain: Uncertainty due to tissue inhomogeneity 80% deterministic 10% needle tip deviates from input orientation by some positive amount 10% needle tip deviates from input orientation by some negative amount Cost function based on Amount of path traversed from current to next state Amount of path traversed from current to next state If next state is the target – C = 0 If next state is the target – C = 0 If next state collides with any obstacle – C = high, terminate If next state collides with any obstacle – C = high, terminate Total cost: Expected value of sum of state transition costs Alterovitz et al., 2005

30. Motion planning optimization Compute a sequence of controls that minimizes total expected cost of needle insertion Stochastic shortest path problem Solved using infinite horizon dynamic programming Solved using infinite horizon dynamic programming

31. Path planning of catheters in Liver Chemoembolization Gayle et al., Path Planning for Deformable Robots in Complex Environments, Proceedings of Robotics: Systems and Science, 2005

32. Liver Chemoembolization Under x-ray guidance, catheter (tube-like cylinder) is inserted into femoral artery and advanced through a set of arteries to reach near the tumor When reached to the artery supplying a tumor, it injects chemotherapy drugs Careful catheter manipulation is critical: Spasms in small vessels Spasms in small vessels Reflux of chemoembolization into other arteries due to size similarity Reflux of chemoembolization into other arteries due to size similarity Gayle et al., 2005

33. A constrained-based planning algorithm Constrained dynamic simulation Motion planning = solving a list of constraints Motion planning = solving a list of constraints Geometric constraints Obstacle avoidance, non-penetration Obstacle avoidance, non-penetration Physical constraints Volume preservation, energy minimization Volume preservation, energy minimization

34. Catheter is designed as a deformable robot Deformation model = mass-spring system Deformation model = mass-spring system Planning problem: finding sequential robot configurations such that No configuration intersects any obstacle No configuration intersects any obstacle All configurations satisfy constraints and minimize the energy of the system All configurations satisfy constraints and minimize the energy of the systemConstraints Hard: Hard: non-penetration (collision detection and response) Soft: Soft: Goal-seeking (initial path = medial axis) Volume preservation Obstacle avoidance Energy minimization Energy minimization A constrained-based planning algorithm

35. Update robot state given the constraints, Fc constraint force, Fe external force Check if constraints are satisfied subject to energy minimization If not: Set the last valid milestone as the next destination Set the last valid milestone as the next destination Back trace one step on the current roadmap Back trace one step on the current roadmap Find a new path from the last valid milestone to the goal configuration Find a new path from the last valid milestone to the goal configuration Compute new constraint forces and solve the ODE, using the previous state of the robot R and Fe Compute new constraint forces and solve the ODE, using the previous state of the robot R and Fe Set the next robot state to be the new ODE solution Set the next robot state to be the new ODE solution

36. A constrained-based planning algorithm - demo

37. Additional references Basdogan, C., Ho, C., Srinivasan, M.A., 2001, "Virtual Environments for Medical Training: Graphical and Haptic Simulation of Common Bile Duct Exploration", accepted to the IEEE/ASME Transactions on Mechatronics