Carol Reiley Johns Hopkins University Dynamic Augmented Reality for Haptic Display in Robot-Assisted Surgical Systems Carol Reiley Johns Hopkins University Special thanks to Dr. Allison Okamura and Takintope Akinbiyi

Outline Definitions Robotic Systems Current Project What is Haptics? Telemanipulation Augmented Reality Robotic Systems Da Vinci Surgical System Current Project Goals Motivation Experiment Setup Validation Results Conclusion Questions and Answers

Outline Definitions Robotic Systems Da Vinci Surgical System What is Haptics? Telemanipulation Augmented Reality Robotic Systems Da Vinci Surgical System Current Project Goals Motivation Experiment Setup Validation Results Conclusion Questions and Answers

Haptics vs. Optics Eye is to Optics Hand is to Haptics Five Senses

What is Haptics? haptesthai – to grasp, to touch Human haptics (psychophysics and neurophysiology) Haptic devices (virtual environments, teleoperation) Robot haptics (sensors, autonomous systems) 3GM, Immersion Haptic Exploration with robotic fingers (A. M. Okamura, et al., "Haptic Exploration of Objects with Rolling and Sliding," 1997 IEEE ICRA, pp. 2485-2490) Study of haptics emerged from VR. Form of HCI providing environment where one can explore through direct interaction with their sensory. To interact with environment, there must be feedback. (haptics) Haptic devices is what the user receives Robotic haptics is what robot receives

How a Person Learns Learning Method % Retention What one reads 10% What one hears 26% What one sees 30% What one sees and hears 50% What one speaks 70% An old Chinese proverb expresses this: "I hear and I forget; I see and I remember; I do and I understand."

Tactile vs. Force Feedback Cutaneous (tactile) Related to the skin Not necessarily related to movement but rely mainly on skin receptors Textures of surfaces (temperature/vibration) Kinesthetic and Proprioceptive(force and position) A sense mediated by end organs located in muscles, tendons, and joints. Stimulated by bodily movements. Tactile Feedback: Sensation applied to the skin, typically in response to contact or other actions in a virtual world. Tactile feedback can be used to produce a symbol, like Braille, or simply a sensation that indicates some condition. Force feedback: The sensation of weight or resistance in a virtual world. Force feedback requires a device which produces a force on the body equivalent (or scaled) to that of a real object. It allows a person in cyberspace to feel the weight of virtual objects, or the resistance to motion that they create. Proprioceptive Feedback: It provides information related to body posture and is based on receptors located at the skeletal joints, in the inner ear, and on impulses from the central nervous system (memory effect). Arm motion

Applications of Haptics Immersion Products Training/simulation Gaming Underwater robotics Space exploration Hazardous environments Medical robotics * Sensations can be associated with environmental interactions, such as the feel of driving surfaces or footsteps, making the gaming experience more immersive. * Sensations can also be used to give gamers information about positioning of other players, confirming a completed task, and identifying objects. * Sensations assist with hand-eye coordination, making it easier to learn and improve mousing skills. * Sensations make navigating software user interfaces easier, especially for beginning users. * Last but not least, it's just plain cool to be able to literally feel what it's like to wield a light saber or cast a lightning spell. Injection trainers: Interfaces that realistically duplicate the sensations of injecting patient with syringe. Currently used in med schools across nation to improve quality of doctors and nurses that save human lives every day. Haptic Scissors, JHU ARQuake, University of South Australia

Virtual Environment Haptic System Motion Digital position Interface Card Haptic Device Computer Force Digital force

Input Devices Input/output devices users receive feedback Ground based Joysticks Body Based Gloves Haptic Devices Omni Phantom, Sensable Technologies Impulse Engine 2000, Immersion Inc Vision Device Cyberglove, Immersion Inc

Virtual Environment Example Cyberglove Video Clip

Teleoperation Telemanipulation Support physical action at a distance "Teleoperation" technology supports a form of control in which the human directly guides and causes each increment of motion of the slave. Typically the slave robot follows the human motion exactly (within its physical capabilities) although in more advanced, computer mediated, systems there may be coordinate transformations (other than the distance or scale separation of master and slave) imposed between the two sides. A teleoperation system typically sends one of the conjugate variables (either force or velocity) from the operator's hand (via a transducer) to the slave robot. If the conjugate variable is sent back from the slave and transduced at the operator's hand, a virtual energetic link can be created. Telesurgery: Single doctor to operate on patients across world, n battlefield during wartime from the comforts of a hospital. 2001 Paris to NY World's First Complete Telesurgery Procedure Performed Using Surgical Robots and Telecommunications Solutions With High Speed Gall Bladder surgery SCU Senior Design Project: Haptic Integration of IBM Manipulator, 2004

Augmented Reality Superimpose graphics, sounds, forces, and smell over real world environment in real time JHU CIRL lab On the spectrum between virtual reality, which creates immersible, computer-generated environments, and the real world, augmented reality is closer to the real world. Augmented reality adds graphics, sounds, haptics and smell to the natural world as it exists. You can expect video games to drive the development of augmented reality, but this technology will have countless applications. Everyone from tourists to military troops will benefit from the ability to place computer-generated graphics in their field of vision. Augmented reality will truly change the way we view the world. Picture yourself walking or driving down the street. With augmented-reality displays, which will eventually look much like a normal pair of glasses, informative graphics will appear in your field of view, and audio will coincide with whatever you see. These enhancements will be refreshed continually to reflect the movements of your head. In this article, we will take a look at this future technology, its components and how it will be used.

Degrees of Freedom (DOF) Number of independent position variables needed to in order to locate all parts of a mechanism DOF does not always correspond to number of joints

Da Vinci Surgical System Current Project Definitions What is Haptics? Telemanipulation Augmented Reality Robotic Systems Da Vinci Surgical System Current Project Goals Motivation Experiment Setup Validation Results Conclusion Questions and Answers

Robot Assisted Surgery More Information Guidance System Overlay Surgical Assistant Holding tool/endoscope Accuracy/Precision Needle Placement JHU Steady Hand Three levels of assistance 1)Just relaying information 2)Assisting surgeon 3)Totally sufficient -Robodoc PROJECTS IN LAB Needle Steering Robodoc

Da Vinci Surgical System Current Project Definitions What is Haptics? Telemanipulation Augmented Reality Robotic Systems Da Vinci Surgical System Current Project Goals Motivation Experiment Setup Validation Results Conclusion Questions and Answers

Open Surgery Used for hundreds of years Large Scar

Minimally Invasive Surgery Benefits Less time in intensive care Faster wound healing Less pain Faster recovery time Current problems with Minimally Invasive Surgery (MIS): Sensory deprivation Confined space Lack of dexterity http://www.shscares.org/services/MIS/misbenefit.asp The main purpose of minimally invasive surgery is to make the experience of surgery easier on the patient, provide better physical results, reduced pain, shorter recovery time and faster return to normal. Compare a common traditional gallbladder surgery with minimally invasive gallbladder surgery. However, it is difficult to learn to do laparoscopy because of two factors: 1. The doctor must reverse all his normal movements. 2. The surgeon loses a great deal of dexterity because standard tools have no wrist motion. Traditional: Minimally Invasive: • 6-9-inch incision in abdomen • 4 incisions in abdomen, all smaller than 1/2-inch in diameter • 5-8 days in the hospital • 0-2 days in the hospital • 4-6 weeks of slow recovery • 1 week recovery and return to normal activities

Robotic MIS Surgery Enhancing ability and precision of surgeons to perform MIS Scaling motions Adding additional DOF to instrument tip Prevents unnecessary tissue damage Real world application Telesurgery Lead placement surgery at Johns Hopkins Hospital using da Vinci surgical system

Da Vinci Surgical System Introduction Oprah’s video clip

Forbes Magazine

Da Vinci Robot First laparoscopic surgical robot FDA approved and widely used in hospitals (More than 300 worldwide) Three components: surgeon console, surgical arm tower, and vision system Seven DOF Laparoscopic surgery uses a tube containing a tiny camera that allows the surgeon to see inside the abdomen on a high resolution video screen. Incisions made during this type of surgery tend to be smaller than those made during conventional types of surgery. There are two major components of the da Vinci system: the viewing console, where the user sits and manipulates the robot and the surgical arm tower that moves the instruments to perform the surgery. To operate the device, a surgeon sits at the console several feet away from the arm tower, peering through an eyepiece that displays 3D images from a high powered camera while remotely manipulating robotic instruments that performs the surgery. The instruments are designed to duplicate the dexterity of the surgeon’s forearms and wrists. The surgery is performed by three incision holes. The first incision is for a tiny camera that displays the images on the console to the user. The other two ports are for the surgical tools which can bend back and forth, side to side and move in a full circle. The robot is controlled by the surgeon’s hands which are connected to manipulation controls on the other side of the room. This robot allows minimally invasive surgical techniques to be performed more quickly and accurately while still looking and feeling like open surgery. It is designed to provide surgeons with the flexibility of traditional open surgery while operating through tiny ports and the precision of the surgery is heightened. For the patient, there’s less pain, less blood loss and faster recovery time. To recap, surgical procedures routinely performed today using MIS techniques may be performed more quickly and easily with the da Vinci Surgical System. http://www.intuitivesurgical.com/

Benefits of MIS With Surgical System Patient Reduced trauma to the body Often less blood loss and need for transfusions Older Patients Less post-operative pain and discomfort Less risk of infection Shorter hospital stay Faster recovery Less scarring Surgeon: Enhanced 3D visualization Scaled motions Improved dexterity Improved accuracy Increased range of motion More intuitive motions Reduces tremors Increases working life of surgeon provide the surgeon with the dexterity not available using conventional laparoscopic instruments to perform a delicate and precise surgical dissection , reconstruction or removal of specific tissue.  Minimally Invasive Surgical Incision Holes, Johns Hopkins Hospital 1/25

Drawbacks With Surgical System Lack of haptic (force and tactile feedback) Longer OR time Increased cost More training for surgeons Limited number of robots available

Da Vinci Surgical System Current Project Definitions What is Haptics? Telemanipulation Augmented Reality Robotic Systems Da Vinci Surgical System Current Project Goals Motivation Experiment Setup Validation Results Conclusion Questions and Answers

Motivation Haptic feedback difficult because of: Limitations: Sensor placement Bilateral telemanipulation control issues Disposable tools Limitations: Broken sutures Tissue damage Increased OR time

The Need for Haptic Feedback “Perfect” haptic feedback No haptic feedback Some haptic feedback Data summary for a single subject (attending surgeon). The forces applied to various sutures change with suture strength. For this subject, the instrument tie force levels and standard deviations of the hand tie and instrument tie are similar, while those of the robot tie are different.

Direct vs. Indirect Haptic Feedback

Previous Work Comprehensive subject testing Preliminary System by Kitagawa, et al. Results: Comprehensive subject testing Overlay improves performance Limitations: Sensors on task board, not on instruments Limited possible tasks (one throw) Primitive overlay/only one eye M. Kitagawa, D. Dokko, A. M. Okamura, D. D. Yuh, “ Effect of Sensory Substitution on Suture Manipulation Forces for Robotic Surgical Systems,” Journal of Thoracic and Cardiovascular Surgery, 2005,129:151-158.

Goals Display forces to user visually through augmented reality system (sensory substitution) Evaluate effectiveness of visual force feedback using a telemanipulation system.

Our Approach Force sensors Inexpensive strain gauges specialized cannulas Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking. Superimpose spherical overlay on each tool Inexpensive strain gauges added to lower shaft of micro-forceps tool of da Vinci Surgical System to measure bending forces. Using specialized cannulas to allow for strain gage wire. Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking. Superimpose a spherical overlay on each tool, whose colors change in response to the forces measured at tool tip. Overlays follow tool in real time.

Our Approach Force sensors Inexpensive strain gauges specialized cannulas Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking. Superimpose spherical overlay on each tool Inexpensive strain gauges added to lower shaft of micro-forceps tool of da Vinci Surgical System to measure bending forces. Using specialized cannulas to allow for strain gage wire. Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking. Superimpose a spherical overlay on each tool, whose colors change in response to the forces measured at tool tip. Overlays follow tool in real time.

Our Approach Force sensors Inexpensive strain gauges specialized cannulas Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking. Superimpose spherical overlay on each tool Inexpensive strain gauges added to lower shaft of micro-forceps tool of da Vinci Surgical System to measure bending forces. Using specialized cannulas to allow for strain gage wire. Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking. Superimpose a spherical overlay on each tool, whose colors change in response to the forces measured at tool tip. Overlays follow tool in real time.

Current System Provides visual sensory substitution Is inexpensive Is real-time Can be easily integrated with existing systems Cost:less expensive than 6dof sensory Surgeon manipulating da Vinci system

Goals Display forces to user visually through augmented reality system (sensory substitution) Evaluate effectiveness of visual force feedback using a telemanipulation system.

New Force Sensing Instruments Sensors Measure bending forces with inexpensive strain gages Sensor Placement da Vinci instrument kinematics Simulation and calibration Tool End Effector Special “clamshell” cannulas Strain gages Wires through tool shaft Two pairs of strain gauges in full bridge configuration to maximize sensitivity and measure bending forces are mounted on a da Vinci instrument

New Force Sensing Instruments Con’t Specialized cannulas Allow for strain gage wire

Tracking Overview

Goals Display forces to user visually through augmented reality system (sensory substitution) Evaluate effectiveness of visual force feedback using a telemanipulation system.

Pilot Study

Pilot Study

Ideal Trial

Excessive Force

Results Number of Broken Sutures Visual overlay on: 1 Visual overlay off: 3 5 out of 6 users preferred task with visual overlay

Conclusion With Visual Overlay: Future Work: Faster time of completion Less broken sutures Future Work: Comprehensive experiments with tasks that rely more heavily on force feedback More controlled experiments for statistically significant data Try different sensory substitution methods Develop direct haptic feedback

Acknowledgements Johns Hopkins University Dr. Allison Okamura, PhD Dr. Darius Burschka, PhD Takintope Akinbiyi, BS Sunipa Saha, BS Haptics Exploration Lab Intuitive Surgical Inc. (California) Dr. Chris Hasser, PhD Johns Hopkins Minimally Invasive Surgical Training Center Dr. Randy Brown, DVM, MSc Sue Eller This work was supported in part by the National Science Foundation (EEC-9731478), the Whitaker Foundation (RG-02-911), and the National Institutes of Health (R01-EB002004)

Questions and Answers

Cool Robotics Video Dancing Humanoid Robot Humanoid Robot