1. Carol Reiley Johns Hopkins University
Dynamic Augmented Reality for Haptic Display in Robot-Assisted Surgical Systems
Carol Reiley
Johns Hopkins University
Special thanks to Dr. Allison Okamura and Takintope Akinbiyi

2. Outline Definitions Robotic Systems Current Project
What is Haptics?
Telemanipulation
Augmented Reality
Robotic Systems
Da Vinci Surgical System
Current Project
Goals
Motivation
Experiment Setup
Validation
Results
Conclusion
Questions and Answers

3. Outline Definitions Robotic Systems Da Vinci Surgical System
What is Haptics?
Telemanipulation
Augmented Reality
Robotic Systems
Da Vinci Surgical System
Current Project
Goals
Motivation
Experiment Setup
Validation
Results
Conclusion
Questions and Answers

4. Haptics vs. Optics
Eye is to Optics
Hand is to Haptics
Five Senses

5. What is Haptics? haptesthai – to grasp, to touch
Human haptics (psychophysics and neurophysiology)
Haptic devices (virtual environments, teleoperation)
Robot haptics (sensors, autonomous systems)
3GM, Immersion
Haptic Exploration with robotic fingers (A. M. Okamura, et al., "Haptic Exploration of Objects with Rolling and Sliding," 1997 IEEE ICRA, pp )
Study of haptics emerged from VR. Form of HCI providing environment where one can explore through direct interaction with their sensory. To interact with environment, there must be feedback. (haptics)
Haptic devices is what the user receives
Robotic haptics is what robot receives

6. How a Person Learns Learning Method % Retention What one reads 10%
What one hears 26%
What one sees 30%
What one sees and hears 50%
What one speaks 70%
An old Chinese proverb expresses this: "I hear and I forget; I see and I remember; I do and I understand."

7. Tactile vs. Force Feedback
Cutaneous (tactile)
Related to the skin
Not necessarily related to movement but rely mainly on skin receptors
Textures of surfaces (temperature/vibration)
Kinesthetic and Proprioceptive(force and position)
A sense mediated by end organs located in muscles, tendons, and joints. Stimulated by bodily movements.
Tactile Feedback: Sensation applied to the skin, typically in response to contact or other actions in a virtual world. Tactile feedback can be used to produce a symbol, like Braille, or simply a sensation that indicates some condition.
Force feedback: The sensation of weight or resistance in a virtual world. Force feedback requires a device which produces a force on the body equivalent (or scaled) to that of a real object. It allows a person in cyberspace to feel the weight of virtual objects, or the resistance to motion that they create.
Proprioceptive Feedback: It provides information related to body posture and is based on receptors located at the skeletal joints, in the inner ear, and on impulses from the central nervous system (memory effect). Arm motion

8. Applications of Haptics
Immersion Products
Training/simulation
Gaming
Underwater robotics
Space exploration
Hazardous environments
Medical robotics
* Sensations can be associated with environmental interactions, such as the feel of driving surfaces or footsteps, making the gaming experience more immersive.
* Sensations can also be used to give gamers information about positioning of other players, confirming a completed task, and identifying objects.
* Sensations assist with hand-eye coordination, making it easier to learn and improve mousing skills.
* Sensations make navigating software user interfaces easier, especially for beginning users.
* Last but not least, it's just plain cool to be able to literally feel what it's like to wield a light saber or cast a lightning spell.
Injection trainers:
Interfaces that realistically duplicate the sensations of injecting patient with syringe. Currently used in med schools across nation to improve quality of doctors and nurses that save human lives every day.
Haptic Scissors, JHU
ARQuake, University of South Australia

9. Virtual Environment Haptic System
Motion
Digital position
Interface Card
Haptic Device
Computer
Force
Digital force

10. Input Devices Input/output devices users receive feedback Ground based
Joysticks
Body Based
Gloves
Haptic Devices
Omni Phantom, Sensable Technologies
Impulse Engine 2000, Immersion Inc
Vision Device
Cyberglove, Immersion Inc

11. Virtual Environment Example
Cyberglove Video Clip

12. Teleoperation Telemanipulation Support physical action at a distance
"Teleoperation" technology supports a form of control in which the human directly guides and causes each increment of motion of the slave. Typically the slave robot follows the human motion exactly (within its physical capabilities) although in more advanced, computer mediated, systems there may be coordinate transformations (other than the distance or scale separation of master and slave) imposed between the two sides. A teleoperation system typically sends one of the conjugate variables (either force or velocity) from the operator's hand (via a transducer) to the slave robot. If the conjugate variable is sent back from the slave and transduced at the operator's hand, a virtual energetic link can be created. Telesurgery:
Single doctor to operate on patients across world, n battlefield during wartime from the comforts of a hospital.
2001 Paris to NY
World's First Complete Telesurgery Procedure Performed Using Surgical Robots and Telecommunications Solutions With High Speed
Gall Bladder surgery
SCU Senior Design Project: Haptic Integration of IBM Manipulator, 2004

13. Augmented Reality
Superimpose graphics, sounds, forces, and smell over real world environment in real time
JHU CIRL lab
On the spectrum between virtual reality, which creates immersible, computer-generated environments, and the real world, augmented reality is closer to the real world. Augmented reality adds graphics, sounds, haptics and smell to the natural world as it exists. You can expect video games to drive the development of augmented reality, but this technology will have countless applications. Everyone from tourists to military troops will benefit from the ability to place computer-generated graphics in their field of vision.
Augmented reality will truly change the way we view the world. Picture yourself walking or driving down the street. With augmented-reality displays, which will eventually look much like a normal pair of glasses, informative graphics will appear in your field of view, and audio will coincide with whatever you see. These enhancements will be refreshed continually to reflect the movements of your head. In this article, we will take a look at this future technology, its components and how it will be used.

14. Degrees of Freedom (DOF)
Number of independent position variables needed to in order to locate all parts of a mechanism
DOF does not always correspond to number of joints

15. Da Vinci Surgical System Current Project
Definitions
What is Haptics?
Telemanipulation
Augmented Reality
Robotic Systems
Da Vinci Surgical System
Current Project
Goals
Motivation
Experiment Setup
Validation
Results
Conclusion
Questions and Answers

16. Robot Assisted Surgery
More Information
Guidance System
Overlay
Surgical Assistant
Holding tool/endoscope
Accuracy/Precision
Needle Placement
JHU Steady Hand
Three levels of assistance
1)Just relaying information
2)Assisting surgeon
3)Totally sufficient
-Robodoc
PROJECTS IN LAB
Needle Steering
Robodoc

17. Da Vinci Surgical System Current Project
Definitions
What is Haptics?
Telemanipulation
Augmented Reality
Robotic Systems
Da Vinci Surgical System
Current Project
Goals
Motivation
Experiment Setup
Validation
Results
Conclusion
Questions and Answers

18. Open Surgery
Used for hundreds of years
Large Scar

19. Minimally Invasive Surgery
Benefits
Less time in intensive care
Faster wound healing
Less pain
Faster recovery time
Current problems with Minimally Invasive Surgery (MIS):
Sensory deprivation
Confined space
Lack of dexterity
The main purpose of minimally invasive surgery is to make the experience of surgery easier on the patient, provide better physical results, reduced pain, shorter recovery time and faster return to normal. Compare a common traditional gallbladder surgery with minimally invasive gallbladder surgery.
However, it is difficult to learn to do laparoscopy because of two factors:
1. The doctor must reverse all his normal movements.
2. The surgeon loses a great deal of dexterity because standard tools have no wrist motion.
Traditional:
Minimally Invasive:
• 6-9-inch incision in abdomen
• 4 incisions in abdomen, all smaller than 1/2-inch in diameter
• 5-8 days in the hospital
• 0-2 days in the hospital
• 4-6 weeks of slow recovery
• 1 week recovery and return to normal activities

20. Robotic MIS Surgery
Enhancing ability and precision of surgeons to perform MIS
Scaling motions
Adding additional DOF to instrument tip
Prevents unnecessary tissue damage
Real world application
Telesurgery
Lead placement surgery at Johns Hopkins Hospital using da Vinci surgical system

21. Da Vinci Surgical System Introduction
Oprah’s video clip

22. Forbes Magazine

23. Da Vinci Robot First laparoscopic surgical robot
FDA approved and widely used in hospitals (More than 300 worldwide)
Three components: surgeon console, surgical arm tower, and vision system
Seven DOF
Laparoscopic surgery uses a tube containing a tiny camera that allows the surgeon to see inside the abdomen on a high resolution video screen. Incisions made during this type of surgery tend to be smaller than those made during conventional types of surgery.
There are two major components of the da Vinci system: the viewing console, where the user sits and manipulates the robot and the surgical arm tower that moves the instruments to perform the surgery. To operate the device, a surgeon sits at the console several feet away from the arm tower, peering through an eyepiece that displays 3D images from a high powered camera while remotely manipulating robotic instruments that performs the surgery. The instruments are designed to duplicate the dexterity of the surgeon’s forearms and wrists. The surgery is performed by three incision holes. The first incision is for a tiny camera that displays the images on the console to the user. The other two ports are for the surgical tools which can bend back and forth, side to side and move in a full circle. The robot is controlled by the surgeon’s hands which are connected to manipulation controls on the other side of the room. This robot allows minimally invasive surgical techniques to be performed more quickly and accurately while still looking and feeling like open surgery. It is designed to provide surgeons with the flexibility of traditional open surgery while operating through tiny ports and the precision of the surgery is heightened. For the patient, there’s less pain, less blood loss and faster recovery time.
To recap, surgical procedures routinely performed today using MIS techniques may be performed more quickly and easily with the da Vinci Surgical System.

24. Benefits of MIS With Surgical System
Patient
Reduced trauma to the body
Often less blood loss and need for transfusions
Older Patients
Less post-operative pain and discomfort
Less risk of infection
Shorter hospital stay
Faster recovery
Less scarring
Surgeon:
Enhanced 3D visualization
Scaled motions
Improved dexterity
Improved accuracy
Increased range of motion
More intuitive motions
Reduces tremors
Increases working life of surgeon
provide the surgeon with the dexterity not available using conventional laparoscopic instruments to perform a delicate and precise surgical dissection , reconstruction or removal of specific tissue. 
Minimally Invasive Surgical Incision Holes, Johns Hopkins Hospital 1/25

25. Drawbacks With Surgical System
Lack of haptic (force and tactile feedback)
Longer OR time
Increased cost
More training for surgeons
Limited number of robots available

26. Da Vinci Surgical System Current Project
Definitions
What is Haptics?
Telemanipulation
Augmented Reality
Robotic Systems
Da Vinci Surgical System
Current Project
Goals
Motivation
Experiment Setup
Validation
Results
Conclusion
Questions and Answers

27. Motivation Haptic feedback difficult because of: Limitations:
Sensor placement
Bilateral telemanipulation control issues
Disposable tools
Limitations:
Broken sutures
Tissue damage
Increased OR time

28. The Need for Haptic Feedback
“Perfect” haptic feedback
No haptic feedback
Some haptic feedback
Data summary for a single subject (attending surgeon). The forces applied to various sutures change with suture strength. For this subject, the instrument tie force levels and standard deviations of the hand tie and instrument tie are similar, while those of the robot tie are different.

29. Direct vs. Indirect Haptic Feedback

30. Previous Work Comprehensive subject testing
Preliminary System by Kitagawa, et al.
Results:
Comprehensive subject testing
Overlay improves performance
Limitations:
Sensors on task board, not on instruments
Limited possible tasks (one throw)
Primitive overlay/only one eye
M. Kitagawa, D. Dokko, A. M. Okamura, D. D. Yuh, “ Effect of Sensory Substitution on Suture Manipulation Forces for Robotic Surgical Systems,” Journal of Thoracic and Cardiovascular Surgery, 2005,129:

31. Goals
Display forces to user visually through augmented reality system (sensory substitution)
Evaluate effectiveness of visual force feedback using a telemanipulation system.

32. Our Approach Force sensors
Inexpensive strain gauges
specialized cannulas
Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking.
Superimpose spherical overlay on each tool
Inexpensive strain gauges added to lower shaft of micro-forceps tool of da Vinci Surgical System to measure bending forces. Using specialized cannulas to allow for strain gage wire.
Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking.
Superimpose a spherical overlay on each tool, whose colors change in response to the forces measured at tool tip. Overlays follow tool in real time.

33. Our Approach Force sensors
Inexpensive strain gauges
specialized cannulas
Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking.
Superimpose spherical overlay on each tool
Inexpensive strain gauges added to lower shaft of micro-forceps tool of da Vinci Surgical System to measure bending forces. Using specialized cannulas to allow for strain gage wire.
Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking.
Superimpose a spherical overlay on each tool, whose colors change in response to the forces measured at tool tip. Overlays follow tool in real time.

34. Our Approach Force sensors
Inexpensive strain gauges
specialized cannulas
Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking.
Superimpose spherical overlay on each tool
Inexpensive strain gauges added to lower shaft of micro-forceps tool of da Vinci Surgical System to measure bending forces. Using specialized cannulas to allow for strain gage wire.
Orientation and position of tool end effector and endoscope are tracked using robot kinematics and used in computer vision for tool tracking.
Superimpose a spherical overlay on each tool, whose colors change in response to the forces measured at tool tip. Overlays follow tool in real time.

35. Current System Provides visual sensory substitution Is inexpensive
Is real-time
Can be easily integrated with existing systems
Cost:less expensive than 6dof sensory
Surgeon manipulating da Vinci system

36. Goals
Display forces to user visually through augmented reality system (sensory substitution)
Evaluate effectiveness of visual force feedback using a telemanipulation system.

37. New Force Sensing Instruments
Sensors
Measure bending forces with inexpensive strain gages
Sensor Placement
da Vinci instrument kinematics
Simulation and calibration
Tool End Effector
Special “clamshell” cannulas
Strain gages
Wires through tool shaft
Two pairs of strain gauges in full bridge configuration to maximize sensitivity and measure bending forces are mounted on a da Vinci instrument

38. New Force Sensing Instruments Con’t
Specialized cannulas
Allow for strain gage wire

39. Tracking Overview

40. Goals
Display forces to user visually through augmented reality system (sensory substitution)
Evaluate effectiveness of visual force feedback using a telemanipulation system.

41. Pilot Study

42. Pilot Study

43. Ideal Trial

44. Excessive Force

45. Results Number of Broken Sutures
Visual overlay on: 1
Visual overlay off: 3
5 out of 6 users preferred task with visual overlay

46. Conclusion With Visual Overlay: Future Work: Faster time of completion
Less broken sutures
Future Work:
Comprehensive experiments with tasks that rely more heavily on force feedback
More controlled experiments for statistically significant data
Try different sensory substitution methods
Develop direct haptic feedback

47. Acknowledgements Johns Hopkins University
Dr. Allison Okamura, PhD
Dr. Darius Burschka, PhD
Takintope Akinbiyi, BS
Sunipa Saha, BS
Haptics Exploration Lab
Intuitive Surgical Inc. (California)
Dr. Chris Hasser, PhD
Johns Hopkins Minimally Invasive Surgical Training Center
Dr. Randy Brown, DVM, MSc
Sue Eller
This work was supported in part by the National Science Foundation (EEC ), the Whitaker Foundation (RG ), and the National Institutes of Health (R01-EB002004)

48. Questions and Answers

49. Cool Robotics Video
Dancing Humanoid Robot
Humanoid Robot