1. Vision Guided Robotics
and Applications in Industry and Medicine
Matthias Rüther

2. Contents Robotics in General Industrial Robotics Medical Robotics
What can Computer Vision do for Robotics?
Vision Sensors
Issues / Problems
Visual Servoing
Application Examples
Summary

3. Robotics What is a robot? Industrial Medical Field Robotics
"A reprogrammable, multifunctional manipulator designed to move material, parts, tools, or specialized devices through various programmed motions for the performance of a variety of tasks"
Robot Institute of America, 1979
Industrial
Mostly automatic manipulation of rigid parts with well-known shape in a specially prepared environment.
Medical
Mostly semi-automatic manipulation of deformable objects in a naturally created, space limited environment.
Field Robotics
Autonomous control and navigation of a mobile vehicle in an arbitrary environment.

4. Robot vs Human Robot Advantages: Human advantages: Strength Accuracy
Speed
Does not tire
Does repetitive tasks
Can Measure
Human advantages:
Intelligence
Flexibility
Adaptability
Skill
Can Learn
Can Estimate

5. Industrial Robot
Requirements:
Accuracy
Tool Quality
Robustness
Strength
Speed
Price Production Cost
Maintenance
Production Quality

6. Medical (Surgical) Robot
Requirements
Safety
Accuracy
Reliability
Tool Quality
Price
Maintenance
Man-Machine Interface

7. What can Computer Vision do for Robotics?
Accurate Robot-Object Positioning
Keeping Relative Position under Movement
Visualization / Teaching / Telerobotics
Performing measurements
Object Recognition
Registration
Visual Servoing

8. Vision Sensors
Single Perspective Camera
Multiple Perspective Cameras (e.g. Stereo Camera Pair)
Laser Scanner
Omnidirectional Camera
Structured Light Sensor

9. Vision Sensors
Single Perspective Camera

10. Vision Sensors
Multiple Perspective Cameras (e.g. Stereo Camera Pair)

11. Vision Sensors
Multiple Perspective Cameras (e.g. Stereo Camera Pair)

12. Vision Sensors
Laser Scanner

13. Vision Sensors
Laser Scanner

14. Vision Sensors
Omnidirectional Camera

15. Vision Sensors
Omnidirectional Camera

16. Vision Sensors
Structured Light Sensor
                                                                             
                                                             
                                                         
Figures from PRIP, TU Vienna

17. Issues/Problems of Vision Guided Robotics
Measurement Frequency
Measurement Uncertainty
Occlusion, Camera Positioning
Sensor dimensions

18. Vision System operates in a closed control loop.
Visual Servoing
Vision System operates in a closed control loop.
Better Accuracy than „Look and Move“ systems
Figures from S.Hutchinson: A Tutorial on Visual Servo Control

19. Visual Servoing Example: Maintaining relative Object Position
Figures from P. Wunsch and G. Hirzinger. Real-Time Visual Tracking of 3-D Objects with Dynamic Handling of Occlusion

20. Visual Servoing Camera Configurations: End-Effector Mounted Fixed
Figures from S.Hutchinson: A Tutorial on Visual Servo Control

21. Visual Servoing Servoing Architectures
Figures from S.Hutchinson: A Tutorial on Visual Servo Control

22. Visual Servoing Position-based and Image Based control Position based:
Alignment in target coordinate system
The 3D structure of the target is rconstructed
The end-effector is tracked
Sensitive to calibration errors
Sensitive to reconstruction errors
Image based:
Alignment in image coordinates
No explicit reconstruction necessary
Insensitive to calibration errors
Only special problems solvable
Depends on initial pose
Depends on selected features
End-effector
target
Image of end effector
Image of target

23. Visual Servoing EOL and ECL control EOL ECL
EOL: endpoint open-loop; only the target is observed by the camera
ECL: endpoint closed-loop; target as well as end-effector are observed by the camera
EOL
ECL

24. Visual Servoing Position Based Algorithm: Example: point alignment p1
Estimation of relative pose
Computation of error between current pose and target pose
Movement of robot
Example: point alignment p1 p2

25. Visual Servoing Position based point alignment p1m p2m
Goal: bring e to 0 by moving p1
e = |p2m – p1m|
u = k*(p2m – p1m)
pxm is subject to the following measurement errors: sensor position, sensor calibration, sensor measurement error
pxm is independent of the following errors: end effector position, target position
p1m
p2m d

26. Visual Servoing Image based point alignment p1 p2
Goal: bring e to 0 by moving p1
e = |u1m – v1m| + |u2m – v2m|
uxm, vxm is subject only to sensor measurement error
uxm, vxm is independent of the following measurement errors: sensor position, end effector position, sensor calibration, target position p1 p2 u1 v1 v2 u2 d1 d2 c1 c2

27. Visual Servoing Example Laparoscopy
Figures from A.Krupa: Autonomous 3-D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing

28. Visual Servoing Example Laparoscopy
Figures from A.Krupa: Autonomous 3-D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing

29. Registration Registration of CAD models to scene features:
Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

30. Registration Registration of CAD models to scene features:
Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

31. Instrument tracking in laparoscopy
Figures from Wei: A Real-time Visual Servoing System for Laparoscopic Surgery

32. Summary
Computer Vision provides accurate and versatile measurements for robotic manipulators
With current general purpose hardware, depth and pose measurements can be performed in real time
In industrial robotics, vision systems are deployed in a fully automated way.
In medicine, computer vision can make more intelligent „surgical assistants“ possible.