1. Introduction to Robot Vision
Ziv Yaniv Computer Aided Interventions and Medical Robotics, Georgetown University

2. Vision
The special sense by which the qualities of an object (as color, luminosity, shape, and size) constituting its appearance are perceived through a process in which light rays entering the eye are transformed by the retina into electrical signals that are transmitted to the brain via the optic nerve.
[Miriam Webster dictionary]

3. Single Lens Reflex (SLR) Camera
The Sensor
endoscope
Single Lens Reflex (SLR) Camera
webcam
C-arm X-ray

4. The Sensor Model: Pin-hole Camera, Perspective Projection y z x
Image plane
optical axis
focal point x y z
principle point

5. Machine Vision Goal: Model:
Obtain useful information about the 3D world from 2D images.
Model:
Regions
Textures
Corners
Lines …
3D Geometry
Object identification
Activity detection …
images
actions

6. Machine Vision Goal: High level (machine learning/pattern recognition)
Obtain useful information about the 3D world from 2D images.
Low level (image processing)
image filtering (smoothing, histogram modification…),
feature extraction (corner detection, edge detection,…)
stereo vision
shape from X (shading, motion,…) …
High level (machine learning/pattern recognition)
object detection
object recognition
clustering

7. Machine Vision
How hard can it be?

8. Machine Vision
How hard can it be?

9. Robot Vision Simultaneous Localization and Mapping (SLAM)
Visual Servoing.

10. Robot Vision
Simultaneous Localization and Mapping (SLAM) – create a 3D map of the world and localize within this map.
NASA stereo vision image processing, as used by the MER Mars rovers

11. Robot Vision
Simultaneous Localization and Mapping (SLAM) – create a 3D map of the world and localize within this map.
“Simultaneous Localization and Mapping with Active Stereo Vision”, J. Diebel, K. Reuterswärd, S. Thrun, J. Davis, R. Gupta, IROS 2004.

12. Robot Vision
Visual Servoing – Using visual feedback to control a robot:
image-based systems: desired motion directly from image.
“An image-based visual servoing scheme for following paths with nonholonomic mobile robots” A. Cherubini, F. Chaumette, G. Oriolo,
ICARCV 2008.

13. Robot Vision
Visual Servoing – Using visual feedback to control a robot:
Position-based systems: desired motion from 3D reconstruction estimated from image.

14. System Configuration
Difficulty of similar tasks in different settings varies widely:
How many cameras?
Are the cameras calibrated?
What is the camera-robot configuration?
Is the system calibrated (hand-eye calibration)?
Common configurations: x y z x y z x y z x y z x y z x y z x y z

15. System Characteristics
The greater the control over the system configuration and environment the easier it is to execute a task.
System accuracy is directly dependent upon model accuracy – what accuracy does the task require?.
All measurements and derived quantitative values have an associated error.

16. Stereo Reconstruction
Compute the 3D location of a point in the stereo rig’s coordinate system:
Rigid transformation between the two cameras is known.
Cameras are calibrated –given a point in the world coordinate system we know how to map it to the image.
Same point localized in the two images.
Camera 1
Camera 2
world

17. Commercial Stereo Vision
Polaris Vicra infra-red system (Northern Digitial Inc.)
MicronTracker visible light system
(Claron Technology Inc.)

18. Commercial Stereo Vision
Images acquired by the Polaris Vicra infra-red stereo system:
left image
right image

19. Stereo Reconstruction
Wide or short baseline – reconstruction accuracy vs. difficulty of point matching
Camera 1
Camera 2
Camera 1
Camera 2
Camera 1
Camera 2

20. Camera Model
Points P, p, and O, given in the camera coordinate system, are collinear.
There is a number a for which O + aP = p
aP = p
a = f/Z
, therefore x y z
P=[X,Y,Z] O
p=[x,y,f] f

21. Camera Model
Transform the pixel coordinates from the camera coordinate system to the image coordinate system:
Image origin (principle point) is at [x0,y0] relative to the camera coordinate system.
Need to change from metric units to pixels, scaling factors kx, ky. x y
principle point
[x’,y’]
Finally, the image coordinate system may be skewed resulting in:

22. Camera Model
As our original assumption was that points are given in the camera coordinate system, a complete projection matrix is of the form:
C – camera origin in the
world coordinate system.
How many degrees of freedom does M have?

23. world coordinate system
Camera Calibration
Given pairs of points, piT=[x,y,w], PiT=[X,Y,Z,W], in homogenous coordinates we have:
image coordinate system z x
calibration object/
world coordinate system y
principle point
Our goal is to estimate M y z x
camera coordinate system
As the points are in homogenous coordinates the vectors p and MP are not necessarily equal, they have the same direction but may differ by a non-zero scale factor.

24. Camera Calibration After a bit of algebra we have:
The three equations are linearly dependent:
Each point pair contributes two equations.
Exact solution: M has 11 degrees of freedom, requiring a minimum of n=6 pairs.
Least squares solution: For n>6 minimize ||Am|| s.t. ||m||=1.

25. Obtaining the Rays
Camera location in the calibration object’s coordinate system, C, is given by the one dimensional right null space of the matrix M (MC=0).
A 3D homogenous point P = M+p is on the ray defined by p and the camera center [it projects onto p, MM+p =Ip=p].
These two points define our ray in the world coordinate system.
As both cameras were calibrated with respect to the same coordinate system the rays will be in the same system too.

26. Intersecting the Rays

27. World vs. Model
Actual cameras most often don’t follow the ideal pin-hole model, usually exhibit some form of distortion (barrel, pin-cushion, S).
Sometimes the world changes to fit your model, improvements in camera/lens quality can improve model performance.
old image-Intensifier x-ray: pin-hole+distortion
replaced by flat panel x-ray: pin-hole

28. Additional Material Code: Machine Vision:
Camera calibration toolbox for matlab (Jean-Yves Bouguet )
Machine Vision:
“Multiple View Geometry in Computer Vision”, Hartley and Zisserman, Cambridge University Press.
"Machine Vision", Jain, Kasturi, Schunck, McGraw-Hill.
Robot Vision:
“Simultaneous Localization and Mapping: Part I”, H. Durant-Whyte, T. Bailey, IEEE Robotics and Automation Magazine, Vol. 13(2), pp , 2006.
“Simultaneous Localization and Mapping (SLAM) : Part II”,T. Bailey, H. Durant-Whyte, IEEE Robotics and Automation Magazine, Vol. 13(3), pp , 2006.
“Visual Servo Control Part I: Basic Approaches”, IEEE Robotics and Automation Magazine, Vol. 13(4), 82-90, 2006.
Visual Servo Control Part II: Advanced Approaches”, IEEE Robotics and Automation Magazine, Vol. 14(1), , 2007.