1. Omnidirectional Stereo Vision
Capstone 2004
Lecture 18
Zhigang Zhu
Computer Science Department
The City College, CUNY

2. Acknowledgements Collaborators at UMass Supported by
Edward Riseman
Allen Hanson
Deepak Karuppiah
Howard Schultz
… …
Supported by
NSF Environmental Monitoring
DARPA/ITO Mobile Autonomous Robot S/W
China NSF Scene Modeling
Paper (with references)
11/22/2018
@Z. Zhu CCNY

3. The Class of Omnistereo (omnidirectional stereo vision)
Omnidirectional Vision : How to look
Viewer-centered: outward looking
Object-centered: inward looking
Omnistereo Vision: How many viewpoints
Binocular/N-Ocular: a few (2 or more) fixed
Circular Projection: many inside a small area
Dynamic Omnistereo: a few, but configurable
Object-centered: many, in a large space
11/22/2018
@Z. Zhu CCNY

4. Important Issues of Omnistereo
What this lecture is about
Omnistereo Imaging principle for sensor designs
Epipolar geometry for correspondence
Depth error characterization in both direction and distance
Other important issues not in this talk
Sensor designs
Calibration methods
Correspondence algorithms
11/22/2018
@Z. Zhu CCNY

5. Omni Imaging & Representation
Omnidirectional (panoramic) Imaging
Catadioptric Camera (single effective viewpoint)
ParaVision by RemoteReality, PAL, and many…
Image Mosaicing
Rotating camera, translating camera, arbitrary motion
Omnidirectional Representation
Cylindrical Representation
Spherical Representation
11/22/2018
@Z. Zhu CCNY

6. Panoramic Annular Lens (PAL)
Panoramic Camera
Panoramic Annular Lens (PAL)
By Pal Greguss
11/22/2018
@Z. Zhu CCNY

7. Panoramic Mosaics from a Rotating Camera (ICMCS99)
11/22/2018
@Z. Zhu CCNY

8. 1st frame Cylindrical Panorama connecting frame
conic mosaic
head-tail stitching
panorama
11/22/2018
@Z. Zhu CCNY

9. Cylindrical Projection
Image projection (f, v) of a 3D point P (X,Y,Z) Z Y X O
P (X, Y, Z) v f D
Distance
Cylindrical image
Vertical axis
11/22/2018
@Z. Zhu CCNY

10. Binocular / N-Ocular Omnistereo
A few fixed viewpoints
Three configurations
Horizontally-aligned binocular (H-Bi) omnistereo
Vertically-aligned binocular (V-Bi) omnistereo
N-ocular omnistereo – trinocular case
Issues
Distance error in the direction of 360 degrees
Distance error versus distance
Epipolar geometry
11/22/2018
@Z. Zhu CCNY

11. H-Bi Omnistereo: depth error
From Image pair { (f1, v1), (f2, v2) } to a 3D point P (X,Y,Z)
Triangulation Z Y X O1 O2
P (X, Y, Z) v1 v2 f1 f2 B D f
- Fixed baseline B
- Horizontal disparity (vergent angle)
Depth Error
dD = B (cosq2 sinq dq2 - sinq2 cosq dq) / sinq**2
= B sin (q+q2) / sinq**2 dq
= D**2 cosq / B sinq2 dq <= D**2 / B sinq2 dq
Instead of providing the above, I will present how to derive error equation for the binocular planar stereo vision
Z = F B/ dx
dZ = Z**2 / FB d(dx)
Why not
dZ =- FB/dx**2 d(dx)
Beacuse dx is also a function of F and B!
Depth accuracy is non-isotropic; max vergent only when f2 =90
Not make full use of the 360 viewing
Depth error proportional to Depth2 / Baseline
11/22/2018
@Z. Zhu CCNY

12. H-Bi Omnistereo: singularity case
Zero Vergent angle when f1=f2=0 or 180 degree v1 v2
P (X,Y,Z) O1 O2 B D
epipoles
Distance Ratio Method
- Visible Epipoles: the images of the camera centers in the others could be visible!
- Vertical disparity and vertical epipolar lines
11/22/2018
@Z. Zhu CCNY

13. H-Bi Omnistereo: Epipolar geometry
Given point (f2, v2), search for (f1, v1) f1 v1
180
360
triangulation
singularity
depth-blind spots
-The epipolar curves are sine curves in the non-singularity cases and
- The epipolar lines are along the v direction in the singularity cases
11/22/2018
@Z. Zhu CCNY

14. V-Bi Omnistereo
From Image pair { (f1, v1), (f2, v2) } to a 3D point P (X,Y,Z) v1 v2 O1 O2 P X Y Z Bv
- Vertical baseline Bv
- Vertical disparity v
- Same as perspective stereo
Depth accuracy isotropic in all directions
- Depth error proportional to square of distance
Epipolar lines are simply vertical lines
- But NO stereo viewing without 3D reconstruction
11/22/2018
@Z. Zhu CCNY

15. N-Ocular Omnistereo Why more viewpoints ?
Every point of the 360 FOV from the center of the sensor-triangle can be covered by at least two pairs of rays from different cameras with good triangulations
depth accuracy is still not isotropic, but is more uniform in directions
- one pair of stereo match can be verified using the second pair
- However no gain in epipolar geometry
11/22/2018
@Z. Zhu CCNY

16. Circular Projection Omnistereo
Many viewpoints on a viewing circle
Omnivergent Stereo (Shum et al ICCV99)
every point in the scene is imaged from two cameras that are vergent on that point with maximum vergence angle; and
stereo recovery yields isotropic depth resolution in all directions.
Solution: Circular Projection/ Concentric Mosiacs
A single off-center rotating camera (Peleg CVPR 99, Shum ICCV99)
Full optical design (Peleg PAMI 2000)
My catadioptric omnistereo rig
11/22/2018
@Z. Zhu CCNY

17. Circular Projection: principle
Many viewpoints on a viewing circle Z
viewing circle
Case 2:
two 1D sensors O
Case 1:
an omni sensor
A virtual camera moving in a viewing circle captures two set of rays on a plane tangent to the viewing circle: the left-eye in clockwise direction, and the right-eye in counterclockwise direction
11/22/2018
@Z. Zhu CCNY

18. Circular Projection: geometry
Max vergent angles for left and right rays O P
left-eye ray
right-eye ray f2 f1 D
viewing circle r B O1 O2 f
“baseline”
“disparity”
P: 3D space point
r: radius of the viewing circle
f1,f2: viewing directions of left and right rays
f: vergent angle (angular disparity)
B: baseline length (< 2r);
D: distance (OP)
11/22/2018
@Z. Zhu CCNY

19. Circular Projection: properties
Depth estimation is isotropic
Same depth error in all directions
Make full use of the 360 viewing
Depth error proportional to depth2/baseline
Same as H-Bi Omnistereo
limited baseline (B < 2r)
Horizontal Epipolar lines
Superior than H-Bi Omnistereo when a single viewing circle for left and right omni-images
Extension to Concentric Mosaics with viewing circles of different radii?
11/22/2018
@Z. Zhu CCNY

20. Circular Projection: Implementation
Cameras: Single? Multiple? Standard? Special? Z
viewing circle
Case 1:
two 1D sensors O Z
viewing circle O
Case 2:
an omni sensor
Requirements: Two sets of rays 180o apart
Methods
1: Two Rectilinear Cameras
2: An Omnidirectional camera
Question: Can we do it with a single rectilinear camera?
11/22/2018
@Z. Zhu CCNY

21. Circular Projection: Implementation (I)
Single camera approach
viewing circle O
rotation axis
image plane
path of optical center
left-eye ray
right-eye
ray 2b d R V V O
Rotate a rectilinear camera off its optical center
Take two columns with angular distance 2b << 180o
Viewing circle smaller than circular path of the optical center
Stretching your arm out, camera viewer may be too far from your eyes
11/22/2018
@Z. Zhu CCNY

22. Circular Projection: Implementation (2)
Catadioptric approach O
image plane
viewing circle
rotation axis
(optical center)
path of two “virtual” cameras
right-eye ray
left-eye ray R d Rv OL OR
mirror pair 2g b
>2b O
Rotate a pair of mirror with a camera around its optical center
Look outward at the scene through two slit windows
Larger viewing circle since mirrors enlarge the viewing angle
Camera viewer right in front of your eyes
11/22/2018
@Z. Zhu CCNY

23. Dynamic Ominstereo a few viewpoints moving freely (OmniVision2000)
Requiements:
Optimal configuration for any given point in the world
Change the vergent angle and the baseline freely
Issues:
Dynamic Calibration
View Planning
Target
Baseline
Camera 1
Camera 2
Image 2
Image 1
11/22/2018
@Z. Zhu CCNY

24. Dynamic Ominstereo: depth error
Question 1: Vergent angle
Max vergent angle (f2 = 90o)
Question 2: Baseline
The larger the better?
The error in estimating the baseline
rotation
shift
11/22/2018
@Z. Zhu CCNY

25. Dynamic Ominstereo: mutual calibration
PAL 2
PAL 1
Sensors as calibration targets
Make use of the visible epipoles
Known target geometry
Cylindrical body of the moving platform O2 O1 B a
cylinder body Rc
11/22/2018
@Z. Zhu CCNY

26. Mutual calibration and human tracking: an example
Pano 1:
Image of the 2nd robot
Images of a person
Pano 2:
Image of the 1st robot
Results: B = 180 cm, D1 = 359 cm, D2 = 208 cm
11/22/2018
@Z. Zhu CCNY

27. Dynamic Ominstereo: Optimal view
Baseline error proportional to B2
Larger baseline, even larger error
Overall distance error is min if
“Best” baseline and max vergent angle
Distance error with optimal configuration proportional to D1.5
11/22/2018
@Z. Zhu CCNY

28. Dynamic Ominstereo: Optimal view application
Track a single target by two robots
One stationary, one moving
Omnistereo head with reconfigurable vergent and baseline O1
O2(1)
O2(2)
T(1)
T(2)
rotation
shift
11/22/2018
@Z. Zhu CCNY

29. Dynamic Ominstereo: error simulation
Student project in the spring of 2003
Java Applet
rotation
shift
11/22/2018
@Z. Zhu CCNY

30. Comparisons Four Cases Java Interactive Simulations
Fixed viewpoint omnistereo
One fixed, one circular projection
Both circular projection
Dynamic omnistereo
Java Interactive Simulations
11/22/2018
@Z. Zhu CCNY

31. Java Interactive Simulations
11/22/2018
@Z. Zhu CCNY

32. Object-Centered OmniStereo
Looking inward rather than Looking outward
Modeling objects rather than scenes
Many viewpoints over a large space
Earth
plane
translation
outward rotation
in-ward rotation
object
Modeling a building
Modeling the Earth
11/22/2018
@Z. Zhu CCNY

33. Omni modeling of an object
Inward-Looking Rotation
Many viewpoints over a large circle
Circular projection: viewing circle within the object
Can rotate the (small) object (e.g. human) instead moving the camera
virtual
viewing circle O
rotation axis
image plane
“path” of optical center
left-eye ray
right-eye ray 2b d R
object
11/22/2018
@Z. Zhu CCNY

34. Omni modeling of the earth
Modeling the earth
Airplane flying along great circles
Taking the leading and trailing edge of each frame
Data amount:
1017 pixels if 10 cm2/pixel
1015 pixels if 1 m2/pixel
1012 = 1 Tera = 1000 Giga
Modeling a small area
Rotation can be approximated as translation
Parallel-perspective stereo mosaics
Virtual flying through
Earth
plane
11/22/2018
@Z. Zhu CCNY

35. Parallel-perspective stereo mosaics
Ideal model: Sensor motion is 1D translation, Nadir view
Two “virtual” Pushbroom cameras
Real Applications
Airborne camera (Umass, Sarnoff..)
Ground Vehicles (Tsinghua, Osaka)
Sensor
Image Plane
“Right” Mosaic
“Left” Mosaic
11/22/2018
@Z. Zhu CCNY

36. Re-Organizing the images….
Stereo pair with large FOVs and adaptive baselines
11/22/2018
@Z. Zhu CCNY

37. Re-Organizing the images….
Stereo pair with large FOVs and adaptive baselines
11/22/2018
@Z. Zhu CCNY

38. Re-Organizing the images….
Stereo pair with large FOVs and adaptive baselines
11/22/2018
@Z. Zhu CCNY

39. Re-Organizing the images….
Stereo pair with large FOVs and adaptive baselines
11/22/2018
@Z. Zhu CCNY

40. Re-Organizing the images….
Stereo pair with large FOVs and adaptive baselines
11/22/2018
@Z. Zhu CCNY

41. Re-Organizing the images….
Stereo pair with large FOVs and adaptive baselines
11/22/2018
@Z. Zhu CCNY

42. Recovering Depth from Mosaics
Parallel-perspective stereo mosaics
Depth accuracy independent of depth (in theory)
Two views from different perspective stereo
P(X,Y,Z)
Height H from
Laser Profiler
GPS/IMU
Adaptive baseline
displacement
disparity
Fixed !
11/22/2018
@Z. Zhu CCNY

43. Stereo mosaics of Amazon rain forest
166-frame telephoto video sequence -> 7056*944 mosaics
Left Mosaic
Right Mosaic
Depth Map
11/22/2018
@Z. Zhu CCNY

44. Stereo viewing Red: Right view; Blue/Green: Left view 11/22/2018
@Z. Zhu CCNY

45. Accuracy of 3D from stereo mosaics (ICCV01, VideoReg01)
Adaptive baselines and fixed disparity -uniform depth resolution in theory and accuracy proportional to depth in practice
3D recovery accuracy of parallel-perspective stereo mosaics is comparable to that of a perspective stereo with an optimal baseline
11/22/2018
@Z. Zhu CCNY

46. Conclusions Config. View-points Epipolar Geometry Error in direction
Error in Distance
Binocular
2, fixed
Sine curve
Non-isoptric
 D2/B
Dynamic
2, free
Optimal for target
 D1.5
VCP
viewer-centered
Many, small circle
Horizontal line
isoptric
 D2 /2r
OCP
object-centered
Many, large circle
PPP
Para-perspective
Many, on a line
uniform everywhere
uniform or D
11/22/2018
@Z. Zhu CCNY