1. Vision, Video and Virtual Reality Omnidirectional Vision Lecture 6 Omnidirectional Cameras CSC 59866CD Fall 2004 Zhigang Zhu, NAC 8/203A http://www-cs.engr.ccny.cuny.edu/~zhu/ Capstone2004/Capstone_Sequence2004.html

2. Vision, Video and Virtual Reality Lecture Outline n Applications l Robot navigation, Surveillance, Smart rooms l Video-conferencing/ Tele-presence l Multimedia/Visualization n Page of Omnidirectional Vision (Many universities and companies….) Page of Omnidirectional Vision l http://www.cis.upenn.edu/~kostas/omni.html n Design Requirements l 360 degree FOV, or semi-sphere or full sphere in one snapshot l Single effective viewpoint l Image Resolutions – one or more cameras? l Image Sharpness – optics as well as geometry n Several Important Designs l Catadioptric imaging : mirror (reflection) + lens ( refraction) l Mirrors: Planar, Conic, Spherical, Hyperboloidal, Ellipsoidal, Paraboloidal l Systematic design ( S. Nayar’s group) n Calibrations l Harder or simpler?

3. Vision, Video and Virtual Reality Sensor Design n Catadioptric imaging : l mirror (reflection) + lens ( refraction) l Theory of Catadioptric Image Formation ( S. Nayar’s group) "A Theory of Single-Viewpoint Catadioptric Image Formation", Simon Baker and Shree K. Nayar, International Journal of Computer Vision, 1999. n Mirrors l Planar l Conic, Spherical l Hyperboloidal, Ellipsoidal l Paraboloidal n Cameras (Lens) l Perspective (pinhole) or orthogonal (tele-centric lens) projection l One or more? n Implementations l Compactness - size, support, and installation l Optics – Image sharpness, reflection, etc.

4. Vision, Video and Virtual Reality Planar Mirror Panoramic camera system using a pyramid with four (or more) planar mirrors and four (or more) cameras (Nalwa96) has a single effective viewpoint 4 camera design and 6 camera prototype: FullView - Lucent Technology http://www.fullview.com/ http://www.fullview.com/ 6 cameras Mirror pyramid

5. Vision, Video and Virtual Reality Planar Mirror Panoramic camera system using a pyramid with four (or more) planar mirrors and four (or more) cameras (Nalwa96) has a single effective viewpoint Geometry of 4 camera approach: four separate cameras in 4 viewpoints can generate images with a single effective viewpoint

6. Vision, Video and Virtual Reality Planar Mirror Approach n A single effective viewpoint n More than one cameras n High image resolution

7. Vision, Video and Virtual Reality Planar Mirror Approach n A single effective viewpoint n More than one cameras n High image resolution

8. Vision, Video and Virtual Reality Conic Mirror n Viewpoints on a circle n semispherical view except occlusion n Perspective projection in each direction Robot Navigation (Yagi90, Zhu96/98) viewpoint pinhole

9. Vision, Video and Virtual Reality Spherical Mirror n Viewpoints on a spherical-like surface Easy to construct ( Hong91 -UMass ) Intersection of incoming rays are along this line Locus of viewpoints

10. Vision, Video and Virtual Reality Hyperboloidal Mirror n Single Viewpoint l if the pinhole of the real camera and the virtual viewpoint are located at the two loci of the hyperboloid n Semi-spherical view except the self occlusion pinhole P1P1 viewpoint P2P2 Rotation of the hyperbolic curve generates a hyperboloid

11. Vision, Video and Virtual Reality Hyperboloidal Mirror n ACCOWLE Co., LTD, A Spin-off at Kyoto University l http://www.pluto.dti.ne.jp/~accowle1/ http://www.pluto.dti.ne.jp/~accowle1/ n n Spherical Mirror n Hyperbolic Mirror Image: High res. in the top

12. Vision, Video and Virtual Reality Ellipsoidal Mirror n Single Viewpoint l if the pinhole of the real camera and the virtual viewpoint are located at the two loci of the ellipsoid n Semi-spherical view except the self occlusion pinhole viewpoint P1P1 P2P2

13. Vision, Video and Virtual Reality Panoramic Annular Lens panoramic annular lens (PAL) - invented by P. Greguss * 40 mm in diameter, C-mount * view: H: 360, V: -15 ~ +20 * single view point (O) - geometric mathematical model for image transform & calibration p p 1 pinhole P1P1 P B O C Ellipsoidal mirror Hyperboloidal mirror

14. Vision, Video and Virtual Reality Panoramic Annular Lens panoramic annular lens (PAL) - invented by P. Greguss * 40 mm in diameter, C-mount * view: H: 360, V: -15 ~ +20 single view point (O) C-Mount to CCD Cameras Image: High res. In the bottom

15. Vision, Video and Virtual Reality Cylindrical panoramic un-warping Two Steps: (1). Center determination (2) Distortion rectification 2-order polynomial approximation

16. Vision, Video and Virtual Reality Paraboloidal Mirror n Semi-spherical view except the self occlusion n Single Viewpoint at the locus of the paraboloid, if l Tele-lens - orthographic projection is used Mapping between image, mirror and the world invariant to translation of the mirror. This greatly simplifies calibration and the computation of perspective images from paraboloidal images P1P1 viewpoint tele-lens P2P2

17. Vision, Video and Virtual Reality Paraboloidal Mirror n Remote Reality – A Spin-off at Columbia University n http://www.remotereality.com/ http://www.remotereality.com/ CamcorderWeb CameraBack to Back : Full Spherical View

18. Vision, Video and Virtual Reality Paraboloidal Mirror n Remote Reality – A Spin-off at Columbia University n http://www.remotereality.com/ http://www.remotereality.com/

19. Vision, Video and Virtual Reality Catadioptric Camera Calibration n Omnidirectional Camera Calibration – Harder or Easier? l In general, the reflection by the 2 nd order surface makes the calibration procedure harder l However, 360 view may be helpful n Paraboloidal mirror + orthogonal projection Mapping between image, mirror and the world invariant to translation of the mirror. l Projections of two sets of parallel lines suffice for intrinsic calibration from one view n C. Geyer and K. Daniilidis, "Catadioptric Camera calibration", In Proc. Int. Conf. on Computer Vision, Kerkyra, Greece, Sep. 22-25, pp. 398-404, 1999.

20. Vision, Video and Virtual Reality Image Properties of Paraboloid System n The Image of a Line l is a circular arc if the line is not parallel to the optical axis l Is projected on a (radial) line otherwise n Dual Vanishing Points l There are two VPs for each set of parallel lines, which are the intersections of the corresponding circles n Collinear Centers l The center of the circles for a set of parallel lines are collinear n Vanishing Circle l The vanishing points of lines with coplanar directions* lie on a circle ( all the lines parallel to a common plane) (Assuming aspect ratio = 1)

21. Vision, Video and Virtual Reality Image Properties of Paraboloid System n The Image Center l Is on the (“vanishing”) line connecting the dual vanishing points of each set of parallel lines l Can be determined by two sets of parallel lines n Projection of a Line with unknown aspect ratio l Is an elliptical arc in the general case n The Aspect Ratio l Is determined by the ratio of the lone-short axes of the ellipse corresponding to a line n Intrinsic Calibration l Estimate aspect ratio by the ratio of ellipse l Estimate the image center by the intersection of vanishing lines of two sets of parallel lines in 3-D space (with aspect ratio)

22. Vision, Video and Virtual Reality Calibration of Paraboloid System n The Image Center l Is on the (“vanishing”) line connecting the dual vanishing points of each set of parallel lines l Can be determined by two sets of parallel lines

23. Vision, Video and Virtual Reality Calibration of Paraboloid System n The Image Center l Yellow “vanishing” line of horizontal set of parallel lines l Pink “vanishing” line of vertical set of parallel lines n The Vanishing Circle (Red dotted) l The vanishing points of lines with coplanar directions ( on a plane in this example) Projected to the plane of the calibration pattern

24. Vision, Video and Virtual Reality Next n Next: Features END