1. 3D Computer Vision and Video Computing Omnidirectional Vision Topic 11 of Part 3 Omnidirectional Cameras CSC I6716 Spring 2003 Zhigang Zhu, NAC 8/203A http://www-cs.engr.ccny.cuny.edu/~zhu/VisionCourse-I6716.html

2. 3D Computer Vision and Video Computing Lecture Outline n Applications l Robot navigation, Surveillance, Smart rooms l Video-conferencing/ Tele-presence l Multimedia/Visualization n Page of Omnidirectional Vision (Many universities and companies….) Page of Omnidirectional Vision l http://www.cis.upenn.edu/~kostas/omni.html n Design Requirements l 360 degree FOV, or semi-sphere or full sphere in one snapshot l Single effective viewpoint l Image Resolutions – one or more cameras? l Image Sharpness – optics as well as geometry n Several Important Designs l Catadioptric imaging : mirror (reflection) + lens ( refraction) l Mirrors: Planar, Conic, Spherical, Hyperboloidal, Ellipsoidal, Paraboloidal l Systematic design ( S. Nayar’s group) n Calibrations l Harder or simpler?

3. 3D Computer Vision and Video Computing Sensor Design n Catadioptric imaging : l mirror (reflection) + lens ( refraction) l Theory of Catadioptric Image Formation ( S. Nayar’s group) "A Theory of Single-Viewpoint Catadioptric Image Formation", Simon Baker and Shree K. Nayar, International Journal of Computer Vision, 1999. n Mirrors l Planar l Conic, Spherical l Hyperboloidal, Ellipsoidal l Paraboloidal n Cameras (Lens) l Perspective (pinhole) or orthogonal (tele-centric lens) projection l One or more? n Implementations l Compactness - size, support, and installation l Optics – Image sharpness, reflection, etc.

4. 3D Computer Vision and Video Computing Planar Mirror Panoramic camera system using a pyramid with four (or more) planar mirrors and four (or more) cameras (Nalwa96) has a single effective viewpoint 4 camera design and 6 camera prototype: FullView - Lucent Technology http://www.fullview.com/ http://www.fullview.com/ 6 cameras Mirror pyramid

5. 3D Computer Vision and Video Computing Planar Mirror Panoramic camera system using a pyramid with four (or more) planar mirrors and four (or more) cameras (Nalwa96) has a single effective viewpoint Geometry of 4 camera approach: four separate cameras in 4 viewpoints can generate images with a single effective viewpoint

6. 3D Computer Vision and Video Computing Planar Mirror Approach n A single effective viewpoint n More than one cameras n High image resolution

7. 3D Computer Vision and Video Computing Planar Mirror Approach n A single effective viewpoint n More than one cameras n High image resolution

8. 3D Computer Vision and Video Computing Conic Mirror n Viewpoints on a circle n semispherical view except occlusion n Perspective projection in each direction Robot Navigation (Yagi90, Zhu96/98) viewpoint pinhole

9. 3D Computer Vision and Video Computing Spherical Mirror n Viewpoints on a spherical-like surface Easy to construct ( Hong91 -UMass ) Intersection of incoming rays are along this line Locus of viewpoints

10. 3D Computer Vision and Video Computing Hyperboloidal Mirror n Single Viewpoint l if the pinhole of the real camera and the virtual viewpoint are located at the two loci of the hyperboloid n Semi-spherical view except the self occlusion pinhole P1P1 viewpoint P2P2 Rotation of the hyperbolic curve generates a hyperboloid

11. 3D Computer Vision and Video Computing Hyperboloidal Mirror n ACCOWLE Co., LTD, A Spin-off at Kyoto University l http://www.pluto.dti.ne.jp/~accowle1/ http://www.pluto.dti.ne.jp/~accowle1/ n n Spherical Mirror n Hyperbolic Mirror Image: High res. in the top

12. 3D Computer Vision and Video Computing Ellipsoidal Mirror n Single Viewpoint l if the pinhole of the real camera and the virtual viewpoint are located at the two loci of the ellipsoid n Semi-spherical view except the self occlusion pinhole viewpoint P1P1 P2P2

13. 3D Computer Vision and Video Computing Panoramic Annular Lens panoramic annular lens (PAL) - invented by P. Greguss * 40 mm in diameter, C-mount * view: H: 360, V: -15 ~ +20 * single view point (O) - geometric mathematical model for image transform & calibration p p 1 pinhole P1P1 P B O C Ellipsoidal mirror Hyperboloidal mirror

14. 3D Computer Vision and Video Computing Panoramic Annular Lens panoramic annular lens (PAL) - invented by P. Greguss * 40 mm in diameter, C-mount * view: H: 360, V: -15 ~ +20 single view point (O) C-Mount to CCD Cameras Image: High res. In the bottom

15. 3D Computer Vision and Video Computing Cylindrical panoramic un-warping Two Steps: (1). Center determination (2) Distortion rectification 2-order polynomial approximation

16. 3D Computer Vision and Video Computing Paraboloidal Mirror n Semi-spherical view except the self occlusion n Single Viewpoint at the locus of the paraboloid, if l Tele-lens - orthographic projection is used Mapping between image, mirror and the world invariant to translation of the mirror. This greatly simplifies calibration and the computation of perspective images from paraboloidal images P1P1 viewpoint tele-lens P2P2

17. 3D Computer Vision and Video Computing Paraboloidal Mirror n Remote Reality – A Spin-off at Columbia University n http://www.remotereality.com/ http://www.remotereality.com/ CamcorderWeb CameraBack to Back : Full Spherical View

18. 3D Computer Vision and Video Computing Paraboloidal Mirror n Remote Reality – A Spin-off at Columbia University n http://www.remotereality.com/ http://www.remotereality.com/

19. 3D Computer Vision and Video Computing Catadioptric Camera Calibration n Omnidirectional Camera Calibration – Harder or Easier? l In general, the reflection by the 2 nd order surface makes the calibration procedure harder l However, 360 view may be helpful n Paraboloidal mirror + orthogonal projection Mapping between image, mirror and the world invariant to translation of the mirror. l Projections of two sets of parallel lines suffice for intrinsic calibration from one view n C. Geyer and K. Daniilidis, "Catadioptric Camera calibration", In Proc. Int. Conf. on Computer Vision, Kerkyra, Greece, Sep. 22-25, pp. 398-404, 1999.

20. 3D Computer Vision and Video Computing Image Properties of Paraboloid System n The Image of a Line l is a circular arc if the line is not parallel to the optical axis l Is projected on a (radial) line otherwise n Dual Vanishing Points l There are two VPs for each set of parallel lines, which are the intersections of the corresponding circles n Collinear Centers l The center of the circles for a set of parallel lines are collinear n Vanishing Circle l The vanishing points of lines with coplanar directions* lie on a circle ( all the lines parallel to a common plane) (Assuming aspect ratio = 1)

21. 3D Computer Vision and Video Computing Image Properties of Paraboloid System n The Image Center l Is on the (“vanishing”) line connecting the dual vanishing points of each set of parallel lines l Can be determined by two sets of parallel lines n Projection of a Line with unknown aspect ratio l Is an elliptical arc in the general case n The Aspect Ratio l Is determined by the ratio of the lone-short axes of the ellipse corresponding to a line n Intrinsic Calibration l Estimate aspect ratio by the ratio of ellipse l Estimate the image center by the intersection of vanishing lines of two sets of parallel lines in 3-D space (with aspect ratio)

22. 3D Computer Vision and Video Computing Calibration of Paraboloid System n The Image Center l Is on the (“vanishing”) line connecting the dual vanishing points of each set of parallel lines l Can be determined by two sets of parallel lines

23. 3D Computer Vision and Video Computing Calibration of Paraboloid System n The Image Center l Yellow “vanishing” line of horizontal set of parallel lines l Pink “vanishing” line of vertical set of parallel lines n The Vanishing Circle (Red dotted) l The vanishing points of lines with coplanar directions ( on a plane in this example) Projected to the plane of the calibration pattern

24. 3D Computer Vision and Video Computing Next n Turn in your projects and schedule meetings with me END