1 CS 223-B Lecture 1 Sebastian Thrun Gary Bradski CORNEA AQUEOUS HUMOR

2 Readings Computer Vision, Forsyth and Ponce –Chapter 1 Introductory Techniques for 3D Computer Vision, Trucco and Verri –Chapter 2

3 Lenses and Cameras* * Slides, where possible, stolen with abandon, many this lecture from Marc Pollefeys comp256, Lect 2 -- Brunelleschi, XVth Century

4 Distant objects appear smaller A “similar triangle’s” approach to vision. Notes 1.1 Marc Pollefeys

5 Consequences: Parallel lines meet There exist vanishing points Marc Pollefeys

6 Vanishing points VPL VPR H VP 1 VP 2 VP 3 Different directions correspond to different vanishing points Marc Pollefeys

7 Implications For Perception* * A Cartoon Epistemology: Same size things get smaller, we hardly notice… Parallel lines meet at a point…

8 Implications For Perception 2 Perception must be mapped to a space variant grid Logrithmic in nature Steve Lehar

9 The Effect of Perspective

10 Different Projections: Affine projection models: Weak perspective projection is the magnification. When the scene relief is small compared its distance from the Camera, m can be taken constant: weak perspective projection. Smoosh everything flat onto a parallel plane at distance z 0 Marc Pollefeys

11 Affine projection models: Orthographic projection When the camera is at a (roughly constant) distance from the scene, take m=1. Marc Pollefeys

12 Limits for pinhole cameras Marc Pollefeys

13 On to Thin Lenses … Snell’s law n 1 sin  1 = n 2 sin  2 Notes 1.2           a b d e F

14 Paraxial (or first-order) optics Snell’s law: n 1 sin  1 = n 2 sin  2 Small angles: n 1  1  n 2  2 Sin  = y/r Tan  = y/x Marc Pollefeys

15 Thin Lenses spherical lens surfaces; incoming light  parallel to axis; thickness << radii; same refractive index on both sides Notes 1.3 z-> Marc Pollefeys 8

16 Thin Lenses summary Marc Pollefeys

17 The depth-of-field  Marc Pollefeys

18 The depth-of-field  yields Similar formula for Marc Pollefeys

19 The depth-of-field decreases with d, increases with Z 0  strike a balance between incoming light and sharp depth range. Notes 1.4 Marc Pollefeys

20 Deviations from the lens model 3 assumptions : 1.all rays from a point are focused onto 1 image point Remember thin lens small angle assumption 2. all image points in a single plane 3. magnification is constant Deviations from this ideal are aberrations  Marc Pollefeys

21 Aberrations chromatic : refractive index function of wavelength 2 types : 1. geometrical 2. chromatic geometrical : small for paraxial rays  study through 3 rd order optics Marc Pollefeys

22 Geometrical aberrations q spherical aberration q astigmatism q distortion q coma aberrations are reduced by combining lenses  Marc Pollefeys

23 Spherical aberration rays parallel to the axis do not converge outer portions of the lens yield smaller focal lenghts  Marc Pollefeys

24 Astigmatism Different focal length for inclined rays Marc Pollefeys

25 Distortion magnification/focal length different for different angles of inclination Can be corrected! (if parameters are know) pincushion (tele-photo) barrel (wide-angle) Marc Pollefeys

26 Coma point off the axis depicted as comet shaped blob Marc Pollefeys

27 Chromatic aberration rays of different wavelengths focused in different planes cannot be removed completely sometimes achromatization is achieved for more than 2 wavelengths  Marc Pollefeys

28 Vignetting Marc Pollefeys

29 Calibration Gist: Invert the image formation process k th collection of points i PikPik pikpik Image plane x y z 0 Camera R k,T k Extrinsic Params Rotation & Translation to image frame coord. system f, c, , k Intrinsic Params focus center of image Skew  = 0 k radial and tangential distortion (the camera will get several (K) views of this grid in rotation) External coordinate system X Y Z Note that rotation matrix R has constraints: determinant is 1, inverse is equal to transpose, optimization routine should make use of this. Then we want the actual projection to be as close as possible to The point given by the projection operator: over all i points and over all k images of grids: This is typically solved through a gradient decent optimization since the problem is manifestly convex. Note that we need a good starting guess for the initial “correct” projection points p’ I the optimization then iterates to solution. Stereo would then just double the parameters adding left l and right r subscripts and additional summations over r & l.

30 Pseudo-orthographic projection If Z is constant  x= kX and y = kY, where k=f/Z i.e. orthographic projection + a scaling Good approximation if ƒ/ Z ± constant, i.e. if objects are small compared to their distance from the camera  Marc Pollefeys

31 Pictoral comparison  Pseudo - orthographic Perspective Marc Pollefeys

32 Assumed Perspective Projection

33 Assumed Perspective Projection

34 Cameras we consider 2 types :  1. CCD 2. CMOS Marc Pollefeys

35 CCD separate photo sensor at regular positions no scanning charge-coupled devices (CCDs) area CCDs and linear CCDs 2 area architectures : interline transfer and frame transfer photosensitive storage  Marc Pollefeys

36 The CCD camera Marc Pollefeys

37 CMOS Same sensor elements as CCD Each photo sensor has its own amplifier More noise (reduced by subtracting ‘black’ image) Lower sensitivity (lower fill rate) Uses standard CMOS technology Allows to put other components on chip ‘Smart’ pixels Foveon 4k x 4k sensor 0.18  process 70M transistors Marc Pollefeys

38 CCD vs. CMOS Mature technology Specific technology High production cost High power consumption Higher fill rate Blooming Sequential readout Recent technology Standard IC technology Cheap Low power Less sensitive Per pixel amplification Random pixel access Smart pixels On chip integration with other components Marc Pollefeys

39 Colour cameras We consider 3 concepts: 1.Prism (with 3 sensors) 2.Filter mosaic 3.Filter wheel … and X3 Marc Pollefeys

40 Prism colour camera Separate light in 3 beams using dichroic prism Requires 3 sensors & precise alignment Good color separation Marc Pollefeys

41 Prism colour camera Marc Pollefeys

42 Filter mosaic Coat filter directly on sensor Demosaicing (obtain full colour & full resolution image) Marc Pollefeys

43 Filter wheel Rotate multiple filters in front of lens Allows more than 3 colour bands Only suitable for static scenes Marc Pollefeys

44 Prism vs. mosaic vs. wheel Wheel 1 Good Average Low Motion 3 or more approach # sensors Separation Cost Frame rate Artifacts Bands Prism 3 High Low 3 High-end cameras Mosaic 1 Average Low High Aliasing 3 Low-end cameras Scientific applications Marc Pollefeys

45 new color CMOS sensor Foveon’s X3 better image quality smarter pixels Marc Pollefeys

46 The Human Eye Looking down the optical axis of the eye Reproduced by permission, the American Society of Photogrammetry and Remote Sensing. A.L. Nowicki, “Stereoscopy.” Manual of Photogrammetry, Thompson, Radlinski, and Speert (eds.), third edition, Cross section of the eye

47 Sensors and image processing RGB + B/W happens here Question: Which way does the light enter? Light

48 Eye cross section

49 The distribution of rods and cones across the retina Reprinted from Foundations of Vision, by B. Wandell, Sinauer Associates, Inc., (1995).  1995 Sinauer Associates, Inc. Cones in the fovea Rods and cones in the periphery Reprinted from Foundations of Vision, by B. Wandell, Sinauer Associates, Inc., (1995).  1995 Sinauer Associates, Inc.

50 There’s a lot more going on in Vision …i.e. Light and Surfaces

51 Real vision includes invisible inference

52 Real vision includes invisible inference

53 Real vision includes invisible inference