1. Instructor: Mircea Nicolescu Lecture 3 CS 485 / 685 Computer Vision

2. Image Formation and Representation Cameras The pinhole model Lenses The human eye Image digitization 2

3. 3 Cameras First photograph due to Niepce (1816) First on record - shown below (1822)

4. 4 Animal eye: a (really) long time ago. Pinhole perspective projection: Brunelleschi, XV th Century. Camera obscura: XVI th Century. Photographic camera: Niepce, 1816. Evolution

5. 5 A Simple Model of Image Formation The scene is illuminated by a single source The scene reflects radiation towards the camera The camera senses it via chemicals on film or in digital sensors

6. Image Formation Two parts to the image formation process: (1)The geometry, which determines where in the image plane the projection of a point in the scene will be located. (2) The physics of light (radiometry), which determines the brightness of a point in the image plane. f(x,y) = i(x,y) r(x,y) Simple model: i: illumination r: reflectance 6

7. Let’s Design a Camera Put a piece of film in front of an object – do we get a reasonable image? −Blurring – need to be more selective! 7

8. Let’s Design a Camera Add a barrier with a small opening (aperture) to block off most of the rays −Reduces blurring 8

9. “Pinhole” Camera Model The simplest device to form an image of a 3D scene on a 2D surface. Rays of light pass through a "pinhole" and form an inverted image of the object on the image plane. Perspective projection: (center of projection) (x,y) (X,Y,Z) f: focal length 9

10. What Is the Effect of Aperture Size? Large aperture: light from the source spreads across the image (not properly focused), making it blurry! Small aperture: reduces blurring but: (i) it limits the amount of light entering the camera and (ii) causes light diffraction. 10

11. Example: Varying Aperture Size 11

12. Example: Varying Aperture Size What happens if we keep decreasing aperture size? When light passes through a small hole, it does not travel in a straight line and is scattered in many directions (diffraction) SOLUTION: refraction 12

13. Refraction Bending of a wave when entering a medium where its speed is different. 13

14. 14 The Reason for Lenses Lens cameras: gather more light from each scene point Pinhole cameras: typically dark, because a very small set of rays from a particular scene point reaches the image plane

15. Lenses Lenses duplicate pinhole geometry without resorting to undesirably small apertures −Gather all the light radiating from an object point towards the lens’s finite aperture −Bring light into focus at a single distinct image point refraction 15

16. Lenses Lenses improve image quality, leading to sharper images. 16

17. Light rays passing through the lens center are not deviated Light rays passing through a point farther from the center are deviated more focal point f Properties of the “Thin” Lens 17

18. All parallel rays converge to a single point For rays perpendicular to the lens, that point is called focal point focal point f Properties of the “Thin” Lens 18

19. Properties of the “Thin” Lens The plane parallel to the lens at the focal point is called the focal plane The distance between the lens and the focal plane is called the focal length (f) focal point f 19

20. Thin Lens Equation Assume an object at distance u from the lens plane: object f u v image 20

21. Thin Lens Equation Using similar triangles: y’/y = v/u f u v y’ y image 21

22. Thin Lens Equation f u v y’ y y’/y = (v-f)/f Using similar triangles: image 22

23. Thin Lens Equation f u v 111 uvf += image Combining the equations: 23

24. Thin Lens Equation The thin lens equation implies that only points at distance u from the lens are “in focus” Other points project to a “blur circle” or “circle of confusion” in the image (blurring occurs) “circle of confusion” 111 uvf += 24

25. Thin Lens Equation When objects move far away from the camera, they will focus on the focal plane focal point f 111 uvf += 25

26. Depth of Field http://www.cambridgeincolour.com/tutorials/depth-of-field.htm The range of depths over which the world is approximately sharp (in focus) 26

27. How Can We Control Depth of Field? The size of blur circle is proportional to aperture size 27

28. Changing aperture size (controlled by diaphragm) affects depth of field −Larger aperture: −decreases the depth of field −but need to decrease exposure time −Smaller aperture: −increases the range in which objects are approximately in focus −but need to increase exposure time How Can We Control Depth of Field? 28

29. Varying Aperture Size Large aperture = small DOFSmall aperture = large DOF 29

30. Another Example Large aperture = small DOF 30

31. Field of View (Zoom) f f The cone of viewing directions of the camera Inversely proportional to focal length 31

32. Field of View (Zoom) 32

33. Reducing Perspective Distortion Small f (large FOV), camera close to car Large f (small FOV), camera far from car Less perspective distortion! by varying distance and focal length 33

34. Same Effect for Faces standardwide-angletelephoto Practical significance: can approximate perspective projection with a simpler model – when using a telephoto lens to view a distant object that has a relatively small range of depths Less perspective distortion! 34

35. Approximating an “Affine” Camera Center of projection is at infinity! 35

36. Real Lenses All but the simplest cameras contain lenses which are actually comprised of several "lens elements" Each element aims to direct the path of light rays such that they recreate the image as accurately as possible on the digital sensor 36

37. Lens Flaws: Chromatic Aberration Lens has different refractive indices for different wavelengths Could cause color fringing: −lens cannot focus all the colors at the same point. 37

38. Chromatic Aberration – Example 38

39. Lens Flaws: Radial Distortion Straight lines become distorted as we move farther away from the center of the image Deviations are most noticeable for rays that pass through the edge of the lens 39

40. No distortionPin cushionBarrel Lens Flaws: Radial Distortion 40

41. Lens Flaws: Tangential Distortion Lens is not exactly parallel to the imaging plane 41

42. 42 Lens Flaws: Vignetting

43. The Human Eye Functions similarly to a camera: aperture (pupil), lens, mechanism for focusing and surface for registering images (retina) 43

44. The Human Eye In a camera, focusing at various distances is achieved by varying the distance between the lens and the imaging plane In the human eye, the distance between the lens and the retina is fixed (14mm to 17mm) 44

45. The Human Eye Focusing is achieved by varying the shape of the lens (flattening or thickening) 45

46. The Human Eye Retina contains light sensitive cells that convert light energy into electrical impulses that travel through nerves to the brain Brain interprets the electrical signals to form images 46

47. The Human Eye Two kinds of light-sensitive cells: rods and cones (unevenly distributed) Cones (6-7 million) are responsible for all color vision and are present throughout the retina, but are concentrated toward the center of the field of vision at the back of the retina Fovea – special area: −Mostly cones −Detail, color sensitivity, and resolution are highest 47

48. The Human Eye Three different types of cones; each type has a special pigment that is sensitive to wavelengths of light in a certain range: −Short (S) corresponds to blue −Medium (M) corresponds to green −Long (L) corresponds to red Ratio of L to M to S cones: −approx. 10:5:1 Almost no S cones in the center of the fovea 48

49. The Human Eye Rods (120 million) are more sensitive to light than cones but cannot discern color −Primary receptors for night vision and detecting motion −Large amount of light overwhelms them, and they take a long time to “reset” and adapt to the dark again −Once fully adapted to darkness, the rods are 10,000 times more sensitive to light than the cones 49

50. Digital Cameras A digital camera replaces film with a sensor array −Each cell in the array is a light-sensitive diode that converts photons to electrons −Two common types −Charge Coupled Device (CCD) −Complementary Metal Oxide Semiconductor (CMOS) http://electronics.howstuffworks.com/digital-camera.htm 50

51. Digital Cameras 51

52. CCD Cameras CCDs move photogenerated charge from pixel to pixel and convert it to voltage at an output node An analog-to-digital converter (ADC) then turns each pixel's value into a digital value http://www.dalsa.com/shared/content/pdfs/CCD_vs_CMOS_Litwiller_2005.pdf 52

53. CMOS Cameras CMOs convert charge to voltage inside each element Use several transistors at each pixel to amplify and move the charge using more traditional wires The CMOS signal is digital, so it needs no ADC http://www.dalsa.com/shared/content/pdfs/CCD_vs_CMOS_Litwiller_2005.pdf 53

54. Image Digitization Sampling: measuring the value of an image at a finite number of points Quantization: representing measured value (voltage) at the sampled point by an integer 54

55. Image Digitization Sampling Quantization 55

56. What Is an Image? 8 bits/pixel 0 255 56

57. We can think of a (grayscale) image as a function f from R 2 to R (or a 2D signal): −f (x,y) gives the intensity at position (x,y) −A digital image is a discrete (sampled, quantized) version of this function x y f (x, y) What Is an Image? 57

58. Image Sampling - Example original image sampled by a factor of 2 sampled by a factor of 4 sampled by a factor of 8 Images have been resized for easier comparison 58

59. Image Quantization - Example 256 gray levels (8 bits/pixel) 32 gray levels (5 bits/pixel) 16 gray levels (4 bits/pixel) 8 gray levels (3 bits/pixel) 4 gray levels (2 bits/pixel) 2 gray levels (1 bit/pixel) 59

60. Color Images Color images are comprised of three color channels – red, green, and, blue – which combine to create most of the colors we can see = 60

61. Color Images 61

62. Color Sensing in Cameras: Prism Requires three chips and precise alignment CCD(B) CCD(G) CCD(R) 62

63. Color Sensing in Cameras: Color Filter Array Why more green? Human Luminance Sensitivity Function Bayer grid In traditional systems, color filters are applied to a single layer of photodetectors in a tiled mosaic pattern 63

64. redgreenblueoutput demosaicing (interpolation) Color Sensing in Cameras: Color Filter Array 64

65. Color Sensing in Cameras: Foveon X3 http://www.foveon.com/article.php?a=67 CMOS sensor; takes advantage of the fact that red, blue and green light silicon to different depths 65