1. Recap from Lecture 2 Pinhole camera model Perspective projections Lenses and their flaws Focus Depth of field Focal length and field of view Chapter 2 of Szeliski

2. What is wrong with this picture?

3. Capturing Light… in man and machine CS 129: Computational Photography James Hays, Brown, Spring 2011 Many slides by Alexei A. Efros

4. Image Formation Digital Camera The Eye Film

5. Digital camera A digital camera replaces film with a sensor array Each cell in the array is light-sensitive diode that converts photons to electrons Two common types –Charge Coupled Device (CCD) –CMOS http://electronics.howstuffworks.com/digital-camera.htm Slide by Steve Seitz

6. Sensor Array CMOS sensor

7. Sampling and Quantization

8. Interlace vs. progressive scan http://www.axis.com/products/video/camera/progressive_scan.htmSlide by Steve Seitz

9. Progressive scan http://www.axis.com/products/video/camera/progressive_scan.htmSlide by Steve Seitz

10. Interlace http://www.axis.com/products/video/camera/progressive_scan.htmSlide by Steve Seitz

11. Rolling Shutter

12. The Eye The human eye is a camera! Iris - colored annulus with radial muscles Pupil - the hole (aperture) whose size is controlled by the iris What’s the “film”? –photoreceptor cells (rods and cones) in the retina Slide by Steve Seitz

13. The Retina

14. Retina up-close Light

15. What humans don’t have: tapetum lucidum

16. © Stephen E. Palmer, 2002 Cones cone-shaped less sensitive operate in high light color vision Two types of light-sensitive receptors Rods rod-shaped highly sensitive operate at night gray-scale vision

17. Rod / Cone sensitivity

18. © Stephen E. Palmer, 2002 Distribution of Rods and Cones Night Sky: why are there more stars off-center? Averted vision: http://en.wikipedia.org/wiki/Averted_vision

19. Eye Movements Saccades Can be consciously controlled. Related to perceptual attention. 200ms to initiation, 20 to 200ms to carry out. Large amplitude. Microsaccades Involuntary. Smaller amplitude. Especially evident during prolonged fixation. Function debated. Ocular microtremor (OMT) involuntary. high frequency (up to 80Hz), small amplitude.

20. Electromagnetic Spectrum http://www.yorku.ca/eye/photopik.htm Human Luminance Sensitivity Function

21. Why do we see light of these wavelengths? © Stephen E. Palmer, 2002 …because that’s where the Sun radiates EM energy Visible Light

22. The Physics of Light Any patch of light can be completely described physically by its spectrum: the number of photons (per time unit) at each wavelength 400 - 700 nm. © Stephen E. Palmer, 2002

23. The Physics of Light Some examples of the spectra of light sources © Stephen E. Palmer, 2002

24. The Physics of Light Some examples of the reflectance spectra of surfaces Wavelength (nm) % Photons Reflected Red 400 700 Yellow 400 700 Blue 400 700 Purple 400 700 © Stephen E. Palmer, 2002

25. The Psychophysical Correspondence There is no simple functional description for the perceived color of all lights under all viewing conditions, but …... A helpful constraint: Consider only physical spectra with normal distributions area mean variance © Stephen E. Palmer, 2002

26. The Psychophysical Correspondence MeanHue # Photons Wavelength © Stephen E. Palmer, 2002

27. The Psychophysical Correspondence VarianceSaturation Wavelength # Photons © Stephen E. Palmer, 2002

28. The Psychophysical Correspondence AreaBrightness # Photons Wavelength © Stephen E. Palmer, 2002

29. Three kinds of cones: Physiology of Color Vision Why are M and L cones so close? Why are there 3?

30. Tetrachromatism Most birds, and many other animals, have cones for ultraviolet light. Some humans, mostly female, seem to have slight tetrachromatism. Bird cone responses

31. More Spectra metamers

32. Color Sensing in Camera (RGB) 3-chip vs. 1-chip: quality vs. cost Why more green? http://www.coolditionary.com/words/Bayer-filter.wikipedia http://www.cooldictionary.com/words/Bayer-filter.wikipedia Why 3 colors? Slide by Steve Seitz

33. Practical Color Sensing: Bayer Grid Estimate RGB at ‘G’ cells from neighboring values http://www.cooldictionary.com/ words/Bayer-filter.wikipedia Slide by Steve Seitz

34. Color Image R G B

35. Images in Matlab Images represented as a matrix Suppose we have a NxM RGB image called “im” –im(1,1,1) = top-left pixel value in R-channel –im(y, x, b) = y pixels down, x pixels to right in the b th channel –im(N, M, 3) = bottom-right pixel in B-channel imread(filename) returns a uint8 image (values 0 to 255) –Convert to double format (values 0 to 1) with im2double 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93 R G B row column

36. Color spaces How can we represent color? http://en.wikipedia.org/wiki/File:RGB_illumination.jpg

37. Color spaces: RGB 0,1,0 0,0,1 1,0,0 Image from: http://en.wikipedia.org/wiki/File:RGB_color_solid_cube.png Some drawbacks Strongly correlated channels Non-perceptual Default color space R (G=0,B=0) G (R=0,B=0) B (R=0,G=0)

38. Color spaces: HSV Intuitive color space H (S=1,V=1) S (H=1,V=1) V (H=1,S=0)

39. Color spaces: YCbCr Y (Cb=0.5,Cr=0.5) Cb (Y=0.5,Cr=0.5) Cr (Y=0.5,Cb=05) Y=0Y=0.5 Y=1 Cb Cr Fast to compute, good for compression, used by TV

40. Color spaces: L*a*b* “Perceptually uniform”* color space L (a=0,b=0) a (L=65,b=0) b (L=65,a=0)

41. Project #1 How to compare R,G,B channels? No right answer Sum of Squared Differences (SSD): Normalized Correlation (NCC):

42. Image half-sizing This image is too big to fit on the screen. How can we reduce it? How to generate a half- sized version?

43. Image sub-sampling Throw away every other row and column to create a 1/2 size image - called image sub-sampling 1/4 1/8 Slide by Steve Seitz

44. Image sub-sampling 1/4 (2x zoom) 1/8 (4x zoom) Aliasing! What do we do? 1/2 Slide by Steve Seitz

45. Gaussian (lowpass) pre-filtering G 1/4 G 1/8 Gaussian 1/2 Solution: filter the image, then subsample Filter size should double for each ½ size reduction. Why? Slide by Steve Seitz

46. Subsampling with Gaussian pre-filtering G 1/4G 1/8Gaussian 1/2 Slide by Steve Seitz

47. Compare with... 1/4 (2x zoom) 1/8 (4x zoom) 1/2 Slide by Steve Seitz

48. Gaussian (lowpass) pre-filtering G 1/4 G 1/8 Gaussian 1/2 Solution: filter the image, then subsample Filter size should double for each ½ size reduction. Why? How can we speed this up? Slide by Steve Seitz

49. Image Pyramids Known as a Gaussian Pyramid [Burt and Adelson, 1983] In computer graphics, a mip map [Williams, 1983] A precursor to wavelet transform Slide by Steve Seitz

50. A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose Figure from David Forsyth

51. What are they good for? Improve Search Search over translations –Like project 1 –Classic coarse-to-fine strategy Search over scale –Template matching –E.g. find a face at different scales Pre-computation Need to access image at different blur levels Useful for texture mapping at different resolutions (called mip-mapping)

52. Gaussian pyramid construction filter mask Repeat Filter Subsample Until minimum resolution reached can specify desired number of levels (e.g., 3-level pyramid) The whole pyramid is only 4/3 the size of the original image! Slide by Steve Seitz