1. Image Formation and Capture Acknowledgment: some figures by B. Curless, E. Hecht, W.J. Smith, B.K.P. Horn, and A. Theuwissen

2. Image Formation and Capture DevicesDevices Sources of ErrorSources of Error Real world OpticsOpticsSensorSensor

3. Optics Pinhole cameraPinhole camera LensesLenses Focus, aperture, distortionFocus, aperture, distortion

4. Pinhole Object Pinhole camera Pinhole Camera “Camera obscura” – known since antiquity“Camera obscura” – known since antiquity Image Image plane

5. Pinhole Camera “Camera obscura” – known since antiquity“Camera obscura” – known since antiquity First recording in 1826 onto a pewter plate (by Joseph Nicéphore Niepce)First recording in 1826 onto a pewter plate (by Joseph Nicéphore Niepce) Pinhole Object Image Pinhole camera Image plane

6. Pinhole Camera Limitations Aperture too big: blurry imageAperture too big: blurry image Aperture too small: requires long exposure or high intensityAperture too small: requires long exposure or high intensity Aperture much too small: diffraction through pinhole  blurry imageAperture much too small: diffraction through pinhole  blurry image

7. Lenses Focus a bundle of rays from a scene point onto a single point on the imagerFocus a bundle of rays from a scene point onto a single point on the imager Result: can make aperture biggerResult: can make aperture bigger

8. Camera Adjustments Iris?Iris? – Changes aperture Focus?Focus? – Changes d i Zoom?Zoom? – Changes f and sometimes d i

9. Zoom Lenses – Varifocal

10. Zoom Lenses – Parfocal

11. For a given d i, “perfect” focus at only one d oFor a given d i, “perfect” focus at only one d o In practice, OK for some range of depthsIn practice, OK for some range of depths – Circle of confusion smaller than a pixel Better depth of field with smaller aperturesBetter depth of field with smaller apertures – Better approximation to pinhole camera Also better depth of field with wide-angle lensesAlso better depth of field with wide-angle lenses Focus and Depth of Field

12. Field of View Q: What does field of view of camera depend on?Q: What does field of view of camera depend on? – Focal length of lens – Size of imager – Object distance?

13. Computing Field of View xoxo xoxo didi didi xixi xixi dodo dodo 1/ d o + 1/ d i = 1/ f   x o / d o = x i / d i  = 2 tan -1 ½ x i (1/ f  1/ d o ) Since typically d o >> f,   2 tan -1 ½ x i / f tan  /2 = ½ x o / d o   x i / f

14. Aperture Controls amount of lightControls amount of light Affects depth of fieldAffects depth of field Affects distortion (since thin-lens approximation is better near center of lens)Affects distortion (since thin-lens approximation is better near center of lens)

15. Aperture Aperture typically given as “ f -number” (also “ f -stops” or just “stops”)Aperture typically given as “ f -number” (also “ f -stops” or just “stops”) What is f /4?What is f /4? – Aperture is ¼ the focal length

16. Sensors FilmFilm VidiconVidicon CCDCCD CMOSCMOS

17. Vidicon Best-known in family of “photoconductive video cameras”Best-known in family of “photoconductive video cameras” Basically television in reverseBasically television in reverse Electron Gun Photoconductive Plate Lens System + +               Scanning Electron Beam

18. MOS Capacitors MOS = Metal Oxide SemiconductorMOS = Metal Oxide Semiconductor Gate (wire) SiO 2 (insulator) p -type silicon

19. MOS Capacitors Voltage applied to gate repels positive “holes” in the semiconductorVoltage applied to gate repels positive “holes” in the semiconductor +10V Depletion region (electron “bucket”) + + + + + +

20. MOS Capacitors Photon striking the material creates electron-hole pairPhoton striking the material creates electron-hole pair +10V + + + + + + Photon   + +            

21. Charge Transfer Can move charge from one bucket to another by manipulating voltagesCan move charge from one bucket to another by manipulating voltages

22. CMOS Imagers Recently, can manufacture chips that combine photosensitive elements and processing elementsRecently, can manufacture chips that combine photosensitive elements and processing elements Benefits:Benefits: – Partial readout – Signal processing – Eliminate some supporting chips  low cost

23. Color 3-chip vs. 1-chip: quality vs. cost3-chip vs. 1-chip: quality vs. cost

24. Errors in Digital Images What are some sources of error in this image?What are some sources of error in this image?

25. Sources of Error Geometric (focus, distortion)Geometric (focus, distortion) Color (1-chip artifacts, chromatic aberration)Color (1-chip artifacts, chromatic aberration) Radiometric (cosine falloff, vignetting)Radiometric (cosine falloff, vignetting) Bright areas (flare, bloom, clamping)Bright areas (flare, bloom, clamping) Signal processing (gamma, compression)Signal processing (gamma, compression) NoiseNoise

26. Monochromatic Aberrations Real lenses do not follow thin lens approximation because surfaces are spherical (manufacturing constraints)Real lenses do not follow thin lens approximation because surfaces are spherical (manufacturing constraints) Result: thin-lens approximation only valid iff sin   Result: thin-lens approximation only valid iff sin   

27. Spherical Aberration Results in blurring of image, focus shifts when aperture is stopped downResults in blurring of image, focus shifts when aperture is stopped down Can vary with the way lenses are orientedCan vary with the way lenses are oriented

28. Distortion Pincushion or barrel radial distortionPincushion or barrel radial distortion Varies with placement of apertureVaries with placement of aperture

29. Distortion

30. Distortion

31. Distortion

32. First-Order Radial Distortion Goal: mathematical formula for distortionGoal: mathematical formula for distortion If small, can be approximated by “first-order” formula (like Taylor series expansion):If small, can be approximated by “first-order” formula (like Taylor series expansion): Higher-order models are possibleHigher-order models are possible r ’ = r (1 +  r 2 ) r = ideal distance to center of image r ’ = distorted distance to center of image

33. Chromatic Aberration Due to dispersion in glass (focal length varies with the wavelength of light)Due to dispersion in glass (focal length varies with the wavelength of light) Result: color fringesResult: color fringes Worst at edges of imageWorst at edges of image Correct by building lens systems with multiple kinds of glassCorrect by building lens systems with multiple kinds of glass

34. Correcting for Aberrations High-quality compound lenses use multiple lens elements to “cancel out” distortion and aberrationHigh-quality compound lenses use multiple lens elements to “cancel out” distortion and aberration Often 5-10 elements, more for extreme wide angleOften 5-10 elements, more for extreme wide angle

35. Other Limitations of Lenses Optical vignetting: less power per unit area transferred for light at an oblique angleOptical vignetting: less power per unit area transferred for light at an oblique angle – Transferred power falls off as cos 4  – Result: darkening of edges of image Mechanical vignetting: due to aperturesMechanical vignetting: due to apertures

36. Other Limitations of Lenses Flare: light reflecting (often multiple times) from glass-air interfaceFlare: light reflecting (often multiple times) from glass-air interface – Results in ghost images or haziness – Worse in multi-lens systems – Ameliorated by optical coatings (thin-film interference)

37. Bloom Overflow of charge in CCD bucketsOverflow of charge in CCD buckets – Spills to adjacent buckets – Streaks (usually vertical) next to bright areas Some cameras have “anti-bloom” circuitrySome cameras have “anti-bloom” circuitry

38. Flare and Bloom Tanaka

39. Dynamic Range Most common cameras have 8-bit (per color channel) dynamic rangeMost common cameras have 8-bit (per color channel) dynamic range – With gamma, this can translate to more than 255:1 Too bright: clamp to maximumToo bright: clamp to maximum Too dim: clamp to 0Too dim: clamp to 0 Specialty cameras with higher dynamic range (usually 10-, 12-, and 16-bit)Specialty cameras with higher dynamic range (usually 10-, 12-, and 16-bit)

40. High Dynamic Range (HDR) from Ordinary Cameras Take pictures of same scene with different shutter speedsTake pictures of same scene with different shutter speeds Identify regions clamped to 0 or 255Identify regions clamped to 0 or 255 Average other pixels, scaled by 1 / shutter speedAverage other pixels, scaled by 1 / shutter speed Can extend dynamic range, but limitations of optics and imager (noise, flare, bloom) still applyCan extend dynamic range, but limitations of optics and imager (noise, flare, bloom) still apply

41. Gamma Vidicon tube naturally has signal that varies with light intensity according to a power law: Signal = E ,   1/2.5Vidicon tube naturally has signal that varies with light intensity according to a power law: Signal = E ,   1/2.5 CRT (televisions) naturally obey a power law with gamma  2.5CRT (televisions) naturally obey a power law with gamma  2.5 Result: standard for video signals has a gamma of 1/2.5Result: standard for video signals has a gamma of 1/2.5 CCDs and CMOS linear, but gamma often appliedCCDs and CMOS linear, but gamma often applied

42. Noise Thermal noise: in all electronicsThermal noise: in all electronics – Noise at all frequencies – Proportional to temperature – Special cooled cameras available for low noise Shot noise: discrete photons / electronsShot noise: discrete photons / electrons – Shows up at extremely low intensities – CCDs / CMOS can have high efficiency – approaching 1 electron per photon

43. Noise 1/f noise – inversely proportional to frequency1/f noise – inversely proportional to frequency – Not completely understood – shows up in semiconductors – Can be dominant source of noise All of the above apply for imager and amplifierAll of the above apply for imager and amplifier

44. Filtering Noise Most common method – simple blurMost common method – simple blur – e.g., convolution with Gaussian Adaptive filters to prevent bleed across intensity edgesAdaptive filters to prevent bleed across intensity edges Other filters for specialized situationsOther filters for specialized situations – e.g., “despeckling” (median filters) for dead pixels