1. Image formation ECE 847: Digital Image Processing Stan Birchfield Clemson University

2. Cameras First photograph due to Niepce Basic abstraction is the pinhole camera –lenses required to ensure image is not too dark –various other abstractions can be applied F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

3. Image formation overview Image formation involves geometry – path traveled by light radiometry – optical energy flow photometry – effectiveness of light to produce “brightness” sensation in human visual system colorimetry – physical specifications of light stimuli that produce given color sensation sensors – converting photons to digital form

4. Pinhole camera D. Forsyth, http://luthuli.cs.uiuc.edu/~daf/book/bookpages/slides.html

5. Parallel lines meet: vanishing point each set of parallel lines (=direction) meets at a different point –The vanishing point for this direction Sets of parallel lines on the same plane lead to collinear vanishing points. –The line is called the horizon for that plane

6. Perspective projection k O P Q j i p q C f Properties of projection: Points go to points Lines go to lines Planes go to whole image Polygons go to polygons Degenerate cases –line through focal point to point –plane through focal point to line F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

7. Perspective projection (cont.) F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

8. Weak perspective projection perspective effects, but not over the scale of individual objects collect points into a group at about the same depth, then divide each point by the depth of its group D. Forsyth, http://luthuli.cs.uiuc.edu/~daf/book/bookpages/slides.html

9. Weak perspective (cont.) F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

10. Orthographic projection Let Z 0 =1: F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

11. Pushbroom cameras

12. Pinhole too big - many directions are averaged, blurring the image Pinhole too small- diffraction effects blur the image Generally, pinhole cameras are dark, because a very small set of rays from a particular point hits the screen. Pinhole size D. Forsyth, http://luthuli.cs.uiuc.edu/~daf/book/bookpages/slides.html

13. The reason for lenses D. Forsyth, http://luthuli.cs.uiuc.edu/~daf/book/bookpages/slides.html

14. The thin lens focal points D. Forsyth, http://luthuli.cs.uiuc.edu/~daf/book/bookpages/slides.html

15. Focusing http://www.theimagingsource.com

16. Thick lens thick lens has 6 cardinal points: –two focal points (F 1 and F 2 ) –two principal points (H 1 and H 2 ) –two nodal points (N 1 and N 2 ) complex lens is formed by combining individual concave and convex lenses http://physics.tamuk.edu/~suson/html/4323/thick.html D. Forsyth, http://luthuli.cs.uiuc.edu/~daf/book/bookpages/slides.html

17. Complex lens http://www.cambridgeincolour.com/tutorials/camera-lenses.htm All but the simplest cameras contain lenses which are actually composed of several lens elements

18. Choosing a lens How to select focal length: –x=fX/Z –f=xZ/X Lens format should be >= CCD format to avoid optical flaws at the rim of the lens http://www.theimagingsource.com/en/resources/whitepapers/download/choosinglenswp.en.pdf

19. Lenses – Practical issues standardized lens mount has two varieties: –C mount –CS mount CS mount lenses cannot be used with C mount cameras http://www.theimagingsource.com/en/resources/whitepapers/download/choosinglenswp.en.pdf

20. Spherical aberration perfect lensactual lens On a real lens, even parallel rays are not focused perfectly http:// en.wikipedia.org/wiki/Spherical_aberration

21. Chromatic aberration On a real lens, different wavelengths are not focused the same http://en.wikipedia.org/wiki/Chromatic_aberration

22. Radial distortion straight lines are curved: uncorrectedcorrected

23. Radial distortion (cont.) barrel distortion (more common) pincushion distortion http://en.wikipedia.org/wiki/Image_distortion pincushion barrel http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpg Two types:

24. Vignetting vignetting – reduction of brightness at periphery of image D. Forsyth, http://luthuli.cs.uiuc.edu/~daf/book/bookpages/slides.html

25. Normalized Image coordinates P O u=X/Z = dimensionless ! 1 F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

26. Pixel units P O u=k f X/Z = in pixels ! [f] = m (in meters) [k] = pixels/m f Pixels are on a grid of a certain dimension F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

27. Pixel coordinates P O u=u 0 + k f X/Z f We put the pixel coordinate origin on topleft F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

28. Pixel coordinates in 2D j i (u 0,v 0 ) (0.5,0.5) 640 480 (640.5,480.5) F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

29. Summary: Intrinsic Calibration skew 5 Degrees of Freedom ! F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

30. Camera Pose In order to apply the camera model, objects in the scene must be expressed in camera coordinates. World Coordinates Camera Coordinates Calibration target looks tilted from camera viewpoint. This can be explained as a difference in coordinate systems. F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

31. Rigid Body Transformations Need a way to specify the six degrees-of-freedom of a rigid body. Why are there 6 DOF? A rigid body is a collection of points whose positions relative to each other can’t change Fix one point, three DOF Fix second point, two more DOF (must maintain distance constraint) Third point adds one more DOF, for rotation around line F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

32. Notations Superscript references coordinate frame A P is coordinates of P in frame A B P is coordinates of P in frame B Example: F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

33. Translation F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

34. Translation Using homogeneous coordinates, translation can be expressed as a matrix multiplication. Translation is commutative F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

35. Rotation means describing frame A in The coordinate system of frame B F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

36. Rotation Orthogonal matrix! F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

37. Example: Rotation about z axis What is the rotation matrix? F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

38. Rotation in homogeneous coordinates Using homogeneous coordinates, rotation can be expressed as a matrix multiplication. Rotation is not communicative F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

39. Rigid transformations F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

40. Rigid transformations (con ’ t) Unified treatment using homogeneous coordinates. F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

41. Projective Camera Matrix 5+6 DOF = 11 ! F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

42. Projective Camera Matrix 5+6 DOF = 11 ! F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

43. Columns & Rows of M m 2 P=0 O F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

44. Effect of Illumination Light source strength and direction has a dramatic impact on distribution of brightness in the image (e.g. shadows, highlights, etc.) (Subject 8 from the Yale face database due to P. Belhumeur et. al.) F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

45. Image formation Light source emits photons Absorbed, transmitted, scattered fluorescence Camera source F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

46. Surfaces receives and emits Incident light from lightfield Act as a light source How much light ? F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

47. Irradiance Irradiance – amount of light falling on a surface patch symbol=E, units = W/m 2 dA F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

48. Radiosity power leaving a point per area symbol=B, units = W/m 2 dA F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

49. Light = Directional Light emitted varies w. direction F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

50. Steradians (Solid Angle) 3D analogue of 2D angle A R F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

51. Steradians (cont’d) F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

52. Polar Coordinates F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

53. Intensity Intensity – amount of light emitted from a point per steradian symbol=I, units = W/sr F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

54. Irradiance and Intensity dA F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

55. Radiance Radiance – amount of light passing through an area dA and symbol=L, units = W x m -2 x sr -1 F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

56. Radiance is important Response of camera/eye is proportional to radiance Pixel values Constant along a ray F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

57. Lightfield = Gibson optic array ! 5DOF: Position = 3DOF, 2 DOF for direction F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

58. Lightfield Sampler F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

59. Lightfield Sample F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

60. Lambertian Emitters Lambertian = constant radiance More photons emitted straight up Oblique: see fewer photons, but area looks smaller Same brightness ! Total power is proportional to wedge area “Cosine law” Sun approximates Lambertian: Different angle, same brightness Moon should be less bright at edges, as gets less light from sun. Reflects more light at grazing angles than a Lambertian reflector F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

61. Radiance Emitted/Reflected Radiance – amount of light emitted from a surface patch per steradian per area foreshortened ! dA F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

62. Calculating Radiosity If reflected light is not dependent on angle, then can integrate over angle: radiosity is an approximate radiometric unit F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

63. Example: Sun Power= 3.91 10 26 W Surface Area:6.07 10 18 m 2 Power = Radiance. Area.  L = 2.05 10 7 W/m 2.sr Example from P. Dutre SIGGRAPH tutorial F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

64. Irradiance (again) Integrate incoming radiance over hemisphere F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

65. Example: Sun F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

66. BRDF L E Symmetric in incoming and outgoing directions – this is the Helmholtz reciprocity principle F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

67. BRDF Example F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

68. Lambertian surfaces and albedo For some surfaces, the DHR is independent of illumination direction too –cotton cloth, carpets, matte paper, matte paints, etc. For such surfaces, radiance leaving the surface is independent of angle Called Lambertian surfaces (same Lambert) or ideal diffuse surfaces Use radiosity as a unit to describe light leaving the surface DHR is often called diffuse reflectance, or albedo for a Lambertian surface, BRDF is independent of angle, too. Useful fact: F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

69. Specular surfaces Another important class of surfaces is specular, or mirror-like. –radiation arriving along a direction leaves along the specular direction –reflect about normal –some fraction is absorbed, some reflected –on real surfaces, energy usually goes into a lobe of directions –can write a BRDF, but requires the use of funny functions F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

70. Lambertian + specular Widespread model –all surfaces are Lambertian plus specular component Advantages –easy to manipulate –very often quite close true Disadvantages –some surfaces are not e.g. underside of CD’s, feathers of many birds, blue spots on many marine crustaceans and fish, most rough surfaces, oil films (skin!), wet surfaces –Generally, very little advantage in modelling behaviour of light at a surface in more detail -- it is quite difficult to understand behaviour of L+S surfaces F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

71. Radiometry vs. Photometry http://www.optics.arizona.edu/Palmer/rpfaq/rpfaq.htm F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

72. Sensors CCD vs. CMOS Types of CCDs: linear, interline, full- frame, frame-transfer Bayer filters progressive scan vs. interlacing NTSC vs. PAL vs. SECAM framegrabbers blooming F. Dellaert, http://www.cc.gatech.edu/~dellaert/vision/html/materials.html

73. Bayer color filter http://en.wikipedia.org/wiki/Bayer_filter