1. Edges Edges = jumps in brightness/color Brightness jumps marked in white

2. Edges Edges = jumps in brightness/color Important! –Give object outlines and shapes Brightness jumps marked in white

3. Edges Edges = jumps in brightness/color Important! –Give object outlines and shapes –Relatively robust not so affected by changes in lighting, camera, viewpoint… Brightness jumps marked in white

4. Edges not so affected by lighting

5. Face outline persists, despite very different lighting

6. What do “edges” look like? Brightness Row number Brightness Column number “Jumps” can be gradual or jagged

7. Brightness jumps are not always obvious when you look at the brightness profile

8. Detecting edges Want to detect jumps brightness profiles at a jump (ideally)

9. Detecting edges Want to detect jumps brightness profiles at a jump (ideally) Of course should detect jumps in any orientation!

10. Detecting edges Want to detect jumps brightness profiles at a jump (ideally) Off course should detect jumps in any orientation!

11. Detecting edges Want to detect jumps Filters detect themselves brightness profiles at a jump (ideally) Eye detection: Convolve with eye filter

12. Detecting edges Want to detect jumps Filters detect themselves  To detect jumps, convolve image with jumps brightness profiles at a jump (ideally) Eye detection: Convolve with eye filter

13. To detect jumps, convolve with jumps Detecting edges

14. To detect jumps, convolve with jumps If image has no jump, convolving should give zero (convolving a constant image should give 0)  Filter weights should sum to 0 In equations: Detecting edges

15. To detect jumps, convolve with jumps Mask weights should sum to 0 To detect all jumps (dark-to-bright and bright-to-dark)… Detecting edges

16. To detect jumps, convolve with jumps Mask weights should sum to 0 To detect all jumps (dark-to-bright and bright-to-dark)…  Use magnitude (absolute value) of convolution result Detecting edges

17. Edge detection To detect jumps along x Convolve with [1, -1] To detect jumps along y Convolve with Vertical jumpsHorizontal jumps After taking absolute value of convolution result, only jumps stand out as bright spots. Smooth image regions, where the brightness changes slowly, are dark.

18. Edge Detection To “detect edges”, decide how big a jump must be to qualify –Choose threshold T –If magnitude of convolution is >T at some pixel, mark it as edge pixel (“edgel”) Marked (white) pixels have T is about 15% of Horizontal edges

19. Edge detection How to detect edges in any orientation? Answer: compute remember:

21. Edgels: T = 12% ofEdgels: T = 25% of Edgels: T = 6% of

22. Edgels: T = 12% ofEdgels: T = 25% of Edgels: T = 6% of Higher threshold: cleaner but more fragmented

23. Edge detection Many questions –We’re convolving with [-1 1]

24. Edge detection Many questions –We’re convolving with [-1 1] (Overall sign isn’t important…)

25. Edge detection Many questions –We’re convolving with [-1 1] = [... 0 0 0 -1 1 0 0 …]

26. Edge detection Many questions –We’re convolving with [-1 1]… Should we use [-1 -1 -1 -1 1 1 1 1 ] instead? (This would detect jumps between large bright areas next to large dark areas)

27. Edge detection Many questions –We’re convolving with [-1 1]… Should we use [-1 -1 -1 -1 1 1 1 1 ] instead? (This would detect jumps between large bright areas next to large dark areas) But not so good for

28. Edge detection Many questions –We’re convolving with [-1 1] –Should we look for smoother, more gradual change? [-0.02 -0.08 -0.06 0.06 0.08 0.02]

29. Edge detection Many questions –We’re convolving with [-1 1] –Should we look for smoother, more gradual change? [-0.02 -0.08 -0.06 0.06 0.08 0.02] Recall smooth brightness profile for car image edge

30. Edge detection Many questions –We’re convolving with [-1 1] –To detect long boundary, should we average along it?

31. Edge detection Many questions –We’re convolving with [-1 1] –To detect long boundary, should we average along it? 0 0 1 1 0 1 0 0 0 Detects jumps along x direction averages over y direction

32. Edge detection Many questions –We’re convolving with [-1 1] –To detect long boundary, should we average along it? 0 0 1 1 0 1 0 0 0 Detects jumps along x direction averages over y direction Good for long low contrast boundaries, accurate orientation measurement

33. Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise

34. Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise To detect fast change in image, compute derivative.

35. Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise To detect fast change, compute derivative.  same as convolving with [1 -1] !

36. Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise To detect fast change, compute derivative.  same as convolving with [1 -1] !

37. Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise To detect fast change, compute derivative.  same as convolving with [1 -1] !

38. Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise To detect fast change, compute derivative.  same as convolving with [1 -1] ! Remember the flip in convolution

39. Convolution and Derivatives Convolving I with [1, -1] gives Convolving I with gives Approximate derivatives in x and y

40. Convolution and Derivatives Convolving I with [1, -1] gives Convolving I with gives Define gradient of I Approximate derivatives in x and y

41. Convolution and Derivatives Convolving I with [1, -1] gives Convolving I with gives Define gradient of I –From calculus: Approximate derivatives in x and y

42. Convolution and Derivatives Convolving I with [1, -1] gives Convolving I with gives Define gradient of I –From calculus: –Approximate image gradient Approximate derivatives in x and y

43. What is the gradient? Change

44. What is the gradient? Change

45. What is the gradient? More Change Less Change

46. What is the gradient? More Change Less Change Gradient direction is perpendicular to edge

47. What is the gradient? More Change Less Change Gradient direction is perpendicular to edge Gradient Magnitude gives size of brightness jump

48. Gradient Brightness change in direction Chain rule:

49. Gradient Brightness change in direction Chain rule:

50. Gradient Brightness change in direction Chain rule:

51. Gradient Brightness change in direction Chain rule:  Can compute derivative in any direction from !

52. Gradient Brightness change in direction Chain rule: Dot product largest for parallel vectors…

53. Gradient Brightness change in direction Chain rule: Dot product largest for parallel vectors, so largest change (largest ) for movement along direction

54. Gradient Brightness change in direction Chain rule: Dot product largest for parallel vectors, so largest change (largest ) for movement along direction Size of this largest change:

55. Edge detection and gradient Conclusions Detect edges (big brightness jumps) by computing and thresholding Image lines perpendicular to

56. But image data is messy! Brightness profile Can we detect jump?

57. But image data is messy! Brightness profile Can we detect jump? Derivative (convolution with [1 -1] ) Where’s the jump???

58. Problem: Brightness fluctuates a lot –Fluctuations are fast (moderate jumps from one pixel to next) –Often transition at edge is smooth So on edge only moderate brightness jumps between pixels  Edge doesn’t stand out Derivative (convolution with [1 -1] ) Brightness profile

59. Problem –Fluctuations are fast –Transition at edge is smooth Solution –First smooth image to reduce fast fluctuations –Then take derivative Brightness profile Smoothed image Derivative (convolution with [1 -1] )

60. Brightness profile Derivative of smoothed image  Edge now stands out. Derivative (convolution with [1 -1] )

61. Smoothed edge detection Recommended strategy –Smooth first –Then convolve with “derivative” filters Smoothing filter (e.g., Gaussian) “derivative” filter (e.g., [1 -1]) image

62. Smoothed edge detection To make more efficient, use associativity of convolution “Derivative” of gaussian D and G are small, convolving them is fast. Also, can just compute once, then apply to many images.

63. Smoothing and Differentiation D * G Derivative of Gaussian filters Horizontal smoothed derivativeVertical smoothed derivative

64. Smoothing and Differentiation D * G Derivative of Gaussian filters Horizontal smoothed derivativeVertical smoothed derivative “ ”

65. Other Smoothed Derivative Kernels –Sobel kernels (more old fashioned) (These already incorporate smoothing) –Discrete directional derivatives 01 -202 01 121 000 -2 0 1 2 -1 0 1 -2 -1 0

66. Smoothed gradient As before, “smoothed gradient” gives “smoothed derivative" in any direction

67. Conclusions First smooth, then take derivative To get gradient, do this in x & y directions To detect edges, threshold magnitude of smoothed gradient

68. Conclusions First smooth, then take derivative To get gradient, do this in x & y directions To detect edges, threshold magnitude of smoothed gradient But how much smoothing should you do?? Depends, no single answer…

69. Amount of smoothing affects semantics of edges More smoothing gives: more significant edges, smoother curves, less noise, better detection… but less detail, worse edge localization 1 pixel 3 pixels 7 pixels Derivatives after smoothing at three different scales

71. Smoothing + localization More smoothing makes it harder to tell exactly where edges are located.

72. But median filter?

74. Threshold What should the threshold be?

75. Threshold What should the threshold be? –No single answer –Also depends on the amount of smoothing

76. Threshold What should the threshold be? –No single answer –Also depends on the amount of smoothing Smoothing reduces amount of change Original image (1D) Smoothed image

77. Threshold What should the threshold be? –No single answer –Also depends on the amount of smoothing Smoothing reduces amount of change Original image (1D) Smoothed image Note: transition blurs, Slope decreases

78. Threshold More smoothing  smaller derivative sizes Gradient magnitude shrinks Original image I (1D) Amount of smoothing (standard deviation of Gaussian)

79. Threshold More smoothing  smaller derivative sizes Gradient magnitude shrinks Original image I (1D) Amount of smoothing (standard deviation of Gaussian) Gradient size measured on the edge

80. Threshold Conclusion –For more smoothing, use lower threshold

82. fine scale high threshold

83. coarse scale, high threshold Using same threshold as before causes loss of detail

84. coarse scale low threshold

85. Origin of Edges Edges are caused by a variety of factors depth discontinuity surface color discontinuity illumination discontinuity surface normal discontinuity

86. Surface Boundaries

87. Sharp surface angles (normal discontinuity)

88. Change in material properties

89. Boundaries of lighting shadow

90. Edge Detection Issues: 1)What smoothing scale (ie, width of Gaussian) + threshold on ? 2) Identify true edge points? Usually > threshold along fat sausage containing true edge line 3) We are really interested in finding curves (that could outline an object) How to link detected edge pixels into curves? at two different scales

91. Edge detection so far 1.Filter out noise Smooth with 2D Gaussian 2.Take derivative

92. Edge detection so far 3. Find edges by thresholding

93. Finding the Peak above threshold on fat “sausage,” not a line Want to thin “sausage” of edge pixels to single curve Technique used is called Non Maximum Suppression

94. We wish to mark a curve within the sausage where the magnitude is biggest. We can do this by looking for a maximum along a slice normal to the curve (non-maximum suppression). These points should form a curve.

95. Non-maximum suppression At q, we have a maximum if the value is larger than those at both p and at r. Interpolate to get these values.

96. Edge Following 3) Link edge points into curves?

97. Predicting next edge point Edge point marked. 1) Look in direction perpendicular to gradient for next edge point 2) Check max of large enough

98. Hysteresis Drop-outs? –High threshold to start curves –Low threshold to continue.

101. longer edge curves!

102. Canny in MATLAB E = edge(Image,’canny’); or E = edge(Image,’canny’,Threshold, ); We still have to choose the smoothing scale and the threshold. By default, MATLAB takes. If you don’t provide the threshold, it tries to estimate a reasonable one.

103. Effect of  (for Gaussian smoothing) Canny with original The choice of depends on desired behavior –large detects large scale edges –small detects fine features

105. The Canny edge detector original image (Lena)

106. The Canny edge detector norm of the gradient

107. The Canny edge detector thresholding

108. The Canny edge detector thinning (non-maximum suppression)

109. Derivative filters: technicalities We use “derivative filters” to detect edges But images are really continuous functions Derivative filter should give good approximation to continuous derivative of original continuous image

110. Discrete Derivatives (1D) Let = pixel size. Convolving I with gives Relate to continuous derivative by Taylor expansion

111. Discrete Derivatives (1D) Better to use odd size kernel! For size 3, need Solve for derivative by subtracting…

112. Discrete Derivatives (1D) Better to use odd size kernel! For size 3, need Solve for derivative by subtracting… Derivative filter [-1/2 0 1/2]

113. Discrete Derivatives (1D) Better to use odd size kernel! For size 3, need Solve for derivative by subtracting… Derivative filter [-1/2 0 1/2] Now no O( ). Better approximation to derivative These even terms cancelled in the expression for

114. Message Use odd size filters (length 3, 5, …)

115. Second Derivative (1D) (size 3, need )

116. Add, divide through by

117. Second Derivative (1D) (size 3, need ) 2 nd derivative filter [1 -2 1] No O( ) since sum cancels odd powers of

118. Discrete Derivative (1D) Convolution filters (remember: flipped from correlation) First derivative: [1/2 0 -1/2] Second derivative: [1 -2 1] Note: [1 -2 1] = [1 -1] * [1 -1] first derivative filter

119. Discrete Derivative (1D) Convolution filters (remember: flipped from correlation) First derivative: [1/2 0 -1/2] Second derivative: [1 -2 1] Note: [1 -2 1] = [1 -1] * [1 -1] As usual, can compute “second derivative” of an image by taking two first “derivatives”…

120. Discrete Derivative (1D) Convolution filters (remember: flipped from correlation) First derivative: [1/2 0 -1/2] Second derivative: [1 -2 1] Note: [1 -2 1] = [1 -1] * [1 -1] First “derivative” filters, eg [-1 1] As usual, can compute “second derivative” of an image by taking two first “derivatives”… Second “derivative” filter

121. Discrete Derivative (1D) (size 5) For bigger filters, taylor expand to higher order First derivative filter (size 5):

122. Second derivative filters

123. is the Laplacian GaussianDerivative of Gaussian Example of 2 nd derivative filter: Laplacian of Gaussian

124. Second derivative filters is the Laplacian GaussianDerivative of Gaussian Example of 2 nd derivative filter: Laplacian of Gaussian Approximate 2 nd derivs

125. Second derivative filters is the Laplacian GaussianDerivative of Gaussian Example of 2 nd derivative filter: Laplacian of Gaussian

126. Second derivative filters is the Laplacian GaussianDerivative of Gaussian Example of 2 nd derivative filter: Laplacian of Gaussian Can approximate Laplacian of an image (or a Gaussian) by convolving with this filter

127. 2 nd Derivative Filters Note: can also use these to detect edges Image First derivative 2 nd derivative

128. 2 nd Derivative Filters Note: can also use these to detect edges Image First derivative 2 nd derivative Edge is inflection point: 2 nd derivative = 0

129. 2 nd Derivative Filters Note: can also use these to detect edges Can detect edges from zeros of 2 nd derivative Image First derivative 2 nd derivative Edge is inflection point: 2 nd derivative = 0

130. Using 2 nd Derivative Filters For 2D images, can use Laplacian-of- Gaussian to detect edges Laplacian of Gaussian

131. Laplacian at a pixel  pixel darker than average of its neighbors

132. Laplacian at a pixel  pixel darker than average of its neighbors Image I

133. Laplacian at a pixel  pixel darker than average of its neighbors Edge boundary between bright and dark regions Bright region Dark region surface plot of I Image I

134. Laplacian at a pixel  pixel darker than average of its neighbors Edge boundary between bright and dark regions Bright region Dark region surface plot of I Image I Pixels brighter than neighbors

135. Laplacian at a pixel  pixel darker than average of its neighbors Edge boundary between bright and dark regions Bright region Dark region surface plot of I Image I Pixels darker than neighbors

136. Laplacian at a pixel  pixel darker than average of its neighbors Edge boundary between bright and dark regions Bright region Dark region surface plot of I Image I On edge

137. Using 2 nd Derivative Filters For 2D images, can find edges by: –Convolve image with a Laplacian-of-Gaussian (ie, compute Laplacian of smoothed image) –Mark zero pixels (in practice, mark pixels on boundary between ‘+’ and ‘–’ regions) Since region boundaries are continuous curves, this always marks continuous curves!

138. Using 2 nd Derivative Filters For 2D images, can find edges by: –Convolve image with a Laplacian-of-Gaussian (ie, compute Laplacian of smoothed image) –Mark zero pixels (in practice, mark pixels on boundary between ‘+’ and ‘–’ regions) Since region boundaries are continuous curves, this always marks continuous curves! Not so good in practice

139. Laplacian at a pixel  pixel darker than average of its neighbors Edge detection: strategy 2

140. Laplacian at a pixel  pixel darker than average of its neighbors Edge detection: strategy 2 –When image is smooth, pixels have brightness similar to their neighbors and is small. –Near edges, I changes rapidly and can be big.

141. Laplacian at a pixel  pixel darker than average of its neighbors Edge detection: strategy 2 –When image is smooth, pixels have brightness similar to their neighbors and is small. –Near edges, I changes rapidly and can be big. So look for pixels where is large.

142. Laplacian at a pixel  pixel darker than average of its neighbors Edge detection: strategy 2 –When image is smooth, pixels have brightness similar to their neighbors and is small. –Near edges, I changes rapidly and can be big. So look for pixels where is large. (But this doesn’t get the position of the edge right…)

143. Laplacian at a pixel  pixel darker than average of its neighbors Edge boundary between bright and dark regions Bright region Dark region surface plot of I Image I Largest values for where surface most curved

144. Laplacian of Gaussian Gaussian Smaller Gaussian

145. Laplacian of Gaussian Why?

146. Laplacian of Gaussian Why?

147. Laplacian of Gaussian Why?

148. Laplacian of Gaussian Why? Convolution of an averaging filter with a Gaussian (which is another averaging filter)

149. Laplacian of Gaussian Why? Convolution of an averaging filter with a Gaussian (which is another averaging filter) Averaging twice is still averaging…. So roughly the same as convolving with a single, bigger Gaussian

150. Caution: D erivatives are noisy Higher derivatives are noisier First derivative: black 2 nd derivative: red

151. Caution: D erivatives are noisy Higher derivatives are noisier Why?

152. Caution: D erivatives are noisy Higher derivatives are noisier Why? Original signal I fluctuations

153. Caution: D erivatives are noisy Higher derivatives are noisier Why? Original signal Correlate with [-1 1] I I’ fluctuations Fluctuations double in size

154. Caution: D erivatives are noisy Higher derivatives are noisier Why? Original signal Correlate with [-1 1] Correlate again with [-1 1] (2 nd derivative) I I’ fluctuations Fluctuations double in size I’’ Fluctuations quadruple

155. Oriented Gaussian smoothing