Edges Edges = jumps in brightness/color Brightness jumps marked in white

Edges Edges = jumps in brightness/color Important! –Give object outlines and shapes Brightness jumps marked in white

Edges Edges = jumps in brightness/color Important! –Give object outlines and shapes –Relatively robust not so affected by changes in lighting, camera, viewpoint… Brightness jumps marked in white

Edges not so affected by lighting

Face outline persists, despite very different lighting

What do “edges” look like? Brightness Row number Brightness Column number “Jumps” can be gradual or jagged

Brightness jumps are not always obvious when you look at the brightness profile

Detecting edges Want to detect jumps brightness profiles at a jump (ideally)

Detecting edges Want to detect jumps brightness profiles at a jump (ideally) Of course should detect jumps in any orientation!

Detecting edges Want to detect jumps brightness profiles at a jump (ideally) Off course should detect jumps in any orientation!

Detecting edges Want to detect jumps Filters detect themselves brightness profiles at a jump (ideally) Eye detection: Convolve with eye filter

Detecting edges Want to detect jumps Filters detect themselves  To detect jumps, convolve image with jumps brightness profiles at a jump (ideally) Eye detection: Convolve with eye filter

To detect jumps, convolve with jumps Detecting edges

To detect jumps, convolve with jumps If image has no jump, convolving should give zero (convolving a constant image should give 0)  Filter weights should sum to 0 In equations: Detecting edges

To detect jumps, convolve with jumps Mask weights should sum to 0 To detect all jumps (dark-to-bright and bright-to-dark)… Detecting edges

To detect jumps, convolve with jumps Mask weights should sum to 0 To detect all jumps (dark-to-bright and bright-to-dark)…  Use magnitude (absolute value) of convolution result Detecting edges

Edge detection To detect jumps along x Convolve with [1, -1] To detect jumps along y Convolve with Vertical jumpsHorizontal jumps After taking absolute value of convolution result, only jumps stand out as bright spots. Smooth image regions, where the brightness changes slowly, are dark.

Edge Detection To “detect edges”, decide how big a jump must be to qualify –Choose threshold T –If magnitude of convolution is >T at some pixel, mark it as edge pixel (“edgel”) Marked (white) pixels have T is about 15% of Horizontal edges

Edge detection How to detect edges in any orientation? Answer: compute remember:

Edgels: T = 12% ofEdgels: T = 25% of Edgels: T = 6% of

Edgels: T = 12% ofEdgels: T = 25% of Edgels: T = 6% of Higher threshold: cleaner but more fragmented

Edge detection Many questions –We’re convolving with [-1 1]

Edge detection Many questions –We’re convolving with [-1 1] (Overall sign isn’t important…)

Edge detection Many questions –We’re convolving with [-1 1] = [ …]

Edge detection Many questions –We’re convolving with [-1 1]… Should we use [ ] instead? (This would detect jumps between large bright areas next to large dark areas)

Edge detection Many questions –We’re convolving with [-1 1]… Should we use [ ] instead? (This would detect jumps between large bright areas next to large dark areas) But not so good for

Edge detection Many questions –We’re convolving with [-1 1] –Should we look for smoother, more gradual change? [ ]

Edge detection Many questions –We’re convolving with [-1 1] –Should we look for smoother, more gradual change? [ ] Recall smooth brightness profile for car image edge

Edge detection Many questions –We’re convolving with [-1 1] –To detect long boundary, should we average along it?

Edge detection Many questions –We’re convolving with [-1 1] –To detect long boundary, should we average along it? Detects jumps along x direction averages over y direction

Edge detection Many questions –We’re convolving with [-1 1] –To detect long boundary, should we average along it? Detects jumps along x direction averages over y direction Good for long low contrast boundaries, accurate orientation measurement

Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise

Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise To detect fast change in image, compute derivative.

Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise To detect fast change, compute derivative.  same as convolving with [1 -1] !

Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise To detect fast change, compute derivative.  same as convolving with [1 -1] !

Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise To detect fast change, compute derivative.  same as convolving with [1 -1] !

Note: edge detection and derivative Original image: Blurry brightness transition First derivative derivative has maximum at point of steepest rise To detect fast change, compute derivative.  same as convolving with [1 -1] ! Remember the flip in convolution

Convolution and Derivatives Convolving I with [1, -1] gives Convolving I with gives Approximate derivatives in x and y

Convolution and Derivatives Convolving I with [1, -1] gives Convolving I with gives Define gradient of I Approximate derivatives in x and y

Convolution and Derivatives Convolving I with [1, -1] gives Convolving I with gives Define gradient of I –From calculus: Approximate derivatives in x and y

Convolution and Derivatives Convolving I with [1, -1] gives Convolving I with gives Define gradient of I –From calculus: –Approximate image gradient Approximate derivatives in x and y

What is the gradient? Change

What is the gradient? Change

What is the gradient? More Change Less Change

What is the gradient? More Change Less Change Gradient direction is perpendicular to edge

What is the gradient? More Change Less Change Gradient direction is perpendicular to edge Gradient Magnitude gives size of brightness jump

Gradient Brightness change in direction Chain rule:

Gradient Brightness change in direction Chain rule:

Gradient Brightness change in direction Chain rule:

Gradient Brightness change in direction Chain rule:  Can compute derivative in any direction from !

Gradient Brightness change in direction Chain rule: Dot product largest for parallel vectors…

Gradient Brightness change in direction Chain rule: Dot product largest for parallel vectors, so largest change (largest ) for movement along direction

Gradient Brightness change in direction Chain rule: Dot product largest for parallel vectors, so largest change (largest ) for movement along direction Size of this largest change:

Edge detection and gradient Conclusions Detect edges (big brightness jumps) by computing and thresholding Image lines perpendicular to

But image data is messy! Brightness profile Can we detect jump?

But image data is messy! Brightness profile Can we detect jump? Derivative (convolution with [1 -1] ) Where’s the jump???

Problem: Brightness fluctuates a lot –Fluctuations are fast (moderate jumps from one pixel to next) –Often transition at edge is smooth So on edge only moderate brightness jumps between pixels  Edge doesn’t stand out Derivative (convolution with [1 -1] ) Brightness profile

Problem –Fluctuations are fast –Transition at edge is smooth Solution –First smooth image to reduce fast fluctuations –Then take derivative Brightness profile Smoothed image Derivative (convolution with [1 -1] )

Brightness profile Derivative of smoothed image  Edge now stands out. Derivative (convolution with [1 -1] )

Smoothed edge detection Recommended strategy –Smooth first –Then convolve with “derivative” filters Smoothing filter (e.g., Gaussian) “derivative” filter (e.g., [1 -1]) image

Smoothed edge detection To make more efficient, use associativity of convolution “Derivative” of gaussian D and G are small, convolving them is fast. Also, can just compute once, then apply to many images.

Smoothing and Differentiation D * G Derivative of Gaussian filters Horizontal smoothed derivativeVertical smoothed derivative

Smoothing and Differentiation D * G Derivative of Gaussian filters Horizontal smoothed derivativeVertical smoothed derivative “ ”

Other Smoothed Derivative Kernels –Sobel kernels (more old fashioned) (These already incorporate smoothing) –Discrete directional derivatives

Smoothed gradient As before, “smoothed gradient” gives “smoothed derivative" in any direction

Conclusions First smooth, then take derivative To get gradient, do this in x & y directions To detect edges, threshold magnitude of smoothed gradient

Conclusions First smooth, then take derivative To get gradient, do this in x & y directions To detect edges, threshold magnitude of smoothed gradient But how much smoothing should you do?? Depends, no single answer…

Amount of smoothing affects semantics of edges More smoothing gives: more significant edges, smoother curves, less noise, better detection… but less detail, worse edge localization 1 pixel 3 pixels 7 pixels Derivatives after smoothing at three different scales

Smoothing + localization More smoothing makes it harder to tell exactly where edges are located.

But median filter?

Threshold What should the threshold be?

Threshold What should the threshold be? –No single answer –Also depends on the amount of smoothing

Threshold What should the threshold be? –No single answer –Also depends on the amount of smoothing Smoothing reduces amount of change Original image (1D) Smoothed image

Threshold What should the threshold be? –No single answer –Also depends on the amount of smoothing Smoothing reduces amount of change Original image (1D) Smoothed image Note: transition blurs, Slope decreases

Threshold More smoothing  smaller derivative sizes Gradient magnitude shrinks Original image I (1D) Amount of smoothing (standard deviation of Gaussian)

Threshold More smoothing  smaller derivative sizes Gradient magnitude shrinks Original image I (1D) Amount of smoothing (standard deviation of Gaussian) Gradient size measured on the edge

Threshold Conclusion –For more smoothing, use lower threshold

fine scale high threshold

coarse scale, high threshold Using same threshold as before causes loss of detail

coarse scale low threshold

Origin of Edges Edges are caused by a variety of factors depth discontinuity surface color discontinuity illumination discontinuity surface normal discontinuity

Surface Boundaries

Sharp surface angles (normal discontinuity)

Change in material properties

Boundaries of lighting shadow

Edge Detection Issues: 1)What smoothing scale (ie, width of Gaussian) + threshold on ? 2) Identify true edge points? Usually > threshold along fat sausage containing true edge line 3) We are really interested in finding curves (that could outline an object) How to link detected edge pixels into curves? at two different scales

Edge detection so far 1.Filter out noise Smooth with 2D Gaussian 2.Take derivative

Edge detection so far 3. Find edges by thresholding

Finding the Peak above threshold on fat “sausage,” not a line Want to thin “sausage” of edge pixels to single curve Technique used is called Non Maximum Suppression

We wish to mark a curve within the sausage where the magnitude is biggest. We can do this by looking for a maximum along a slice normal to the curve (non-maximum suppression). These points should form a curve.

Non-maximum suppression At q, we have a maximum if the value is larger than those at both p and at r. Interpolate to get these values.

Edge Following 3) Link edge points into curves?

Predicting next edge point Edge point marked. 1) Look in direction perpendicular to gradient for next edge point 2) Check max of large enough

Hysteresis Drop-outs? –High threshold to start curves –Low threshold to continue.

longer edge curves!

Canny in MATLAB E = edge(Image,’canny’); or E = edge(Image,’canny’,Threshold, ); We still have to choose the smoothing scale and the threshold. By default, MATLAB takes. If you don’t provide the threshold, it tries to estimate a reasonable one.

Effect of  (for Gaussian smoothing) Canny with original The choice of depends on desired behavior –large detects large scale edges –small detects fine features

The Canny edge detector original image (Lena)

The Canny edge detector norm of the gradient

The Canny edge detector thresholding

The Canny edge detector thinning (non-maximum suppression)

Derivative filters: technicalities We use “derivative filters” to detect edges But images are really continuous functions Derivative filter should give good approximation to continuous derivative of original continuous image

Discrete Derivatives (1D) Let = pixel size. Convolving I with gives Relate to continuous derivative by Taylor expansion

Discrete Derivatives (1D) Better to use odd size kernel! For size 3, need Solve for derivative by subtracting…

Discrete Derivatives (1D) Better to use odd size kernel! For size 3, need Solve for derivative by subtracting… Derivative filter [-1/2 0 1/2]

Discrete Derivatives (1D) Better to use odd size kernel! For size 3, need Solve for derivative by subtracting… Derivative filter [-1/2 0 1/2] Now no O( ). Better approximation to derivative These even terms cancelled in the expression for

Message Use odd size filters (length 3, 5, …)

Second Derivative (1D) (size 3, need )

Add, divide through by

Second Derivative (1D) (size 3, need ) 2 nd derivative filter [1 -2 1] No O( ) since sum cancels odd powers of

Discrete Derivative (1D) Convolution filters (remember: flipped from correlation) First derivative: [1/2 0 -1/2] Second derivative: [1 -2 1] Note: [1 -2 1] = [1 -1] * [1 -1] first derivative filter

Discrete Derivative (1D) Convolution filters (remember: flipped from correlation) First derivative: [1/2 0 -1/2] Second derivative: [1 -2 1] Note: [1 -2 1] = [1 -1] * [1 -1] As usual, can compute “second derivative” of an image by taking two first “derivatives”…

Discrete Derivative (1D) Convolution filters (remember: flipped from correlation) First derivative: [1/2 0 -1/2] Second derivative: [1 -2 1] Note: [1 -2 1] = [1 -1] * [1 -1] First “derivative” filters, eg [-1 1] As usual, can compute “second derivative” of an image by taking two first “derivatives”… Second “derivative” filter

Discrete Derivative (1D) (size 5) For bigger filters, taylor expand to higher order First derivative filter (size 5):

Second derivative filters

is the Laplacian GaussianDerivative of Gaussian Example of 2 nd derivative filter: Laplacian of Gaussian

Second derivative filters is the Laplacian GaussianDerivative of Gaussian Example of 2 nd derivative filter: Laplacian of Gaussian Approximate 2 nd derivs

Second derivative filters is the Laplacian GaussianDerivative of Gaussian Example of 2 nd derivative filter: Laplacian of Gaussian

Second derivative filters is the Laplacian GaussianDerivative of Gaussian Example of 2 nd derivative filter: Laplacian of Gaussian Can approximate Laplacian of an image (or a Gaussian) by convolving with this filter

2 nd Derivative Filters Note: can also use these to detect edges Image First derivative 2 nd derivative

2 nd Derivative Filters Note: can also use these to detect edges Image First derivative 2 nd derivative Edge is inflection point: 2 nd derivative = 0

2 nd Derivative Filters Note: can also use these to detect edges Can detect edges from zeros of 2 nd derivative Image First derivative 2 nd derivative Edge is inflection point: 2 nd derivative = 0

Using 2 nd Derivative Filters For 2D images, can use Laplacian-of- Gaussian to detect edges Laplacian of Gaussian

Laplacian at a pixel  pixel darker than average of its neighbors

Laplacian at a pixel  pixel darker than average of its neighbors Image I

Laplacian at a pixel  pixel darker than average of its neighbors Edge boundary between bright and dark regions Bright region Dark region surface plot of I Image I

Laplacian at a pixel  pixel darker than average of its neighbors Edge boundary between bright and dark regions Bright region Dark region surface plot of I Image I Pixels brighter than neighbors

Laplacian at a pixel  pixel darker than average of its neighbors Edge boundary between bright and dark regions Bright region Dark region surface plot of I Image I Pixels darker than neighbors

Laplacian at a pixel  pixel darker than average of its neighbors Edge boundary between bright and dark regions Bright region Dark region surface plot of I Image I On edge

Using 2 nd Derivative Filters For 2D images, can find edges by: –Convolve image with a Laplacian-of-Gaussian (ie, compute Laplacian of smoothed image) –Mark zero pixels (in practice, mark pixels on boundary between ‘+’ and ‘–’ regions) Since region boundaries are continuous curves, this always marks continuous curves!

Using 2 nd Derivative Filters For 2D images, can find edges by: –Convolve image with a Laplacian-of-Gaussian (ie, compute Laplacian of smoothed image) –Mark zero pixels (in practice, mark pixels on boundary between ‘+’ and ‘–’ regions) Since region boundaries are continuous curves, this always marks continuous curves! Not so good in practice

Laplacian at a pixel  pixel darker than average of its neighbors Edge detection: strategy 2

Laplacian at a pixel  pixel darker than average of its neighbors Edge detection: strategy 2 –When image is smooth, pixels have brightness similar to their neighbors and is small. –Near edges, I changes rapidly and can be big.

Laplacian at a pixel  pixel darker than average of its neighbors Edge detection: strategy 2 –When image is smooth, pixels have brightness similar to their neighbors and is small. –Near edges, I changes rapidly and can be big. So look for pixels where is large.

Laplacian at a pixel  pixel darker than average of its neighbors Edge detection: strategy 2 –When image is smooth, pixels have brightness similar to their neighbors and is small. –Near edges, I changes rapidly and can be big. So look for pixels where is large. (But this doesn’t get the position of the edge right…)

Laplacian at a pixel  pixel darker than average of its neighbors Edge boundary between bright and dark regions Bright region Dark region surface plot of I Image I Largest values for where surface most curved

Laplacian of Gaussian Gaussian Smaller Gaussian

Laplacian of Gaussian Why?

Laplacian of Gaussian Why?

Laplacian of Gaussian Why?

Laplacian of Gaussian Why? Convolution of an averaging filter with a Gaussian (which is another averaging filter)

Laplacian of Gaussian Why? Convolution of an averaging filter with a Gaussian (which is another averaging filter) Averaging twice is still averaging…. So roughly the same as convolving with a single, bigger Gaussian

Caution: D erivatives are noisy Higher derivatives are noisier First derivative: black 2 nd derivative: red

Caution: D erivatives are noisy Higher derivatives are noisier Why?

Caution: D erivatives are noisy Higher derivatives are noisier Why? Original signal I fluctuations

Caution: D erivatives are noisy Higher derivatives are noisier Why? Original signal Correlate with [-1 1] I I’ fluctuations Fluctuations double in size

Caution: D erivatives are noisy Higher derivatives are noisier Why? Original signal Correlate with [-1 1] Correlate again with [-1 1] (2 nd derivative) I I’ fluctuations Fluctuations double in size I’’ Fluctuations quadruple

Oriented Gaussian smoothing