1. ELE 488 F06 ELE 488 Fall 2006 Image Processing and Transmission 09-26-06 Linear 2-D Image Filtering 1-D discrete convolution 2-D discrete convolution 2-D spatial masks Mask filtering Mask filtering and 2-D convolution Spatial Averaging, Blurring, Image Sharpening Edge Map - gradient 9/26/06

2. ELE 488 F06 Signal Processing output signal y[n] input signal x[n] y[n] = Σ h[k] x[n – k], convolution Processor LTI system Example: h[ ] [ 0.25 0.5 0.25 ], y[n] = 0.25 x[n–1] + 0.5 x[n] + 0.25 x[n–1] weighted sum of input Commutative: A * B = B * A Associative: (A * B ) * C = A * (B * C) Distributive: A * (B + C) = A * B + A * C

3. ELE 488 F06 2D Signal Processing (image) output signal y[m,n] input signal x[m,n] y[m,n] = Σ h[k, j] x[m – k, n – j], 2D convolution Processor LSI system Example: h[ ]: y[m,n] = 0.25 x[m,n] + 0.25 x[m–1, n] + 0.25 x[m, n–1] + 0.25 x[m–1, n–1] weighted sum of 4 pixels

4. ELE 488 F06 0101 0 10 0 0 0 1 01 0 10 10 10 0 0 0 1 01 0 10 10 10 0 0 0 1 01 0 10 0 0 0 1 01 0 10 0 0 0 1 10 Rotate 180 Do same for next row x y h Example: y[m,n] = Σ h[k, j] x[m – k, n – j]

5. ELE 488 F06 Spatial Masks Spatial Mask Finite, 2-D, real valued, array. One element is designated as (0,0). Used to compute the weighted sum of an input image, where weights are specified by the mask. Usually has small support 2x2, 3x3, 5x5, 7x7 Examples: These examples are spatial averaging masks. Used for image smoothing, LPF before subsampling (anti-aliasing), etc. 1/4 0 1 0101 1/9 -1 0 1 0 1 1/9 01/8 1/2 -1 0 1 0 1 0 1/8 0 0

6. ELE 488 F06 Applying a 2X2 Spatial Mask to an Image (1) 1/4 Input ImageOutput Image 1/4

7. ELE 488 F06 Applying a 2X2 Spatial Mask to an Image (2) 1/4 Input ImageOutput Image Shift mask over input image = Shift input image under mask 1/4

8. ELE 488 F06 Applying a 2X2 Spatial Mask to an Image (3) 1/4 Input Image Output Image 1/4

9. ELE 488 F06 Applying a 2X2 Spatial Mask to an Image (4) 1/4 Input Image Output Image 1/4 Edge effect: what to do?

10. ELE 488 F06 Applying a 2X2 Spatial Mask to an Image (5) 1/4 Input Image Output Image 1/4

11. ELE 488 F06 Applying a 3X3 Spatial Mask to an Image (1) Input Image Output Image -1/4 1 00 00 1 00 00 mask

12. ELE 488 F06 Applying a 3X3 Spatial Mask to an Image (2) Input Image Output Image -1/4 1 00 00 1 00 00 mask

13. ELE 488 F06 Applying a 3X3 Spatial Mask to an Image (3) Input Image Output Image -1/4 1 00 00 1 00 00 mask

14. ELE 488 F06 Spatial Masks Spatial Mask Finite, 2-D, real valued, array. One element is designated as (0,0). Used to compute the weighted sum of an input image, where weights are specified by the mask. Usually has small support 2x2, 3x3, 5x5, 7x7 Examples: These examples are spatial averaging masks. Used for image smoothing, LPF before subsampling (anti-aliasing), etc. 1/4 0 1 0101 1/9 -1 0 1 0 1 1/9 01/8 1/2 -1 0 1 0 1 0 1/8 0 0

15. ELE 488 F06 Spatial Averaging Masks For softening, noise smoothing, LPF before subsampling(anti-aliasing), etc. 1/4 0 1 0101 1/9 -1 0 1 0 1 1/9 01/8 1/2 -1 0 1 0 1 0 1/8 0 0 UMCP ENEE408G/631 Slides (created by M.Wu & R.Liu © 2002/2004)

16. ELE 488 F06 Spatial Averaging Masks 1/4 0 1 0101 UMCP ENEE408G/631 Slides (created by M.Wu & R.Liu © 2002/2004)

17. ELE 488 F06 Spatial Averaging Masks 1/9 -1 0 1 0 1 1/9 UMCP ENEE408G/631 Slides (created by M.Wu & R.Liu © 2002/2004)

18. ELE 488 F06 Spatial Averaging Masks 01/8 1/2 -1 0 1 0 1 0 1/8 0 0 UMCP ENEE408G/631 Slides (created by M.Wu & R.Liu © 2002/2004)

19. ELE 488 F06 Spatial Averaging Masks 1/9 -1 0 1 0 1 1/9 01/8 1/2 -1 0 1 0 1 0 1/8 0 0 UMCP ENEE408G/631 Slides (created by M.Wu & R.Liu © 2002/2004)

20. ELE 488 F06 The Equations Let x(m,n) denote the input image, M(m,n) denote the mask and y(m,n) denote the output image. Then for each pixel (m,n) in the output image: where the indices i and j range over the extent of the mask. This corresponds to shifting the image under the mask. This equation assumes values are provided in the edge regions. 1/4 0 1 0101 1/9 -1 0 1 0 1 1/9 01/8 1/2 -1 0 1 0 1 0 1/8 0 0

21. ELE 488 F06 The Equations, cont. Alternatively we can shift the mask over the image. In this case, for each pixel (m,n) in the output image: If we pad the input image and the mask with zeros so that they are both of infinite extent, then we can write:

22. ELE 488 F06 Point Spread Function For linear shift invariant (LSI) processing, the impulse response h[m,n] is called the point spread function : Delta function (point) in, point spread function out

23. ELE 488 F06 2-D Masking = Convolution with a PSF Mask Filter: Convolution: Applying a mask filter M to an image is equivalent to convolution with the PSF h with : M(I, j) = h(– i, – j)

24. ELE 488 F06 The transformation is equivalent to a rotation by 180 degrees. Or flip on both indeces. PSF to Mask 01/8 1/2 -1 0 1 0 1 0 1/8 0 0 0 1/2 -1 0 1 0 1 0 1/8 0 0 0 -20 1 2 01 10 20 -2 10 0110 01 0 0 10 Notation: Hence:

25. ELE 488 F06 Filtering an Image with a Mask Filtering an image with a mask is a linear shift-invariant operation. Hence: Masks can be applied in any order Masks applied sequentially can first be combined into one mask. Exercise: 0101 0 10 0 0 0 1

26. ELE 488 F06 LPF, HPF, & BPF Spatial averaging filters (usually positive entries in mask) are Low Pass Filters (LPF) HPF and BPF can be constructed from LPF –High-pass filter h HP (m,n) =  (m,n) – h LP (m,n) Useful in edge extraction and image sharpening –Band-pass filter h BP (m,n) = h L2 (m,n) – h L1 (m,n) Useful in edge enhancement Also good for high-pass tasks in the presence of noise –avoid amplifying high-frequency noise

27. ELE 488 F06 Example: Image Sharpening B(m,n) = A(m,n) +a A(m,n) x M(m,n) Original circuit image is from Matlab Image Toolbox. 0-¼ 1 -1 0 1 0 1 0 -¼ 0 0

28. ELE 488 F06 Image Sharpening: Edge Enhancement Use LPF to generate HPF –Subtract a low pass filtered image I LP from the original I 0 –HPF extracts edges and transitions Enhance edges Edge I0I0 I HP I LP ISIS

29. ELE 488 F06 Example of Image Sharpening Original moon image is from Matlab Image Toolbox. UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

30. ELE 488 F06 Gradient, 1 st -order Derivatives, and Edges Edge: where luminance changes abruptly For binary image –Take black pixels with immediate white neighbors as edge pixel For continuous-tone image –Find luminance gradient partial derivatives along two orthogonal directions gives the direction with highest rate of changes If gradient is larger than a threshold => edge –To represent edge Edge map ~ edge intensity & directions UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

31. ELE 488 F06 The Gradient of Function f(x,y) The gradient of a smooth function f(x,y) at (x,y) is derivative of f in the x-direction, partial derivative of f w.r.t. x derivative of f in the y-direction, partial derivative of f w.r.t. y 2-vector pointing in the direction of greatest increase of f fr. rate of change of f in the direction h f

32. ELE 488 F06 Example

33. ELE 488 F06 Gradient of Example Function Gradient is a vector field: a 2-vector associated with each point (x,y)

34. ELE 488 F06 Gradient of Discrete Function (Symmetric Difference) The symmetric difference of a real-valued discrete function f [n] at the point j is The symmetric rate of change of f around the point j

35. ELE 488 F06 The Discrete Gradient The gradient of a real-valued discrete function f(m,n) at (j.k) is The symmetric rate of change of f in the m-direction at (j,k) The symmetric rate of change of f in the n-direction at (j,k) (Note: for simplicity, drop the scale factor of ½ - it can always be reinserted again if needed.)

36. ELE 488 F06 The Gradient Vector Field (GVF) The gradient at pixel is the 2 vector The gradient assigns a 2-vector at each pixel. This is called the gradient vector field. Illustration of the gradient vector field showing a scaled version of the gradient at every 3 rd pixel. The red lines illustrate the direction and magnitude of the gradient vector at each pixel. Note: the gradient points in the direction of increasing image values.

37. ELE 488 F06 The Discrete Gradient Filtering with the mask [-1 0 1] yields: Filtering with the mask [-1 0 1] T yields: The gradient can be computed by applying two LSI filters

38. ELE 488 F06 Averaging GVF gradient is computed using high pass filters (discrete differentiation)  will amplify noise. So average neighboring gradient vectors using a spatial filtering on each of two gradient components. The averaging filters and gradient filters are LSI and hence can be applied in any order, i.e., Can compute gradient and average, or Average pixels and then compute gradient. Should we average uniformly in all directions or along a selected direction? e.g. along the perpendicular to the direction of the derivative? Perpendicular to the gradient?

39. ELE 488 F06 Averaging the GVF Spatial averaging filter perpendicular to direction of discrete derivative discrete derivative filter in horizontal direction Combination of the two masks gives single “averaged gradient mask” in horizontal direction.

40. ELE 488 F06 Common GVF Masks –Apply both masks to image and combine results to determine magnitude and angle (if desired). –Note: Prewitt and Sobel operators spatially average the horizontal and vertical differences of 3 local pixels to reduce the effect of noise 01 0 0 0 1 1 01 0 -20 1 2 01 10 0 00 0 111 -2 00 0 121 Roberts PrewittSobel

41. ELE 488 F06 Averaged GVF and Edges Scaled & averaged gradient of apple image plotted at every 3 rd pixel. Large gradient magnitude Edge: Normal to direction of gradient Small gradient magnitude No edge Subjective Issue: How to select the threshold on the magnitude of the gradient to “detect” an edge?

42. ELE 488 F06 Edge Detection: Summary Edge: pixel locations of abrupt intensity change For binary image – black pixels with immediate white neighbors For continuous-tone image, measure gradient vector –Along two orthogonal directions ~ usually horizontal and vertical g x =  L /  x, g y =  L /  y –Magnitude of gradient vector g(m,n) 2 = g x (m,n) 2 + g y (m,n) 2 OR g(m,n) = |g x (m,n) | + |g y (m,n)| (preferred in hardware implement.) –Direction of gradient vector tan –1 [ g y (m,n) / g x (m,n) ] Characterizing edges in an image –(binary) Edge map: specify “edge point” locations with g(m,n) > thresh. –Edge intensity map: specify gradient magnitude at each pixel –Edge direction map: specify directions UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

43. ELE 488 F06 Example: Edge Detection from GVF Threshold magnitude of the GVF: Above threshold: edge pixel Below threshold: no edge

44. ELE 488 F06 Sobel GVF Sobel Prewitt

45. ELE 488 F06 Example: Edge Detectors Based on the GVF RobertsPrewittSobel Threshold the magnitude of the GVF: Above threshold – pixel is on an “edge” Below threshold – no “edge”

46. ELE 488 F06 Examples of Edge Detectors –Quantize edge intensity to 0/1: set a threshold white pixel denotes strong edge RobertsPrewittSobel UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

47. ELE 488 F06 Example Edges in a Grayscale Image Line 1Line 2 Line 3 Line 4 Line 1 Line 2 Line 3 Line 4 Close up

48. ELE 488 F06 Common Gradient Operators for Edge Detection –Move the operators across the image and take the inner products Magnitude of gradient vector g(m,n) 2 = g x (m,n) 2 + g y (m,n) 2 Direction of gradient vector tan –1 [ g y (m,n) / g x (m,n) ] –Gradient operator is HPF in nature ~ could amplify noise Prewitt and Sobel operators compute horizontal and vertical differences of local sum to reduce the effect of noise 01 0 0 0 1 1 01 0 -20 1 2 01 10 0 00 0 111 -2 00 0 121 Roberts H 1 (m,n) H 2 (m,n) PrewittSobel UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

49. ELE 488 F06 Suppressing Noise via Spatial Averaging Image with iid noise y(m,n) = x(m,n) + N(m,n) Averaged version v(m,n) = (1/N w )  x(m-k, n-l) + (1/N w )  N(m-k, n-l) Noise variance reduced by a factor of N w –N w ~ # of pixels in the averaging window SNR improved by a factor of Nw if x(m,n) is constant in local window Window size is limited to avoid blurring UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

50. ELE 488 F06 Coping with Salt-and-Pepper Noise Matlab Image Toolbox Guide Fig.10-12, 10-13 UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

51. ELE 488 F06 Median Filtering Salt-and-Pepper noise –Isolated white/black pixels spread randomly over the image –Spatial averaging filter may incur blurred output Median filtering –Take median value over a small window as output ~ nonlinear Median{ x(m) + y(m) }  Median{x(m)} + Median{y(m)} –Odd window size is commonly used 3x3, 5x5, 7x7 5-pixel “ + ”-shaped window –Even-sized window ~ take the average of two middle values as output UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

52. ELE 488 F06 Other Variations of Image Sharpening High boost filter (Gonzalez-Woods 2/e pp132 & pp188) I 0  I LP  I HP = I 0 – I LP  I 1 = (b-1) I 0 + I HP –Equiv. to high pass filtering for b=1 –Amplify or suppress original image pixel values when b  2 Combine sharpening with histogram equalization UMCP ENEE408G Slides (created by M.Wu © 2002)