1. Computer Vision Group Digital Image Filters 26 th November 2012

2. Computer Vision Group Outline  Definitions  Linear, shift invariant filters: convolution.  Examples of convolutional filters Smoothing Filters (denoising) Differentiating Filters (edge detector)  Examples of Nonlinear Filtering

3. Computer Vision Group Systems  A System H is considered as a black box that processes some input signal (f) and gives the output (Hf)  In our case, f is a digital image or a 1-D digital signal, but in principle could be analogic and n-D signal as well as a scalar

4. Computer Vision Group Linearity

5. Computer Vision Group Signal and System - Property  A system is said to be time (or shift) – invariant i.i.f.  Systems that are Linear and Translation Invariant are represented by a convolutoin. LTI systems are characterized entirely by a single function called the system's impulse response. The impulse response corresponds to the filter associated to the system.

6. Computer Vision Group Convolution - LTI System

7. Computer Vision Group Convolution LIT Systems on Images h7h7 h8h8 h9h9 h4h4 h5h5 h6h6 h1h1 h2h2 h3h3 h9h9 h8h8 h7h7 h6h6 h5h5 h4h4 h3h3 h2h2 h1h1 h1h1 h2h2 h3h3 h4h4 h5h5 h6h6 h7h7 h8h8 h9h9 y1y1 y2y2 y3y3 y4y4 y5y5 y6y6 y7y7 y8y8 y9y9 h9h9 h8h8 h7h7 h6h6 h5h5 h4h4 h3h3 h2h2 h1h1 Point-wise product h9y1 h9y1 h8y2h8y2 h7y3h7y3 h6y4h6y4 h5y5h5y5 h4y6h4y6 h3y7h3y7 h2y8h2y8 h1y9h1y9 sum z 5 = h 9 y 1 + h 8 y 2 + h 7 y 3 + h 6 y 4 + h 5 y 5 + h 5 y 6 + h 3 y 7 + h 2 y 8 + h 1 y 9

8. Computer Vision Group Convolution LIT Systems on Images h7h7 h8h8 h9h9 h4h4 h5h5 h6h6 h1h1 h2h2 h3h3 h9h9 h8h8 h7h7 h6h6 h5h5 h4h4 h3h3 h2h2 h1h1 h1h1 h2h2 h3h3 h4h4 h5h5 h6h6 h7h7 h8h8 h9h9 y1y1 y2y2 y3y3 y4y4 y5y5 y6y6 y7y7 y8y8 y9y9 h9h9 h8h8 h7h7 h6h6 h5h5 h4h4 h3h3 h2h2 h1h1 Point-wise product h9y1 h9y1 h8y2h8y2 h7y3h7y3 h6y4h6y4 h5y5h5y5 h4y6h4y6 h3y7h3y7 h2y8h2y8 h1y9h1y9 sum z 5 = h 9 y 1 + h 8 y 2 + h 7 y 3 + h 6 y 4 + h 5 y 5 + h 5 y 6 + h 3 y 7 + h 2 y 8 + h 1 y 9

9. Computer Vision Group Convolution LIT Systems on Images h7h7 h8h8 h9h9 h4h4 h5h5 h6h6 h1h1 h2h2 h3h3 h9h9 h8h8 h7h7 h6h6 h5h5 h4h4 h3h3 h2h2 h1h1 h1h1 h2h2 h3h3 h4h4 h5h5 h6h6 h7h7 h8h8 h9h9 y1y1 y2y2 y3y3 y4y4 y5y5 y6y6 y7y7 y8y8 y9y9 h9h9 h8h8 h7h7 h6h6 h5h5 h4h4 h3h3 h2h2 h1h1 Point-wise product h9y1 h9y1 h8y2h8y2 h7y3h7y3 h6y4h6y4 h5y5h5y5 h4y6h4y6 h3y7h3y7 h2y8h2y8 h1y9h1y9 sum z 5 = h 9 y 1 + h 8 y 2 + h 7 y 3 + h 6 y 4 + h 5 y 5 + h 5 y 6 + h 3 y 7 + h 2 y 8 + h 1 y 9

10. Computer Vision Group Convolution LIT Systems on Images h7h7 h8h8 h9h9 h4h4 h5h5 h6h6 h1h1 h2h2 h3h3 h9h9 h8h8 h7h7 h6h6 h5h5 h4h4 h3h3 h2h2 h1h1 h1h1 h2h2 h3h3 h4h4 h5h5 h6h6 h7h7 h8h8 h9h9 y1y1 y2y2 y3y3 y4y4 y5y5 y6y6 y7y7 y8y8 y9y9 h9h9 h8h8 h7h7 h6h6 h5h5 h4h4 h3h3 h2h2 h1h1 Point-wise product h9y1 h9y1 h8y2h8y2 h7y3h7y3 h6y4h6y4 h5y5h5y5 h4y6h4y6 h3y7h3y7 h2y8h2y8 h1y9h1y9 sum z 5 = h 9 y 1 + h 8 y 2 + h 7 y 3 + h 6 y 4 + h 5 y 5 + h 5 y 6 + h 3 y 7 + h 2 y 8 + h 1 y 9

11. Computer Vision Group Linear Filters for Digital Images  Linear Filters are particular systems where the output at each pixel (sample) is given by a linear combination of neighboring pixels (samples) 103010 201120 1191 * * * * 5.7 * * * * Original Signal Filtered Output

12. Computer Vision Group Filters or Kernels  The coefficients used in the linear combination are given by the kernel 130 2102 411 Image 10 10.1 10 Kernel = - 3 Filter Output

13. Computer Vision Group A well-known Test Image - Lena

14. Computer Vision Group Trivial example 000 010 000

15. Computer Vision Group 000 100 000 Transalation

16. Computer Vision Group Linear Filtering 111 111 111 -local averaging

17. Computer Vision Group The original Lena

18. Computer Vision Group Filtered Lena

19. Computer Vision Group Linear Filtering 11111 11111 11111 11111 11111 - local averages

20. Computer Vision Group

22. What about normalization? 11111 11111 11111 11111 11111

23. Computer Vision Group … convolution is linear

24. Computer Vision Group …what about 11111 11111 11111 11111 11111

25. Computer Vision Group

26. Convolution Properties

27. Computer Vision Group 2D Gaussian Filter Continuous Function Discrete Function

28. Computer Vision Group Linear Filtering(Gaussian Filter) -weighted local averages-

29. Computer Vision Group Linear Filtering(Gaussian Filter) -weighted local averages-

30. Computer Vision Group Gaussian Vs Average Gaussian Smoothing Support 7x7 Smoothing by Averaging On 7x7 window

31. Computer Vision Group 31 Denoising: The Issue  A Noisy Detail in Camera Raw Data.

32. Computer Vision Group 32 Denoising: The Issue  Denoised

33. Computer Vision Group 33 Denoising: The Issue  A Noisy Detail in Camera Raw Data.

34. Computer Vision Group 34 Denoising: The Issue  Denoised

35. Computer Vision Group  Observation model is  The goal is to obtain, a reliable estimate of, given and the distribution of.  For the sake of simplicity it is often assumed and independent.  The noise standard deviation is assumed as known. Image Formation Model Sensed image Pixel index Original (unknown) image noise July 23, 2010 35

36. Computer Vision Group  Observation model is  Thus we can pursue a “regression-approach”, Image Formation Model July 23, 2010 36

37. Computer Vision Group  Observation model is  Thus we can pursue a “regression-approach”, but on images it may not be convenient to assume a parametric expression of on Image Formation Model July 23, 2010 37

38. Computer Vision Group  Observation model is  Thus we can pursue a “regression-approach”, but on images it may not be convenient to assume a parametric expression of on Image Formation Model July 23, 2010 38

39. Computer Vision Group Denoising Approaches  Parametric Approaches Transform Domain Filtering, they assume the noisy-free signal is somehow sparse in a suitable domain (e.g Fourier, DCT, Wavelet) or w.r.t. some dictionary based decomposition) 27th October 2009 39

40. Computer Vision Group Denoising Approaches  Parametric Approaches Transform Domain Filtering, they assume the noisy-free signal is somehow sparse in a suitable domain (e.g Fourier, DCT, Wavelet) or w.r.t. some dictionary based decomposition)  Non Parametric Approaches Local Smoothing / Local Approximation Non Local Methods 27th October 2009 40

41. Computer Vision Group Denoising Approaches  Parametric Approaches Transform Domain Filtering, they assume the noisy-free signal is somehow sparse in a suitable domain (e.g Fourier, DCT, Wavelet) or w.r.t. some dictionary based decomposition)  Non Parametric Approaches Local Smoothing / Local Approximation Non Local Methods 27th October 2009 41

42. Computer Vision Group Denoising Approaches  Parametric Approaches Transform Domain Filtering, they assume the noisy-free signal is somehow sparse in a suitable domain (e.g Fourier, DCT, Wavelet) or w.r.t. some dictionary based decomposition)  Non Parametric Approaches Local Smoothing / Local Approximation Non Local Methods Estimating from can be statistically treated as regression of on 27th October 2009 42

43. Computer Vision Group Fitting and Convolution  One can prove that the least square fit of polynomial of 0- th order (i.e constant) is given by where and thus 03/12/2009

44. Computer Vision Group Denoising Approaches  Non Parametric Approaches: there are no global model for and basically Local Methods: define, in each image pixel, the “best neighborhood” where a simple parametric model can be enforced to perform regression is described by a (polynomial) model July 23, 2010 44

45. Computer Vision Group Ideal neighborhood – an illustrative example  Ideal in the sense that it defines the support of a pointwise Least Square Estimator of the reference point.  Typically, even in simple images, every point has its own different ideal neighborhood.  For practical reasons, the ideal neighborhood is assumed starshaped Further details at LASIP c/o Tampere University of Technology http://www.cs.tut.fi/~lasip/ http://www.cs.tut.fi/~lasip/ July 23, 2010 45

46. Computer Vision Group Neighborhood discretization  A suitable discretization of this neighborhood is obtained by using a set of directional LPA kernels where determines the orientation of the kernel support, and where controls by the scale of kernel support. Ideal Neighborhood Directional kernels Discrete Adaptive Neighborhood 27th October 2009 46

47. Computer Vision Group Neighborhood discretization  A suitable discretization of this neighborhood is obtained by using a set of directional LPA kernels where determines the orientation of the kernel support, and where controls by the scale of kernel support.  The initial shape optimization problem can be solved by using standard easy-to-implement varying-scale kernel techniques, such as the ICI rule. Ideal Neighborhood Directional kernels Discrete Adaptive Neighborhood 27th October 2009 47

48. Computer Vision Group Ideal neighborhood – an illustrative example  Ideal in the sense that it defines the support of pointwise Least Square Estimator of the reference point. July 23, 2010 48

49. Computer Vision Group Examples of Adaptively Selected Neighorhoods  Define,, the “ideal” neigborhood  Compute the denoised estimate at, “using” July 23, 2010 49

50. Computer Vision Group Examples of adaptively selected neighorhoods  Adaptively selected neighborhoods selected using the LPA-ICI rule July 23, 2010 50

51. Computer Vision Group Denoising Additive Gaussian White Noise After Gaussian Smoothing After Averaging

52. Computer Vision Group Salt & Pepper Noise Additive Noise, 7% outliers After Gaussian Smoothing After Averaging Denoising

53. Computer Vision Group Correlation A Target can be used as a filter. Correlation between target and image assumes maximum at image pixels that match the target

54. Computer Vision Group Correlation for BINARY target matching Easier with binary images Target used as a filter ?

55. Computer Vision Group Correlation function Maximum value line profile Correlation for target matching

56. Computer Vision Group Examples of Non Linear Filters Median Filter (Weighted Median) Ordered Statistics based Filters Threshold There are many others, such as data adaptive filtering procedures (e.g ICI)

57. Computer Vision Group Blockwise Median  Substititute to each pixel the median of its neighbors 130 2102 411 2

58. Computer Vision Group Denoising with median 3x3 Salt and Pepper noise

59. Computer Vision Group Denoising with median 3x3 Additive Gaussian White Noise (AGWN)

60. Computer Vision Group What is an Edge  Lets define an edge to be a discontinuity in image intensity function.  Several Models Step Edge Ramp Edge Roof Edge Spike Edge  They can be thus detected as discontinuities of image Derivatives 03/12/2009

61. Computer Vision Group Differentiation and convolution  Recall  Now this is linear and shift invariant. Therefore, in discrete domain, it will be represented as a convolution 03/12/2009

62. Computer Vision Group Differentiation and convolution  Recall  Now this is linear and shift invariant. Therefore, in discrete domain, it will be represented as a convolution  We could approximate this as (which is obviously a convolution with Kernel ; it’s not a very good way to do things, as we shall see) 03/12/2009

63. Computer Vision Group Finite Difference in 2D Discrete Approximation Horizontal Convolution Kernels Vertical 03/12/2009

64. Computer Vision Group A 1D Example  Take a line on a grayscale image 03/12/2009

65. Computer Vision Group A 1D Example (II)  Filter the discrete image values, convolution against [1 - 1] 03/12/2009

66. Computer Vision Group Differentiating Filters 1D Discrete derivatives 10 2D Discrete derivatives (separable) 10 t 03/12/2009

67. Computer Vision Group Classical Operators : Prewitt Smooth Differentiate Horizontal Vertical 03/12/2009

68. Computer Vision Group Classical Operators: Sobel Smooth Differentiate Horizontal Vertical 03/12/2009

69. Computer Vision Group Another famous test image - cameraman 03/12/2009

70. Computer Vision Group Horizontal Derivatives using Sobel 03/12/2009

71. Computer Vision Group Vertical Derivatives using Sobel 03/12/2009

72. Computer Vision Group Gradient Magnitude and edge detectors 03/12/2009

73. Computer Vision Group A brief overview on Morphological Operators in Image Processing Giacomo Boracchi 24/11/2010 boracchi@elet.polimi.it home.dei.polimi.it/boracchi/t eaching/IAS.htm

74. Computer Vision Group An overview on morphological operations  Erosion, Dilation  Open, Closure  We assume the image being processed is binary, as these operators are typically meant for refining “mask” images. 03/12/2009

75. Computer Vision Group AND operator in Binary images 03/12/2009

76. Computer Vision Group OR in Binary Images 03/12/2009

77. Computer Vision Group Erosion  General definition: Nonlinear Filtering procedure that replace to each pixel value the minimum on a given neighbor  As a consequence on binary images E(x)=1 iff the image in the neighbor is constantly 1  This operation reduces thus the boundaries of binary images  It can be interpreted as an AND operation of the image and the neighbor overlapped at each pixel 03/12/2009

78. Computer Vision Group Erosion Example 03/12/2009

79. Computer Vision Group Erosion Example 03/12/2009

80. Computer Vision Group Dilation  General definition: Nonlinear Filtering procedure that replace to each pixel value the maximum on a given neighbor  As a consequence on binary images E(x)=1 iff at least a pixel in the neighbor is 1  This operation grows fat the boundaries of binary images  It can be interpreted as an OR operation of the image and the neighbor overlapped at each pixel 03/12/2009

81. Computer Vision Group Dilation Example 03/12/2009

82. Computer Vision Group Dilation Example 03/12/2009

83. Computer Vision Group In matlab  They are performed using the bwmorph.m script passing the name of the operation as a parameter Examples… 03/12/2009

84. Computer Vision Group Open and Closure  Open Erosion followed by a Dilation  Closure Dilation followed by an Erosion 03/12/2009

85. Computer Vision Group Open  Open Erosion followed by a Dilation Smoothes the contours of an object Typically eliminate thin protrusions 03/12/2009

86. Computer Vision Group Open  Figure to Open,  Structuring element a Disk 03/12/2009

87. Computer Vision Group Open  First Erode, 03/12/2009

88. Computer Vision Group Open  Then Dilate,  Open results, the bridge has been eliminate from the first erosion and cannot be replaced by the dilation.  Edges are smoothed  Corners are rounded 03/12/2009

89. Computer Vision Group Closure  Closure Dilation followed by an Erosion Smoothes the contours of an object, typically creates Generally fuses narrow breaks 03/12/2009

90. Computer Vision Group Open  Figure to Open,  Structuring element a Disk 03/12/2009

91. Computer Vision Group Close  First Dilate, 03/12/2009

92. Computer Vision Group Close  Then Erode,  Close results, the bridge has been preserved  Edges are smoothed  Corners are rounded 03/12/2009

93. Computer Vision Group There are several other Non Linear Filters  Ordered Statistic based Median Filter Weight Ordered Statistic Filter Trimmed Mean Hybrid Median  Thresholding 03/12/2009