2. IMAGE RESTORATION

3. Outline A model of the image degradation / restoration process Noise models Restoration in the presence of noise only – spatial filtering Periodic noise reduction by frequency domain filtering Linear, position-invariant degradations Estimating the degradation function Inverse filtering & Wiener filtering Constrained Least square filtering Geometric mean filter Geometric and spatial transformation

4. What is Image Restoration? Image restoration is to restore a degraded image back to the original image Image enhancement is to manipulate the image so that it is suitable for a specific application.

5. 4 Image Restoration Image restoration attempts to restore images that have been degraded – Identify the degradation process and attempt to reverse it – Similar to image enhancement, but more objective

6. A model of the image degradation/restoration process g(x,y)=f(x,y)*h(x,y)+  (x,y) – Spatial domain G(u,v)=F(u,v)H(u,v)+N(u,v) – Frequency domain

7. A model of the image degradation/restoration process Where, f(x,y) - input image f^(x,y) - estimated original image g(x,y) - degraded image h(x,y) - degradation function  (x,y) - additive noise term

8. Noise models Sources of noise – Image acquisition (digitization) - Imaging sensors can be affected by ambient conditions – Image transmission - Interference can be added to an image during transmission Spatial properties of noise – Statistical behavior of the gray-level values of pixels – Noise parameters, correlation with the image Frequency properties of noise – Fourier spectrum – Ex. white noise (a constant Fourier spectrum)

9. Noise models Noise model is decided based on understanding of the physics of the sources of noise. – Gaussian: poor illumination – Rayleigh: range image – Gamma(Erlang), Exponential: laser imaging – Impulse: faulty switch during imaging – Uniform: least used

10. Noise probability density functions Noise cannot be predicted but can be approximately described in statistical way using the probability density function (PDF)

11. Gaussian noise Mathematical tractability in spatial and frequency domains Used frequently in practice Electronic circuit noise and sensor noise mean variance Note: Intensity

12. Gaussian noise PDF 70% in [(  ), (  )] 95% in [(  ), (  )]

13. – The mean and variance of this density are given by – a and b can be obtained through mean and variance Rayleigh noise

14. Rayleigh noise PDF

15. – The mean and variance of this density are given by – a and b can be obtained through mean and variance Erlang (Gamma) noise

16. Gamma noise (PDF)

17. The mean and variance of this density are given by Exponential noise

18. Exponential Noise PDF Special case of Erlang PDF with b=1

19. Uniform noise Mean: Variance: Less practical, used for random number generator

20. Uniform PDF z

21. Impulse (salt-and-pepper) nosie If either P a or P b is zero, it is called unipolar. Otherwise, it is called bipolar. Quick transients, such as faulty switching during imaging

22. Impulse (salt-and-pepper) nosie PDF

23. Image Degradation with Additive Noise Original image Histogram Degraded images

24. Histogram Degraded images Image Degradation with Additive Noise (cont.)

25. Periodic noise Is an image that arises from electrical and electromechanical interference during image acquisition It can be observed by visual inspection both in the spatial domain and frequency domain can be reduced significantly via frequency domain filtering

26. Periodic Noise Periodic noise looks like dots In the frequency domain

27. 31.10.2015 Estimation of noise parameters Periodic noise – Observe the frequency spectrum Random noise with unknown PDFs – Case 1: imaging system is available Capture images of “ flat ” environment – Case 2: noisy images available Take a strip from constant area Draw the histogram and observe it Measure the mean and variance

28. 31.10.2015 Estimation of Noise Image histogram cannot be used to estimate noise PDF. It is better to use the histogram of one area of an image that has constant intensity to estimate noise PDF.

29. 31.10.2015 Noise Removal Restoration Methods Mean filters – Arithmetic mean filter – Geometric mean filter – Harmonic mean filter – Contra-harmonic mean filter Order statistics filters – Median filter – Max and min filters – Mid-point filter – alpha-trimmed filters Adaptive filters – Adaptive local noise reduction filter – Adaptive median filter

30. 31.10.2015 Mean Filters Arithmetic mean filter or moving average filter mn = size of moving window Degradation model: To remove this part - is the simplest of the mean filters - smooths local variations in an image - noise is reduced as a result of blurring

31. 31.10.2015 Mean Filters Contd... Geometric mean filter - achieves smoothing as compared to arithmetic mean filter - it tends to lose less image detail in the process

32. 31.10.2015 Mean Filter: Example Original image Image corrupted by AWGN Image obtained using a 3x3 geometric mean filter Image obtained using a 3x3 arithmetic mean filter

33. 31.10.2015 Harmonic and Contraharmonic Filters Harmonic mean filter Contraharmonic mean filtermn = size of moving window Works well for salt noise but fails for pepper noise Q = the filter order Positive Q is suitable for eliminating pepper noise. Negative Q is suitable for eliminating salt noise. For Q = 0, the filter reduces to an arithmetic mean filter. For Q = -1, the filter reduces to a harmonic mean filter.

34. 31.10.2015 Contraharmonic Filters: Example Image corrupted by pepper noise with prob. = 0.1 Image corrupted by salt noise with prob. = 0.1 Image obtained using a 3x3 contra- harmonic mean filter With Q = 1.5 Image obtained using a 3x3 contra- harmonic mean filter With Q=-1.5

35. 31.10.2015 Order-Statistic Filters subimage Original image Moving window Statistic parameters Mean, Median, Mode, Min, Max, Etc. Output image

36. 31.10.2015 Order-Statistics Filters Median filter Max filter Min filter Midpoint filter Reduce “dark” noise (pepper noise) Reduce “bright” noise (salt noise)

37. 31.10.2015 Median Filter : How it works A median filter is good for removing impulse, isolated noise Degraded image Salt noise Pepper noise Moving window Sorted array Salt noise Pepper noise Median Filter output Normally, impulse noise has high magnitude and is isolated. When we sort pixels in the moving window, noise pixels are usually at the ends of the array. Therefore, it’s rare that the noise pixel will be a median value.

38. 31.10.2015 Median Filter : Example Image corrupted by salt-and- pepper noise with p a =p b = 0.1 Images obtained using a 3x3 median filter 1 4 2 3

39. 31.10.2015 Max and Min Filters: Example Image corrupted by pepper noise with prob. = 0.1 Image corrupted by salt noise with prob. = 0.1 Image obtained using a 3x3 max filter Image obtained using a 3x3 min filter

40. 31.10.2015 Alpha-trimmed Mean Filter where, g r (s,t) represent the remaining mn-d pixels after removing the d/2 highest and d/2 lowest values of g(s,t). This filter is useful in situations involving multiple types of noise such as a combination of salt-and-pepper and Gaussian noise.

41. 31.10.2015 Alpha-trimmed Mean Filter: Example Image corrupted by additive uniform noise Image obtained using a 5x5 arithmetic mean filter Image additionally corrupted by additive salt-and- pepper noise 12 2 Image obtained using a 5x5 geometric mean filter 2

42. 31.10.2015 Alpha-trimmed Mean Filter: Example (cont.) Image corrupted by additive uniform noise Image obtained using a 5x5 median filter Image additionally corrupted by additive salt-and- pepper noise 12 2 Image obtained using a 5x5 alpha- trimmed mean filter with d = 5 2

43. 31.10.2015 Alpha-trimmed Mean Filter: Example (cont.) Image obtained using a 5x5 arithmetic mean filter Image obtained using a 5x5 geometric mean filter Image obtained using a 5x5 median filter Image obtained using a 5x5 alpha- trimmed mean filter with d = 5

44. 31.10.2015 A model of the image degradation /restoration process g(x,y)=f(x,y)*h(x,y)+  (x,y) G(u,v)=F(u,v)H(u,v)+N(u,v)

45. 31.10.2015 Linear, position-invariant degradation Linear system – H[af 1 (x,y)+bf 2 (x,y)]=aH[f 1 (x,y)]+bH[f 2 (x,y)] Position(space)-invariant system – H[f(x,y)]=g(x,y) is position invariant if H[f(x- , y-  )]=g(x- , y-  ) Properties of the degradation function H

46. impulse linear Impulse response (point spread function) If position-invariant

47. 31.10.2015 Estimation of Degradation Function Degradation model: Methods: 1. Estimation by Image Observation 2. Estimation by Experiment 3. Estimation by Modeling or If we know exactly h(x,y), regardless of noise, we can do deconvolution to get f(x,y) back from g(x,y).

48. 31.10.2015 Estimation by Image Observation f(x,y)f(x,y)f(x,y)*h(x,y)g(x,y)g(x,y) Subimage Reconstructed Subimage DFT Restoration process by estimation Original image (unknown)Degraded image Estimated Transfer function Observation This case is used when we know only g(x,y) and cannot repeat the experiment!

49. 31.10.2015 Estimation by Experiment Used when we have the same equipment set up Input impulse image System H( ) Response image from the system DFT

50. 31.10.2015 Estimation by Modeling Used when we know physical mechanism underlying the image formation process that can be expressed mathematically. Atmospheric Turbulence model Example: Original imageSevere turbulence k = 0.00025 k = 0.001 k = 0.0025 Low turbulenceMild turbulence

51. 31.10.2015 Estimation by Modeling: Motion Blurring Assume that camera velocity is The blurred image is obtained by where T = exposure time.

52. 31.10.2015 Estimation by Modeling: Motion Blurring (cont.) Then we get, the motion blurring transfer function: For constant motion

53. 31.10.2015 Motion Blurring Example For constant motion Original imageMotion blurred image a = b = 0.1, T = 1

54. 31.10.2015 Inverse Filtering after we obtain H(u,v), we can estimate F(u,v) by the inverse filter: From degradation model: Noise is enhanced when H(u,v) is small. To avoid the side effect of enhancing noise, we can apply this formulation to freq. component (u,v) with in a radius D 0 from the center of H(u,v). In practical, the inverse filter is not popularly used.

55. Inverse Filtering The simplest approach to restoration is direct inverse filtering, where we compute an estimate, (u,v), of the transform of the original image simply by dividing the transform of the degraded image, G(u,v), by the degraded function. g(m,n) = f(m,n)*h(m,n)+η(m,n) g = Hf + η If the noise is zero the error function is denoted as T() = ||g- || 2 T() = g2+H 2 -2H = 0

56. Inverse Filtering Contd... To find Mean Square Error 0+2H 2 -2H =0 2H 2 2H H >>>>> H = (Inverse Filtering) G(u,v) = H(u,v)(u,v) (u,v) = With addition of noise the observed degraded image in frequency domain G(u,v) = F(u,v)H(u,v)+N(u,v) ……(1)

57. Inverse Filtering Contd... Divide equation one by H(u,v) = …….(2) We know that (u,v) = Substitute (u,v) in eqn (2) (u,v) = F(u,v)+ If noise is zero the estimated image (u,v) is equal to original image, but noise will not be properly removed in inverse filtering.

58. 31.10.2015 Inverse Filtering: Example Original image Blurred image Due to Turbulence Result of applying the full filter Result of applying the filter with D 0 =70 Result of applying the filter with D 0 =40 Result of applying the filter with D 0 =85

59. Inverse Filtering Limitations: 1.Even if the degradation function is known the undegraded image cannot be recovered exactly because N(u,v) is the random function which is not known. 2.If the degradation function has ‘0’ or small value the ratio easily dominates the estimate F(u,v,) one approach to get ride of 0 (or) small value problem to limits the filter frequency to the value near the origin.

60. WIENER FILTERING Inverse filtering has no explicit provision for handling noise but the wiener filtering it incorporates both degradation function, statistical characteristics of noise taken into the restoration process. e 2 = E[(f-] Objective of the wiener filter is to find the estimate of uncorrupted image f, such that the mean square error is minimize the wiener filter is optimum filter Diagram

61. Wiener Filtering Contd... The error between the input signal and the estimated signal is given by the mean square error. – e(x,y) = f(x,y) –(x,y) – E[f(x,y) –(x,y) 2 ] =0 According to the principle of orthogonality the expected value of f(x,y) –(x,y) totally orthogonal with g(x,y) is zero. E[f(x,y) –(x,y)g(x,y)] = 0 (x,y) = g(x,y)*r(x,y) E[f(x,y)- (g(x,y)*r(x,y))g(x,y)] =0 E[f(x,y)g(x,y)] = E[(r(x,y)*g(x,y))g(x,y)] =E{g(x,y)}

62. Wiener Filtering Contd... γ fg γ fg (x,y) = γ fg (x,y) = γ gg (k,l) γ fg (x,y) = r(x,y)* γ gg (x,y) On taking Fourier transform S fg (u,v) = R(u,v)S gg (u,v) Sfg and Sgg are the Power Spectral Density R(u,v) = W.k.t g(x,y) = f(x,y)*h(x,y) Cross correlation between f(x,y) and g(x,y) is given as

63. Wiener Filtering Contd... γ fg (x,y) = E [f(x+l 1,y+l 2 )g*(x,y)] = E [f(x+l 1,y+l 2 )(f(x,y)*h(x,y))] = E [f(x+l 1,y+l 2 ) ] = γ fg (x,y) = ff (l 1 +k 1,l 2 +k 2 ) = h*(x,y)* ff (-x,-y) On taking Fourier transform S fg (u,v) = H*(u,v)S ff (u,v) Without the presence of noise g=h*f γ gg (l 1,l 2 ) = γ hh (l 1,l 2 )* ff (l 1,l 2 ) γ gg (l 1,l 2 ) = [h(l 1,l 2 )*h*(-l 1,-l 2 )]* ff (l 1,l 2 )

64. Wiener Filtering Contd... S gg (u,v) = H(u,v)H*(u,v). S ff (u,v) = |H(u,v)| 2 S ff (u,v) R(u,v)= = With presence of noise S gg (u,v) = |H(u,v)| 2 S ff (u,v)+ N(u,v) = |N(u,v)| 2 R(u,v)= = = R(u,v)G(u,v) Multiply and divide by H(u,v) in R(u,v) and sub in = G(u,v)

65. Wiener Filtering Contd... = = G(u,v) Wiener filter also know as minimum mean square filter or least mean square filter. Wiener filter does not have the same problem as the inverse filter unless both H(u,v) and Sη(u,v) are zero for the same value of u&v H(u,v) = degradation function H*(u,v) = complex conjugate of H(u,v) |H(u,v)|2 = H*(u,v) H(u,v) Sη(u,v) = |N(u,v)|2 = Power spectrum of the noise Sf(u,v) = |F(u,v)|2 = Power spectrum of an undegraded image.

66. Wiener Filtering Consideration: 1.When a noise is zero η(x,y)=0, Sη(u,v) =0 = It reduces to inverse filtering 2. IF H(u,v)=1 =

67. Wiener Filtering Signal to Noise ratio 3. Signal to noise ratio is greater than 1 >>1 Then = G(u,v) --- Here the wiener filter act as a all pass filters. ADVANTAGES: 1.The wiener filter does not have zero value problem untill both H(u,v) and is equal to zero. 2.The result obtained by wiener filter is more closer to the original image than inverse filter.

68. 31.10.2015 Approximation of Wiener Filter Wiener Filter Formula: Approximated Formula: Difficult to estimate In Practice, K is chosen manually to obtain the best visual result!

69. 31.10.2015 Wiener Filter: Example Original image Blurred image Due to Turbulence Result of the full inverse filter Result of the inverse filter with D 0 =70 Result of the full Wiener filter

70. 31.10.2015 Wiener Filter: Example (cont.) Original image Result of the inverse filter with D 0 =70 Result of the Wiener filter Blurred image Due to Turbulence

71. 31.10.2015 Example: Wiener Filter and Motion Blurring Image degraded by motion blur + AWGN Result of the inverse filter Result of the Wiener filter   2 =650   2 =325   2 =130

72. 31.10.2015 Degradation model: In matrix form, Constrained Least Squares Filter Aims to find the minimum of a criterion function Subject to the constraint Constrained least square filter is given by, where P(u,v) = Fourier transform of p(x,y) = where

73. 31.10.2015 Constrained Least Squares Filter: Example Constrained least square filter  is adaptively adjusted to achieve the best result. Results from the previous slide obtained from the constrained least square filter

74. 31.10.2015 Constrained Least Squares Filter: Example (cont.) Image degraded by motion blur + AWGN Result of the Constrained Least square filter Result of the Wiener filter   2 =650   2 =325   2 =130

75. 31.10.2015 Constrained Least Squares Filter:Adjusting  DefineIt can be shown that We want to adjust gamma so that where a = accuracy factor 1.Specify an initial value of  2.Compute 3.Stop if is satisfied Otherwise return step 2 after increasing  if or decreasing  if Use the new value of  to recompute 1 1

76. 31.10.2015 Constrained Least Squares Filter:Adjusting  (cont.) For computing

77. 31.10.2015 Constrained Least Squares Filter: Example Original image Blurred image Due to Turbulence Results obtained from constrained least square filters Use wrong noise parameters Correct parameters: Initial  = 10 -5 Correction factor = 10 -6 a = 0.25   2 = 10 -5 Wrong noise parameter   2 = 10 -2 Use correct noise parameters

78. 31.10.2015 Geometric Mean filter This filter represents a family of filters combined into a single expression  = 1  the inverse filter  = 0  the Parametric Wiener filter  = 0,  = 1  the standard Wiener filter  = 1,  < 0.5  More like the inverse filter  = 1,  > 0.5  More like the Wiener filter Another name: the spectrum equalization filter

79. 31.10.2015 Geometric Transformation These transformations are often called rubber-sheet transformations: Printing an image on a rubber sheet and then stretch this sheet according to some predefine set of rules. A geometric transformation consists of 2 basic operations: 1. A spatial transformation : Define how pixels are to be rearranged in the spatially transformed image. 2. Gray level interpolation : Assign gray level values to pixels in the spatially transformed image.

80. 31.10.2015 Geometric Transformation : Algorithm Distorted image g 1.Select coordinate (x,y) in f to be restored 2.Compute 3. Go to pixel in a distorted image g Image f to be restored 4. get pixel value at By gray level interpolation 5. store that value in pixel f(x,y) 1 3 5

81. 31.10.2015 Spatial Transformation To map between pixel coordinate (x,y) of f and pixel coordinate (x’,y’) of g For a bilinear transformation mapping between a pair of Quadrilateral regions To obtain r(x,y) and s(x,y), we need to know 4 pairs of coordinates and its corresponding which are called tiepoints.

82. 31.10.2015 Gray Level Interpolation: Nearest Neighbor Since may not be at an integer coordinate, we need to Interpolate the value of Example interpolation methods that can be used: 1. Nearest neighbor selection 2. Bilinear interpolation 3. Bicubic interpolation

83. 31.10.2015 Geometric Distortion and Restoration Example Original image and tiepoints Tiepoints of distorted image Distorted imageRestored image Use nearest neighbor intepolation

84. 31.10.2015 Geometric Distortion and Restoration Example (cont.) Original image and tiepoints Tiepoints of distorted image Distorted imageRestored image Use bilinear intepolation

85. 31.10.2015 Example: Geometric Restoration Original imageGeometrically distorted image Difference between 2 above images Restored image Use the same Spatial Trans. as in the previous example