1. Digital Image Processing
Image Restoration
Matrix Formulation
5/3/2019
Duong Anh Duc - Digital Image Processing

2. Matrix Formulation of Image Restoration Problem
1-D Case:
We will consider the 1-D version first, for simplicity:
g(m) = f(m)*h(m) + (m)
We will assume that the arrays f and h have been zero-padded to be of size M, where M  length(f) + length(h) - 1.
Henceforth, we will not explicitly mention the zero-padding.
The degradation equation:
can be written in matrix-vector form as follows:
g = Hf + n, where
5/3/2019
Duong Anh Duc - Digital Image Processing

3. Matrix Formulation of Image Restoration Problem
5/3/2019
Duong Anh Duc - Digital Image Processing

4. Matrix Formulation of Image Restoration Problem
However, since the arrays f and h are zero-padded, we can equivalently set:
Notice that the (second) matrix H is circulant; i.e., each row of H is a circular shift of the previous row.
5/3/2019
Duong Anh Duc - Digital Image Processing

5. Matrix Formulation of Image Restoration Problem
Example:
A = length of array f = 3
B = length of array h = 2
M ³ A + B – 1 = 4, say M = 4.
5/3/2019
Duong Anh Duc - Digital Image Processing

6. Matrix Formulation of Image Restoration Problem
5/3/2019
Duong Anh Duc - Digital Image Processing

7. Matrix Formulation of Image Restoration Problem
Notice that H1f = H2f. Indeed
5/3/2019
Duong Anh Duc - Digital Image Processing

8. Matrix Formulation of Image Restoration Problem
Henceforth, we will use H = H2 , so that we can apply properties of circulant matrices to H.
5/3/2019
Duong Anh Duc - Digital Image Processing

9. Matrix Formulation of Image Restoration Problem
2-D Case:
Suppose g, f, h, are MN arrays (after zero-padding). The degradation equation can be written in matrix-vector format as follows:
g = Hf + n, where
5/3/2019
Duong Anh Duc - Digital Image Processing

10. Matrix Formulation of Image Restoration Problem
5/3/2019
Duong Anh Duc - Digital Image Processing

11. Matrix Formulation of Image Restoration Problem
Note that H is a MNMN block-circulant matrix with MM blocks.
5/3/2019
Duong Anh Duc - Digital Image Processing

12. Matrix Formulation of Image Restoration Problem
Each block Hj is itself an NN circulant matrix. Indeed, the matrix Hj is a circulant matrix formed from the j-th row of array h(m,n):
5/3/2019
Duong Anh Duc - Digital Image Processing

13. Matrix Formulation of Image Restoration Problem
Given the degradation equation:
g = Hf + n,
our objective is to recover f from observation g.
We will assume that the array h(m,n) (usually referred to as the blurring function) and statistics of the noise h(m,n) are known. The problem becomes very complicated if array h(m,n) is unknown and this case is usually referred to as blind restoration or blind deconvolution.
5/3/2019
Duong Anh Duc - Digital Image Processing

14. Matrix Formulation of Image Restoration Problem
Notice that, even when there is no noise; i.e. h(m,n) = 0, or the values of h(m,n) were exactly known, and matrix H is invertible, computing
^f = H-1(g-n)
directly would not be practical.
Example: Suppose M = N = 256. Therefore MN = and H would be a by matrix to be inverted!
5/3/2019
Duong Anh Duc - Digital Image Processing

15. Matrix Formulation of Image Restoration Problem
Naturally, direct inversion of H would not be feasible.
But H has several useful properties; in particular:
H is block circulant.
H is usually sparse (has very few non-zero entries).
We will exploit these properties to obtain ˆf more efficiently.
In particular, we will derive the theoretical solutions to the restoration problem using matrix algebra. However, when it comes to implementing the solution, we can resort to the Fourier domain, thanks to the properties of circulant matrices.
5/3/2019
Duong Anh Duc - Digital Image Processing

16. Constrained least squares filtering (restoration)
Recall that the knowledge of blur function h(m, n) is essential to obtain a meaningful solution to the restoration problem.
Often, knowledge of h(m, n) is not perfect and subject to errors.
One way to alleviate sensitivity of the result to errors in h(m, n) is to base optimality of restoration on a measure of smoothness, such as the second derivative of the image.
5/3/2019
Duong Anh Duc - Digital Image Processing

17. Constrained least squares filtering (restoration)
We will approximate the second derivative (Laplacian) by a matrix Q. Indeed, we will first formulate the constrained restoration problem and obtain its solution in terms of a general matrix Q.
5/3/2019
Duong Anh Duc - Digital Image Processing

18. Constrained least squares filtering (restoration)
Later different choices of matrix Q will be considered, each giving rise to a different restoration filter.
Suppose Q is any matrix (of appropriate dimension). In constrained image restoration, we choose ^f to minimize ||Qˆf||2, subject to the constraint,
||g-Hˆf||2= ||n||2.
(Recall the degradation equation
g=Hˆf +n  g-Hˆf = n.)
5/3/2019
Duong Anh Duc - Digital Image Processing

19. Constrained least squares filtering (restoration)
Introduction of matrix Q allows considerable flexibility in the design of appropriate restoration filters (we will discuss specific choices of Q later). So our problem is formulated as follows:
min ||Qˆf||2
subject to ||g-Hˆf||2= ||n||2 or ||g-Hˆf||2- ||n||2=0
5/3/2019
Duong Anh Duc - Digital Image Processing

20. A brief review of matrix differentiation
Suppose
and f(x1,x2) is a function of two variables. Then
5/3/2019
Duong Anh Duc - Digital Image Processing

21. A brief review of matrix differentiation
f(x1,x2) = ||Ax-b||2 = (Ax-b)T (Ax-b)
for some matrix A and some vector b , then
where superscript T denotes matrix transpose.
5/3/2019
Duong Anh Duc - Digital Image Processing

22. A brief review of matrix differentiation
Recall from calculus that such a constrained minimization problem can be solved by means of Lagrange multipliers. We need to minimize the augmented objective function J(ˆf):
J(ˆf) = ||Qˆf||2+(||g-Hˆf||2- ||n||2),
where a is a Lagrange multiplier.
We set the derivative of J(ˆf) with respect to ˆf to zero.
J(ˆf) = 2QT Qˆf-2HT(g-Hˆf) = 0
 (QT Q + HTH)ˆf = HTg
5/3/2019
Duong Anh Duc - Digital Image Processing

23. A brief review of matrix differentiation
Therefore,
ˆf = (QT Q + HTH)-1HTg
= (QT Q/ + HTH)-1HTg
= (QT Q + HTH)-1HTg
where  = 1/ is chosen to satisfy the constraint ||g-Hˆf||2= ||n||2.
We will now use the above formulation to derive a number of restoration filters.
5/3/2019
Duong Anh Duc - Digital Image Processing

24. Pseudo-inverse Filtering
The pseudo-inverse filter tries to avoid the pitfalls of applying an inverse filter in the presence of noise.
Consider the constrained restoration solution,
^f = (QT Q + HTH)-1HTg
with Q = I. This gives,
^f = (I + HTH)-1HTg
5/3/2019
Duong Anh Duc - Digital Image Processing

25. Pseudo-inverse Filtering
It can be implemented in the Fourier domain by the following equation:
^F(u,v) = R(u,v)G(u,v), where
The parameter g is a constant to be chosen.
Note that g = 0 gives us back the inverse filter. For g > 0, the denominator of R(u, v) is strictly positive and the pseudo-inverse filter is well defined.
5/3/2019
Duong Anh Duc - Digital Image Processing

26. Pseudo-inverse Filtering example
Added Zero-mean Gaussian noise
with variance s2 = 0.003
f(m,n)
g(m,n) MSE = 0.01
5/3/2019
Duong Anh Duc - Digital Image Processing

27. Pseudo-inverse Filtering
^f(x,y)
g = MSE = 0.075
g = MSE =
5/3/2019
Duong Anh Duc - Digital Image Processing

28. Pseudo-inverse Filtering
^f(x,y)
g = MSE =
g = 0 (Inv. Filter) MSE = 2.7
5/3/2019
Duong Anh Duc - Digital Image Processing

29. Minimum Mean Square Error (Wiener) Filter
This is a restoration technique based on the statistics (mean and correlation) of the image and noise.
We consider each element of f and n as random variables. Define the correlation matrices
5/3/2019
Duong Anh Duc - Digital Image Processing

30. Minimum Mean Square Error (Wiener) Filter
The matrices Rf and Rn are real and symmetric, with all eigenvalues being non-negative.
The 2D-DFT of the correlations Rf and Rn are called the power spectra and are denoted by Sf(u,v) and Sn(u,v) respectively.
5/3/2019
Duong Anh Duc - Digital Image Processing

31. Minimum Mean Square Error (Wiener) Filter
Recall the constrained restoration solution given by
^f = (QT Q + HTH)-1HTg
Choose matrix Q such that
QT Q = Rf-1Rn
In a sense, we are trying to minimize the noise-to-signal ratio.
The constrained restoration is then given by
^f = (HTH +  Rf-1Rn)-1HTg
5/3/2019
Duong Anh Duc - Digital Image Processing

32. Minimum Mean Square Error (Wiener) Filter
This can be implemented using DFT as
5/3/2019
Duong Anh Duc - Digital Image Processing

33. Minimum Mean Square Error (Wiener) Filter
Here Sf(u,v) = E(|F(u,v)|2) is the power spectral density of the image f(m, n) and Sh(u,v) = E(|N(u,v)|2) is the power spectral density of the noise h(m, n).
The restoration filter R(u,v) is called the parametric Wiener filter, with parameter g
Special cases:
g = 1: Wiener Filter
g = 0: Inverse Filter
g  0,1: Parametric Wiener Filter
5/3/2019
Duong Anh Duc - Digital Image Processing

34. Minimum Mean Square Error (Wiener) Filter
According to the constrained restoration filter derived earlier, parameter g should be chosen to satisfy ||g-Hˆf||2= ||n||2.
However, choice of g = 1 yields an optimal filter in the sense of minimizing the error function e2= E{[f(m,n)-^f(m,n)]2}. In other words, setting g = 1 yields a statistically optimal restoration.
Implementation of the parametric Wiener filter requires knowledge of the image and noise power spectra Sf(u,v) and Sn(u,v). In particular, we need the so called signal-to-noise ratio (SNR) r(u,v) = Sf(u,v)/Sn(u,v)
5/3/2019
Duong Anh Duc - Digital Image Processing

35. Minimum Mean Square Error (Wiener) Filter
This is not always available and a simple approximation is to replace r(u,v) by a constant r. In this case, the Wiener filter is given by
Note that as    (no noise), the Wiener filter tends to the inverse filter.
5/3/2019
Duong Anh Duc - Digital Image Processing

36. Duong Anh Duc - Digital Image Processing
Wiener Filter example
f(m,n)
5/3/2019
Duong Anh Duc - Digital Image Processing

37. Duong Anh Duc - Digital Image Processing
Wiener Filter example
 = 25.9dB
5/3/2019
Duong Anh Duc - Digital Image Processing

38. Duong Anh Duc - Digital Image Processing
Wiener Filter example
 = 15.9dB
5/3/2019
Duong Anh Duc - Digital Image Processing

39. Duong Anh Duc - Digital Image Processing
Wiener Filter example
 = 5.9dB
5/3/2019
Duong Anh Duc - Digital Image Processing

40. Parametric Wiener Filter example (effect of parameter g)
g(m,n)
^f(m,n), g = 0.01
^f(m,n), g = 0.1
5/3/2019
Duong Anh Duc - Digital Image Processing

41. Parametric Wiener Filter example (effect of parameter g)
^f(m,n), g = 1
^f(m,n), g = 5
^f(m,n), g = 50
5/3/2019
Duong Anh Duc - Digital Image Processing

42. Parametric Wiener Filter example (effect of parameter g)
Small values of g result in better “blur removal” and poor noise filtering.
Large values of g result in poor “blur removal” and better noise filtering.
5/3/2019
Duong Anh Duc - Digital Image Processing

43. Constrained Least Squares Restoration
Recall the constrained restoration problem:
min ||Q^f ||2
subject to ||g-Hˆf||2 - ||n||2 = 0 or ||g-Hˆf||2= ||n||2
Its solution
^f = (HTH + QT Q )-1HTg
depends on the choice of Q.
5/3/2019
Duong Anh Duc - Digital Image Processing

44. Constrained Least Squares Restoration
One possibility is to formulate a criterion of optimality (choice of Q) that is based on a measure of smoothness (minimize “roughness” or oscillatory behavior of the solution).
This is normally done by choosing Q to represent a second derivative of the image.
5/3/2019
Duong Anh Duc - Digital Image Processing

45. Constrained Least Squares Restoration
Consider the 1-D case: A discrete approximation of the second derivative at a point x = mDx can be obtained as follows:
5/3/2019
Duong Anh Duc - Digital Image Processing

46. Constrained Least Squares Restoration
In a discrete formulation with Dx = 1, this can be written as
Therefore, we seek an estimate ˆf of f which is smooth in the sense that it minimizes the above “roughness measure.”
5/3/2019
Duong Anh Duc - Digital Image Processing

47. Constrained Least Squares Restoration
This can be formulated in our standard matrix notation as follows:
min ||C^f ||2 = {^fTCTC^f}
subject to ||g-Hˆf||2 - ||n||2 = 0 or ||g-Hˆf||2= ||n||2
where (recall zero-padding)
5/3/2019
Duong Anh Duc - Digital Image Processing

48. Constrained Least Squares Restoration
or equivalently
is a “smoothing matrix” and ^f is a vector representing the restored image.
5/3/2019
Duong Anh Duc - Digital Image Processing

49. Constrained Least Squares Restoration
In the 2D case (with Dx = Dy = 1 ), we have
5/3/2019
Duong Anh Duc - Digital Image Processing

50. Constrained Least Squares Restoration
The roughness measure can then be written as
5/3/2019
Duong Anh Duc - Digital Image Processing

51. Constrained Least Squares Restoration
This can be formulated in our standard matrix notation as follows:
min ||C^f ||2 = {^fTCTC^f}
subject to ||g-Hˆf||2 - ||n||2 = 0 or ||g-Hˆf||2= ||n||2
where (recall zero-padding):
5/3/2019
Duong Anh Duc - Digital Image Processing

52. Constrained Least Squares Restoration
is a “smoothing matrix” and ˆf is a vector representing the restored image.
5/3/2019
Duong Anh Duc - Digital Image Processing

53. Constrained Least Squares Restoration
Notice that C is a block circulant matrix.
As before, the solution to the above optimization problem is given by
^f = (HTH + CT C )-1HTg
5/3/2019
Duong Anh Duc - Digital Image Processing

54. Constrained Least Squares Restoration
Using properties of the block circulant matrix C, we get the following implementation of this filter:
5/3/2019
Duong Anh Duc - Digital Image Processing

55. Constrained Least Squares Restoration
Here P(u,v) is the 2D-DFT of matrix p(m,n), after appropriate zeropadding.
Compare this with the parametric Wiener filter:
no power spectrum information is required in the constrained leastsquares restoration!
5/3/2019
Duong Anh Duc - Digital Image Processing

56. Constrained Least Squares Restoration
However, for the new filter to be optimal, the parameter g must be chosen to satisfy the constraint ||g-Hˆf||= ||n||.
Define the residual vector
r = g-Hˆf = g - H (HTH + CT C )-1HTg
Therefore, we need to choose g such that ||r|| = ||n||
It can be shown that the function
()= rTr = ||r||2
is a monotonically increasing function of g.
We want to adjust g so that
() = ||r||2 = ||n||2 ± a
for some accuracy factor a.
5/3/2019
Duong Anh Duc - Digital Image Processing

57. Constrained Least Squares Restoration
Since () is monotonically increasing, this can be accomplished by the following procedure:
1. Specify an initial value of g = g0
2. For k = 1, 2 , …,
compute ^fk = (HTH + kCT C )-1HTg. This can be done using a frequency domain implementation: ^Fk(u,v) = Rk(u,v)G(u,v), where
5/3/2019
Duong Anh Duc - Digital Image Processing

58. Constrained Least Squares Restoration
Compute
3. If (k)<||n||2–a, k+1=k+b, set k=k+1, return to step 2
If (k)>||n||2+a, k+1=k-b, set k=k+1, return to step 2
Otherwise, STOP (current ^fk or ^Fk(u,v) is the restored image and gk is the optimal choice of parameter g).
5/3/2019
Duong Anh Duc - Digital Image Processing

59. Constrained Least Squares Restoration
Implementation of this procedure requires knowledge of ||n||2, which denotes the strength of noise. If
are the noise mean and variance, respectively, then
5/3/2019
Duong Anh Duc - Digital Image Processing

60. Constrained Least Squares Restoration
Therefore
which can be computed from knowledge of the mean and variance of noise.
5/3/2019
Duong Anh Duc - Digital Image Processing

61. Constrained LS Example
f(m,n)
g(m,n)
5/3/2019
Duong Anh Duc - Digital Image Processing

62. Constrained LS Example
^f(m,n),  = 0.01
^f(m,n),  = 0.442
^f(m,n),  = 1
5/3/2019
Duong Anh Duc - Digital Image Processing

63. Duong Anh Duc - Digital Image Processing
Geometric Distortion
5/3/2019
Duong Anh Duc - Digital Image Processing

64. Gray-level Interpolation
5/3/2019
Duong Anh Duc - Digital Image Processing

65. Duong Anh Duc - Digital Image Processing
Example
5/3/2019
Duong Anh Duc - Digital Image Processing

66. Duong Anh Duc - Digital Image Processing
Example
5/3/2019
Duong Anh Duc - Digital Image Processing