1. ENEE631 Digital Image Processing (Spring'04) Image Restoration Spring ’04 Instructor: Min Wu ECE Department, Univ. of Maryland, College Park   www.ajconline.umd.edu (select ENEE631 S’04)   minwu@eng.umd.edu Based on ENEE631 Spring’04 Section 7

2. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [2] Overview Last Time: basics on 2-D random signal (random field) Relation and extension from 1-D random process (1) Sequences of random variables & joint distributions (2) First two moment functions and their properties (3) Wide-sense stationarity (4) Unique to 2-D case: separable and isotropic covariance function (5) Power spectral density and properties Today: image restoration Assignment#2: Due Friday 3/5/2004 5pm Suggested readings: implementation issues on 2-D FT and filtering (Gonzalez Section 4.6) UMCP ENEE631 Slides (created by M.Wu © 2004)

3. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [3] Image Restoration UMCP ENEE631 Slides (created by M.Wu © 2004)

4. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [4] From Matlab ImageToolbox Documentation pp12-4 UMCP ENEE631 Slides (created by M.Wu © 2001)

5. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [5] Imperfectness in Image Capturing Blurring ~ linear spatial-invariant filter model w/ additive noise Impulse response h(n 1, n 2 ) & H(  1,  2 ) –Point Spread Function (PSF) ~ positive I/O –[No blur] h(n 1, n 2 ) =  (n 1, n 2 ) –[Linear translational motion blur] u local average along motion direction –[Uniform out-of-focus blur] u local average in a circular neighborhood –Atomspheric turbulence blur, etc. H u(n 1, n 2 )v(n 1, n 2 ) N(n 1, n 2 ) UMCP ENEE631 Slides (created by M.Wu © 2001)

6. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [6] Fourier Transform of PSF for Common Distortions From Bovik’s Handbook Sec.3.5 Fig.2&3 UMCP ENEE631 Slides (created by M.Wu © 2001)

7. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [7] Undo Linear Spatial-Invariant Distortion Assume noiseless and PSF of distortion h(n 1, n 2 ) is known Restoration by Deconvolution / Inverse-Filtering –Often used for deblurring –Want to find g(n 1, n 2 ) satisfies h(n 1, n 2 )  g(n 1, n 2 ) =  (n 1, n 2 ) u  h(k 1, k 2 ) g(n 1 -k 1, n 2 -k 2 ) =  (n 1, n 2 ) for all n 1, n 2 –Easy to solve in spectrum domain u Convolution  Multiplication u H(  1,  2 ) G(  1,  2 ) = 1 u Interpretation: choose G to compensate distortions from H H u(n 1, n 2 )v(n 1, n 2 )  (n 1, n 2 )=0 G u’(n 1, n 2 )w(n 1, n 2 ) UMCP ENEE631 Slides (created by M.Wu © 2001)

8. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [8] Problems With Inverse Filtering Under Noise Zeros in H(  1,  2 ) –Interpretation: distortion by H removes all info. in those freq. –Inverse filter tries to “compensate” by assigning infinite gains –Amplifies noise: u W(  1,  2 ) = H (  1,  2 ) U (  1,  2 ) u U’ (  1,  2 ) = (W+N) / H  N/H if W=0 Solutions? UMCP ENEE631 Slides (created by M.Wu © 2001)

9. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [9] Examples of Inverse & Pseudo-inverse Filtering From Jain Fig.8.10 UMCP ENEE631 Slides (created by M.Wu © 2001)

10. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [10] Problems With Inverse Filtering Under Noise Zeros in H(  1,  2 ) –Interpretation: distortion by H removes all info. in those freq. –Inverse filter tries to “compensate” by assigning infinite gains –Amplifies noise: u W(  1,  2 ) = H (  1,  2 ) U (  1,  2 ) u U’ (  1,  2 ) = (W+N) / H  N/H if W=0 Solutions ~ Pseudo-inverse Filtering –Assign zero gain for G at spectrum nulls of H –Interpretation: not bother to make impossible compensations UMCP ENEE631 Slides (created by M.Wu © 2001)

11. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [11] Handling Spectrum Nulls Via High-Freq Cut-off  Limit the restoration to lower frequency components to avoid amplifying noise at spectrum nulls UMCP ENEE631 Slides (created by M.Wu © 2004) Figure is from slides at Gonzalez/ Woods DIP book website (Chapter 5)

12. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [12] Handling Noise in Deconvolution Inverse filtering is sensitive to noise –Does not explicitly modeling and handling noise Try to balance between deblurring vs. noise suppression –Minimize MSE between the original and restored u e = E{ [ u(n 1, n 2 ) – u’(n 1, n 2 ) ] 2 } where u’(n 1, n 2 ) is a func. of {v(m 1, m 2 ) } –Best estimate is conditional mean E[ u(n 1, n 2 ) | all v(m 1, m 2 ) ] u usually difficult to solve for general restoration (need conditional probability distribution, and estimation is nonlinear in general) Get the best linear estimate instead  Wiener filtering –Consider the (desired) image and noise as random fields –Produce a linear estimate from the observed image to minimize MSE H u(n 1, n 2 )v(n 1, n 2 )  (n 1, n 2 ) G u’(n 1, n 2 )w(n 1, n 2 ) UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

13. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [13] Wiener Filtering Get the best linear estimate minimizing MSE –Assume spatial-invariant filter u’(n 1, n 2 ) = g (n 1, n 2 )  v(n 1, n 2 ) –Assume wide-sense stationarity for original signal and noise –Assume noise is zero-mean and uncorrelated with original signal Solutions –Bring into orthogonal condition E{ [ u(n 1, n 2 ) – u’(n 1, n 2 ) ] v(m 1, m 2 ) }=0 –Represent in correlation functions: R uv (k,l) = g(k,l)  R vv (k,l) –Take DFT to get representation in power spectrum density UMCP ENEE631 Slides (created by M.Wu © 2001)

14. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [14] More on Wiener Filtering Balancing between two jobs for deblurring noisy image –HPF filter for de-blurring (undo H distortion) –LPF for suppressing noise Noiseless case ~ S  = 0 (inverse filter) –Wiener filter becomes pseudo-inverse filter for S   0 No-blur case ~ H = 1 (Wiener Smoothing Filter) –Zero-phase filter to attenuate noise according to SNR at each freq. UMCP ENEE631 Slides (created by M.Wu © 2001)

15. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [15] Comparisons From Jain Fig.8.11 UMCP ENEE631 Slides (created by M.Wu © 2001)

16. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [16] Example: Wiener Filtering vs. Inverse Filtering UMCP ENEE631 Slides (created by M.Wu © 2004) Figure is from slides at Gonzalez/ Woods DIP book website (Chapter 5)

17. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [17] Example (2): Wiener Filtering vs. Inverse Filtering UMCP ENEE631 Slides (created by M.Wu © 2004) Figure is from slides at Gonzalez/ Woods DIP book website (Chapter 5)

18. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [18] To Explore Further   UMCP ENEE631 Slides (created by M.Wu © 2004)

19. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [19] Review: Handling Noise in Deconvolution Inverse filtering is sensitive to noise –Does not explicitly modeling and handling noise Try to balance between deblurring vs. noise suppression –Consider the (desired) image and noise as random fields –Minimize MSE between the original and restored u e = E{ [ u(n 1, n 2 ) – u’(n 1, n 2 ) ] 2 } where u’(n 1, n 2 ) is a func. of {v(m 1, m 2 ) } –Best estimate is conditional mean E[ u(n 1, n 2 ) | all v(m 1, m 2 ) ] Get the best linear estimate  Wiener filtering –Produce a linear estimate from the observed image to minimize MSE H u(n 1, n 2 )v(n 1, n 2 )  (n 1, n 2 ) G u’(n 1, n 2 )w(n 1, n 2 ) UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

20. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [20] Wiener Filter: From Theory to Practice Recall: assumed p.s.d. of image & noise random fields and freq. response of distortion filter are known Why make the assumptions? Are these reasonable assumptions? What do they imply in our implementation of Wiener filter? UMCP ENEE631 Slides (created by M.Wu © 2004)

21. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [21] Wiener Filter: Issues to Be Addressed Wiener filter’s size –Theoretically has infinite impulse response ~ require large-size DFTs –Impose filter size constraint: find the best FIR that minimizes MSE Need to estimate power spectrum density of orig. signal –Estimate p.s.d. of blurred image v and compensate variance due to noise –Estimate p.s.d. from a set of representative images similar to the images to be restored –Or use statistical model for the orig. image and estimate parameters –Constrained least square filter ~ see Jain’s Sec.8.8 & Gonzalez Sec.5.9 u Optimize smoothness in restored image (least-square of the rough transitions) u Constrain differences between blurred image and blurred version of reconstructed image u Estimate the restoration filter w/o the need of estimating p.s.d. Unknown distortion H ~ Blind Deconvolution UMCP ENEE631 Slides (created by M.Wu © 2001)

22. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [22] EE630 Review: Periodogram Spectral Estimator UMCP ENEE624 Slides (created by M.Wu © 2003)

23. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [23] EE630 Review: Averaged Periodogram As one solution to the variance problem of periodogram –Average K periodograms computed from K sets of data records UMCP ENEE624 Slides (created by M.Wu © 2003)

24. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [24] Basic Ideas of Blind Deconvolution Three ways to estimate H: observation, experimentation, math. modeling Estimate H via spectrum’s zero patterns –Two major classes of blur (motion blur and out-of-focus) –H has nulls related to the type and the parameters of the blur Maximum-Likelihood blur estimation –Each set of image model and blur parameters gives a “typical” blurred output; Probability comes into picture because of the existence of noise –Given the observation of blurred image, try to find the set of parameters that is most likely to produce that blurred output Iteration ~ Expectation-Maximization approach (EM) u Given estimated parameters, restore image via Wiener filtering u Examine restored image and refine parameter estimation u Get local optimums  To explore more: Bovik’s Handbook Sec.3.5 (subsection-4, pp136) “Blind Image Deconvolution” by Kundur et al, IEEE Sig. Proc. Magazine, vol.13, 1996 UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

25. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [25] Filtering Through Transform Domain Operation E.g.1 Realize Wiener in DFT domain E.g.2 Use zonal mask in transform domain –Realize “ideal” LPF/BPF/HPF –Computation complexity for transform could be high for large image From Jain Fig.7.31&7.32 UMCP ENEE631 Slides (created by M.Wu © 2001)

26. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [26] Frequency Domain View of LPF / HPF UMCP ENEE631 Slides (created by M.Wu © 2004) Image example is from Gonzalez-Woods 2/e online slides.

27. ENEE631 Digital Image Processing (Spring'04) Lec8 – Image Restoration [27] Summary of Today’s Lecture Image restoration –Deconvolution / Inverse filtering u Pseudo inverse filtering to cope with spectral nulls –Wiener filtering Next time –Basic compression techniques Readings –Jain’s book 8.1-8.4 Gonzalez’s book 5.5-5.8 UMCP ENEE631 Slides (created by M.Wu © 2004)