1. EE565 Advanced Image Processing Copyright Xin Li 20081 Different Frameworks for Image Processing Statistical/Stochastic Models: Wiener’s MMSE estimation (e.g., denoising  filtering) Transform-Based Models: Fourier/Wavelets transform ( e.g., denoising  thresholding ) Variational PDE Models: Evolve image according to local derivative/geometric info, (e.g. denoising  diffusion) Concepts are related mathematically: Brownian motion – Fourier Analysis --- Diffusion Equation

2. EE565 Advanced Image Processing Copyright Xin Li 20082 PDE-based Image Processing  Image as a surface  Image Interpolation Implication of artifacts on surface area Minimal surface solution via mean curvature diffusion  Image inpainting Variational formulation Energy minimization solution  Image denoising From linear to nonlinear diffusion Perona-Malik diffusion

3. EE565 Advanced Image Processing Copyright Xin Li 20083 PDE-based Image Interpolation* Bilinear Interpolation PDE-based post-processing Low-resolution image Intermediate result High-resolution image

4. EE565 Advanced Image Processing Copyright Xin Li 20084 Image as a Surface 3D visualizationsingle-edge image If image can be viewed as a surface, it is then natural to ask: can we apply geometric tools to process this surface (or its equivalent image signals)?

5. EE565 Advanced Image Processing Copyright Xin Li 20085 Geometric Formulation  Image I: R 2 →R may be viewed as a two-dimensional surface in three-dimensional space, i.e., G: symmetric and positive definite matrix

6. EE565 Advanced Image Processing Copyright Xin Li 20086 Key Motivation  Why these concepts are useful for image processing? Image surface containing artifacts do not have minimal surface minimize S(M) leads to Euler-Lagrange Equation: (A)

7. EE565 Advanced Image Processing Copyright Xin Li 20087 Minimal Surface Unit normal of this surface is Mean curvature is Theorem Surfaces of zero mean curvature have minimal areas (B) Exercise: Derive (B) from (A) by direct calculation

8. EE565 Advanced Image Processing Copyright Xin Li 20088 Mean Curvature Diffusion Diffusion equation Discrete Implementation http://www.cmla.ens-cachan.fr/Cmla/Megawave/index.html NOT straightforward! Reference: MegaWave 2.0 software We will discuss more numerical implementation next

9. EE565 Advanced Image Processing Copyright Xin Li 20089 Experiment Result Before post-processing After post-processing

10. EE565 Advanced Image Processing Copyright Xin Li 200810 Further Diffusion After 3 iterationsAfter 10 iterations

11. EE565 Advanced Image Processing Copyright Xin Li 200811 PDE-based Image Processing  Image as a surface  Image Interpolation Implication of artifacts on surface area Minimal surface solution via mean curvature diffusion  Image inpainting Variational formulation Energy minimization solution  Image denoising From linear to nonlinear diffusion Perona-Malik diffusion

12. EE565 Advanced Image Processing Copyright Xin Li 200812 Image Inpainting Extended inpainting domain Assumption: inpainting domain is local and does not contain texture (complimentary to texture-synthesis based inpainting techniques) Image example

13. EE565 Advanced Image Processing Copyright Xin Li 200813 Total Variation Key idea: it is L 1 instead of L 2 norm (minimizing L 2 will not preserve edges) 050100150200250300 0 20 40 60 80 100 120 140 160 180 200 Clean (TV small) noisy (TV large)

14. EE565 Advanced Image Processing Copyright Xin Li 200814 Variational Problem Formulation Restored image degraded image Rational: The first term describes the smoothness constraint within the extend inpainting domain The second term describes the observation constraint Total variation (TV)

15. EE565 Advanced Image Processing Copyright Xin Li 200815 How to obtain the corresponding PDE? Euler-Lagrangian Equation Where TV-inpainting:

16. EE565 Advanced Image Processing Copyright Xin Li 200816 Numerical Implementation of PDEs m n n+1 n-1 m+1 m-1 or

17. EE565 Advanced Image Processing Copyright Xin Li 200817 Inpainting Example (Courtesy: Jackie Shen, UMN MATH)

18. EE565 Advanced Image Processing Copyright Xin Li 200818 PDE-based Image Processing  Image as a surface  Image Interpolation Implication of artifacts on surface area Minimal surface solution via mean curvature diffusion  Image inpainting Variational formulation Energy minimization solution  Image denoising From linear to nonlinear diffusion Perona-Malik diffusion

19. EE565 Advanced Image Processing Copyright Xin Li 200819 Geometry-driven PDEs x y I(x,y) image I image I viewed as a 3D surface (x,y,I(x,y))

20. EE565 Advanced Image Processing Copyright Xin Li 200820 Simplest Case: Laplace Equation Linear Heat Flow Equation: scale A Gaussian filter with zero mean and variance of t Isotropic diffusion:

21. EE565 Advanced Image Processing Copyright Xin Li 200821 Example t=0 t=1 t=2

22. EE565 Advanced Image Processing Copyright Xin Li 200822 Example (Cont.) t=4t=8t=16

23. EE565 Advanced Image Processing Copyright Xin Li 200823 From Isotropic to Anisotropic  Gaussian filtering (isotropic diffusion) could remove noise but it would blur images as well  Ideally, we want Filtering (diffusion) within the object boundary No filtering across the edge orientation  How to achieve such “ anisotropic diffusion ” ?

24. EE565 Advanced Image Processing Copyright Xin Li 200824 Perona-Malik ’ s Idea Isotropic diffusion: edge stopping function

25. EE565 Advanced Image Processing Copyright Xin Li 200825 Pursuit of Appropriate g Define 1D case: Encourage diffusion: Discourage diffusion: Edge slope decreases Edge slope increases

26. EE565 Advanced Image Processing Copyright Xin Li 200826 Examples K Choice-I Choice-II

27. EE565 Advanced Image Processing Copyright Xin Li 200827 Discrete Implementation

28. EE565 Advanced Image Processing Copyright Xin Li 200828 Numerical Examples 100 110 100 200 100

29. EE565 Advanced Image Processing Copyright Xin Li 200829 Scale-space with Anisotropic Diffusion original P-M filter (K=16,100 iterations)

30. EE565 Advanced Image Processing Copyright Xin Li 200830 P-M Filter for Image Denoising Noisy image (PSNR=28.13) P-M filtered image (PSNR=29.83)

31. EE565 Advanced Image Processing Copyright Xin Li 200831 Variational Interpretation

32. EE565 Advanced Image Processing Copyright Xin Li 200832 Comparison between Wavelet - based and PDE-based denoising  Wavelet theory Strength: offers a basis to distinguish signals from noise (signal behaves as significant coefficients while noise will not) Weakness: ignore geometry  Diffusion theory Strength: geometry-driven Weakness: localized model (poor to realize global trend in the signal)