1. Image Decomposition, Inpainting, and Impulse Noise Removal by Sparse & Redundant Representations Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000, Israel

2. Sparse representations for Image Decomposition 2 General  Sparsity and over-completeness have important roles in analyzing and representing signals.  We have already discussed the image separation task (image=cartoon + texture) and inpainting based on it, adopting a global treatment of these problems.  We have seen in the context of denoising that migration from global to local processing led to far better results.  We Now return to these tasks (and beyond), with a patch-based approach, with the hope to do better.  Before – Lets recall the global treatment

3. Sparse representations for Image Decomposition 3 Global Separation & Inpainting

4. Sparse representations for Image Decomposition 4 Decomposition via Sparsity yy xx = + The dictionaries  x  y should be chosen such that they sparsify the desired content, while giving poor service to the opposite content

5. Sparse representations for Image Decomposition 5 Noise Considerations Forcing exact representation is sensitive to additive noise and model mismatch

6. Sparse representations for Image Decomposition 6 Original ‘Barbara’ image Separated texture using local overlapped DCT (32×32 blocks) Separated Cartoon using Curvelets (5 resolution layers) Results – Good old ‘Barbara’

7. Sparse representations for Image Decomposition 7 Results – Zoom in Zoom in on the result shown in the previous slide (the texture part) Zoom in on the results shown in the previous slide (the cartoon part) The same part taken from Vese’s et. al.

8. Sparse representations for Image Decomposition 8 Application - Inpainting For separation What if some values in s are unknown (with known locations!!!)? The image will be the inpainted outcome. Interesting comparison to [Bertalmio et.al. ’02]

9. Sparse representations for Image Decomposition 9 Results - Inpainting Source Cartoon Part Texture Part Outcome

10. Sparse representations for Image Decomposition 10 Results - Inpainting Source Cartoon Part Texture Part Outcome

11. Sparse representations for Image Decomposition 11 Results - Inpainting Outcome Source

12. Sparse representations for Image Decomposition 12 Patch-Based (Local) Separation

13. Sparse representations for Image Decomposition 13 MCA for Patches Lets impose the MCA model on each and every patch-set in these images

14. Sparse representations for Image Decomposition 14 Start by Denoising  We define, and formulate a patch-based MAP denoising problem:  We are familiar with this expression: it aims to remove the noise V from Y, bypassing altogether the separation task. Compute Sparse Coding of patches Compute D K-SVD Update Compute X Patch- averaging

15. Sparse representations for Image Decomposition 15 Decompose the Dictionary  The denoising process leads to a dictionary D, assumed to be composed of D C and D T.  We need to divide the content of D into these two “sub-dictionaries”, and we can do this by measuring the Total-Variation.  Any atom with “activity” below a threshold is assigned as a cartoon atom, and similarly, to the texture one if above the threshold. D C & D T

16. Sparse representations for Image Decomposition 16 Final Image Separation  Now that D C and D T are defined, we can separate.  Starting with the coefficients in q k are split between the two dictionaries.  We obtain the system:

17. Sparse representations for Image Decomposition 17 Results The obtained results are of similar quality (and slightly better) than the global approach

18. Sparse representations for Image Decomposition 18 Results Zoom in on the texture global result shown before Zoom in on the cartoon global result shown before The same part taken from the local approach

19. Sparse representations for Image Decomposition 19 Patch-Based (Local) Inpainting

20. Sparse representations for Image Decomposition 20 Lets Start with a Simple Experiment  We assume that the image is corrupted by additive noise (  =20), and z% missing pixels (the Mask is given)  Lets break the image into a set of non- overlapping patches, and inpaint each separately by (D: 2D-DCT)  Once done, the patches are put into position to create the inpainted image.

21. Sparse representations for Image Decomposition 21 Lets Start with a Simple Experiment 25% missing 50% missing 75% missing RMSE for 75%missing RMSE for 50% missing RMSE for 25% missing Algorithm 29.7019.6114.55No-overlap

22. Sparse representations for Image Decomposition 22 How About Overlapped Patches? RMSE for 75%missing RMSE for 50% missing RMSE for 25% missing Algorithm 29.7019.6114.55No-overlap 18.1811.559.00Overlap 25% missing 50% missing 75% missing

23. Sparse representations for Image Decomposition 23 Formulating This Problem: K-SVD Compute Sparse Coding of patches Compute D K-SVD Update Compute X Patch- averaging  Initialize with X=M G Y  Sparse coding:  Dictionary Update:  Image Update:

24. Sparse representations for Image Decomposition 24 Results RMSE for 75% missing RMSE for 50% missing RMSE for 25% missing Alg. 29.7019.6114.55No-overlap 18.1811.559.00Overlap 17.7410.058.1K-SVD For the Peppers image This is a more challenging case, where the DCT is not a suitable dictionary. For Redundant DCT we get RMSE=16.13, and For K-SVD (15 iterations) we get RMSE=12.74

25. Sparse representations for Image Decomposition 25 Inpainting Color Images

26. Sparse representations for Image Decomposition 26 Inpainting Video

27. Sparse representations for Image Decomposition 27 Patch-Based Impulse-Noise Removal

28. Sparse representations for Image Decomposition 28 Introduction  In the inpainting problem discussed above, the mask is KNOWN. What do we do if the mask (missing pixels) is unknown – this is the impulse noise problem.  Definition: we obtain an image with p% of the pixels in random locations corrupted by adding ±50 to the gray-level. This is known as “salt-and-pepper” noise.  This is an example with 10% corrupted pixels:  Most common method for filtering this noise: Median filtering

29. Sparse representations for Image Decomposition 29 Sparse-Land Alternative  We will use the median filter as a simple (and primitive) detector of missing pixels (there are better ways!):  Once the mask is available, we shall proceed with regular inpainting … BUT …  Applying the inpainting ONLY for the masked pixels  For a fair comparison, we will adopt the same mask for the median filter., i.e., modify the pixels having Mask=0 only.

30. Sparse representations for Image Decomposition 30 Results The image with the detected mask Masked median filter, RMSE=6.52 Inpainting by DCT, RMSE=5.86 Regular Median filter, RMSE=7.42