1. Patch-based Image Deconvolution via Joint Modeling of Sparse Priors Chao Jia and Brian L. Evans The University of Texas at Austin 12 Sep 2011 1

2. Non-blind Image Deconvolution  Reconstruct natural image from blurred version  Camera shake; astronomy; biomedical image reconstruction  2D convolution matrix H and Gaussian additive noise vector n  Maximum a-posteriori (MAP) estimation for vector X  Prior model for p(X) for natural images? [Elad 2007]  Optimization method? 2

3. Analysis-based modeling [Krishnan 2009]  Prior based on hyper-Laplacian distribution of the spatial derivative of natural images  Linear filtering to compute spatial derivative  Fit (0.5-0.8) and (normalization factor) to empirical data 3

4. Patch-based modeling  Sparse coding of patches  Spatial receptive fields of visual cortex [Olshausen 1997]  For 10 10 patches  Learn an overcomplete dictionary from natural images.  Application in image restoration  Denoising, superresolution [Yang 2010]  Localized algorithm: patches can overlap  Use this model in deconvolution? [Lee 2007] 4

5. Prior model in natural images  From local to global  Slow convergence (EM Algorithm)  Patches should not overlap (Why?) boundary artifacts 5

6. Joint modeling  Take advantage of patch-based sparse representation while resolving the problems in?  Combine analysis-based prior and synthesis-based prior Sparse spatial gradient Patch-based sparse coding Accelerate convergence Keep consistency on the boundary of adjacent patches Keep details and textures 6

7. Joint modeling  Discard the generative model  Prior probability  After training, we fix the parameters for all images sparsity of representation coefficients compatibility term sparsity of gradients 7

8. MAP estimation using the joint model  Problem:  Iteratively updating w and X until convergence  w sub-problem small-scale L 1 regularized square loss minimization  X sub-problem Half-quadratic splitting [Krishnan 2009] likelihood prior 8

9. Experimental results  Initialization: Wiener estimates / blurred images  Dictionary: learned from Berkeley Segmentation database  Patch size 12 12  Prior parameters:  Runtime: (Matlab) 16s with Intel Core2 Duo CPU @2.26GHz  Experiment settings: 9

10. Experimental results PASCAL Visual Object Classes Challenge (VOC) 2007 database 10

11. Experimental results 11

12. Experimental results keeps more brick textures [Krishnan 2009] Original image Blurred imageProposed 12 [Portilla 2009]

13. Experimental results 13  Textures zoomed in [Krishnan 2009] Original image Proposed[Portilla 2009]

14. Conclusions  Global model for MAP estimation  Able to solve general non-blind image deconvolution  Joint model of image pixels and representation coefficients  Sparsity of spatial derivative (analysis-based)  Sparsity of representation of patches in overcomplete dictionary (synthesis-based)  Iterative algorithm  converges in a few iterations  Matlab code for the proposed method is available at http://users.ece.utexas.edu/~bevans/papers/2011/sparsity/ 14

15. References  [Elad 2007] M. Elad, P. Milanfar and R. Rubinstein, “Analysis versus synthesis in signal priors”, Inverse Problems, vol. 23, 2007.  [Krishnan 2009] D. Krishnan and R. Fergus, “Fast image deconvolution using hyper-Laplacian priors,” Advances in Neural Information Processing Systems, vol. 22, pp. 1-9, 2009.  [Olshausen 1997] B.A. Olshausen and D.J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1,” Vision Research, vol. 37, no. 23, pp. 3311-3325, 1997.  [Portilla 2009] J. Portilla, “Image restoration through L0 analysis-based sparse optimization in tight frames,” in Proc. IEEE Int. Conf. on Image Processing, 2009, pp. 3909-3912.  [Yang 2010] J. yang, J. Wright, T.S. Huang and Y. Ma, “Image super- resolution via sparse representation,” IEEE Trans. on Image Processing, vol. 19, no. 11, pp. 2861-2873, 2010. 15

16.  Thank you! 16

17. w sub-problem patches do not overlap small-scale l 1 regularized square loss minimization 17

18. X sub-problem  Conjugate gradient iteratively reweighted least squares  Half-quadratic splitting [Krishnan 2009] auxiliary variable No need to solve the equation component-wise quartic function component-wise quartic function 18

19. MAP estimation using the joint model blurred image; noise level; blurring kernel; initialization of recovered image Update the coefficient of patches (w sub-problem) Set α=α 0 α>α max ? Update auxiliary variable Y (quartic equation) Update image X (FFT) α=kα X converges? finish X sub-problem 19

20. Image Quality Assessment 20  Full reference metric  ISNR -- increment in PSNR (peak signal-to-noise ratio)  SSIM -- structural similarity [Wang 2004]

21. Prior model of natural images  Analysis-based prior  Fast convergence  Over smooth the images  Synthesis-based prior (patch-based sparse representation)  Dictionary well adapted to nature images  Captures textures well  Slow convergence  Boundary artifacts 21

22. Computational complexity 22  Computational complexity  For each iteration:  N is the total number of pixels in the image  Average runtime comparison [Krishnan 2009][Portilla 2009]Proposed 2s15s16s