Patch-based Image Deconvolution via Joint Modeling of Sparse Priors Chao Jia and Brian L. Evans The University of Texas at Austin 12 Sep

Non-blind Image Deconvolution  Reconstruct natural image from blurred version  Camera shake; astronomy; biomedical image reconstruction  2D convolution matrix H and Gaussian additive noise vector n  Maximum a-posteriori (MAP) estimation for vector X  Prior model for p(X) for natural images? [Elad 2007]  Optimization method? 2

Analysis-based modeling [Krishnan 2009]  Prior based on hyper-Laplacian distribution of the spatial derivative of natural images  Linear filtering to compute spatial derivative  Fit ( ) and (normalization factor) to empirical data 3

Patch-based modeling  Sparse coding of patches  Spatial receptive fields of visual cortex [Olshausen 1997]  For patches  Learn an overcomplete dictionary from natural images.  Application in image restoration  Denoising, superresolution [Yang 2010]  Localized algorithm: patches can overlap  Use this model in deconvolution? [Lee 2007] 4

Prior model in natural images  From local to global  Slow convergence (EM Algorithm)  Patches should not overlap (Why?) boundary artifacts 5

Joint modeling  Take advantage of patch-based sparse representation while resolving the problems in?  Combine analysis-based prior and synthesis-based prior Sparse spatial gradient Patch-based sparse coding Accelerate convergence Keep consistency on the boundary of adjacent patches Keep details and textures 6

Joint modeling  Discard the generative model  Prior probability  After training, we fix the parameters for all images sparsity of representation coefficients compatibility term sparsity of gradients 7

MAP estimation using the joint model  Problem:  Iteratively updating w and X until convergence  w sub-problem small-scale L 1 regularized square loss minimization  X sub-problem Half-quadratic splitting [Krishnan 2009] likelihood prior 8

Experimental results  Initialization: Wiener estimates / blurred images  Dictionary: learned from Berkeley Segmentation database  Patch size  Prior parameters:  Runtime: (Matlab) 16s with Intel Core2 Duo  Experiment settings: 9

Experimental results PASCAL Visual Object Classes Challenge (VOC) 2007 database 10

Experimental results 11

Experimental results keeps more brick textures [Krishnan 2009] Original image Blurred imageProposed 12 [Portilla 2009]

Experimental results 13  Textures zoomed in [Krishnan 2009] Original image Proposed[Portilla 2009]

Conclusions  Global model for MAP estimation  Able to solve general non-blind image deconvolution  Joint model of image pixels and representation coefficients  Sparsity of spatial derivative (analysis-based)  Sparsity of representation of patches in overcomplete dictionary (synthesis-based)  Iterative algorithm  converges in a few iterations  Matlab code for the proposed method is available at 14

References  [Elad 2007] M. Elad, P. Milanfar and R. Rubinstein, “Analysis versus synthesis in signal priors”, Inverse Problems, vol. 23,  [Krishnan 2009] D. Krishnan and R. Fergus, “Fast image deconvolution using hyper-Laplacian priors,” Advances in Neural Information Processing Systems, vol. 22, pp. 1-9,  [Olshausen 1997] B.A. Olshausen and D.J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1,” Vision Research, vol. 37, no. 23, pp ,  [Portilla 2009] J. Portilla, “Image restoration through L0 analysis-based sparse optimization in tight frames,” in Proc. IEEE Int. Conf. on Image Processing, 2009, pp  [Yang 2010] J. yang, J. Wright, T.S. Huang and Y. Ma, “Image super- resolution via sparse representation,” IEEE Trans. on Image Processing, vol. 19, no. 11, pp ,

 Thank you! 16

w sub-problem patches do not overlap small-scale l 1 regularized square loss minimization 17

X sub-problem  Conjugate gradient iteratively reweighted least squares  Half-quadratic splitting [Krishnan 2009] auxiliary variable No need to solve the equation component-wise quartic function component-wise quartic function 18

MAP estimation using the joint model blurred image; noise level; blurring kernel; initialization of recovered image Update the coefficient of patches (w sub-problem) Set α=α 0 α>α max ? Update auxiliary variable Y (quartic equation) Update image X (FFT) α=kα X converges? finish X sub-problem 19

Image Quality Assessment 20  Full reference metric  ISNR -- increment in PSNR (peak signal-to-noise ratio)  SSIM -- structural similarity [Wang 2004]

Prior model of natural images  Analysis-based prior  Fast convergence  Over smooth the images  Synthesis-based prior (patch-based sparse representation)  Dictionary well adapted to nature images  Captures textures well  Slow convergence  Boundary artifacts 21

Computational complexity 22  Computational complexity  For each iteration:  N is the total number of pixels in the image  Average runtime comparison [Krishnan 2009][Portilla 2009]Proposed 2s15s16s