1. Image Priors and Optimization Methods for Low-level Vision Dilip Krishnan TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AA A AA A A A

2. Introduction We consider underconstrained image processing problems such as denoising, deblurring and image inpainting. Also called Linear Inverse Problems. To overcome ill-posed nature and to introduce robustness to noise, image priors (regularization) are needed.

3. Introduction Canonical problem for prior-based methods: There are 2 issues one seeks to address: Develop better priors: lead to higher quality results (visual and/or SNR). Develop fast numerical algorithms to solve the resulting problems (to handle large image sizes) fast and accurately.

4. Existing Approaches to Denoising Prior-based Methods: Nonlinear Total Variation based Noise Removal Algorithms, Rudin, Osher, Fatemi, 1992. Multiscale (wavelet) Decomposition Methods: Image Denoising using Scale Mixtures of Gaussians in the Wavelet Domain, Portilla, Strela, Wainwright, Simoncelli, 2003.

5. Existing Approaches to Denoising Dictionary-based Methods: Learning Multiscale Sparse Representations for Image and Video Restoration, Mairal, Shapiro, Elad, 2008. Filtering Methods: A Fast Approximation of the Bilateral Filter using a Signal Processing Approach, Paris, Durand, 2006. Extension: called non-local means which provides state-of-the art results.

6. Existing Approaches to Non-Blind Deblurring In non-blind deblurring, we assume that the blurring kernel (blur filter) is known. Wiener Filter: Classical method; tries to find an ‘inverse blur filter’ based on noise characteristics – requires knowledge of unknown signal and noise. Richardson-Lucy: Simple iterative method – causes ringing.

7. Existing Approaches to Non-Blind Deblurring Prior-based methods: l_1: Linearized Bregman Methods for Frame-Based Image Deblurring, Cai, Osher, Shen, 2009. Total Variation: A Nonlinear Primal-Dual Method for Total Variation-Based Image Restoration, Chan, Golub, Mulet, 1999: inspired by earlier idea of Conn and Overton on joint primal- dual minimization for minimizing sum of Euclidean norms. Sub l_1 on gradients: Fast Image Deconvolution using Hyper-Laplacian Priors, Krishnan, Fergus 2009.

8. Existing Approaches to Blind Deconvolution Blurring kernel is unknown. Difficult problem as blurring kernel and image are all unknown, and there is noise in the image. Prior-based methods: l_1:, Blind Motion Deblurring from a single image using Sparse Approximation Priors, Cai et. al. 2009. Gradient Based: Removing Camera Shake from a Single Photograph, Fergus et. al. 2006. Sub l_1 on gradients: Blind Image Deblurring using Image Statistics, Levin, 2006.

9. Spatially Varying Deconvolution Most generalized (and therefore, difficult) problem where the blur kernel is spatially varying. Potentially, there could be a different blur kernel at every point in the image. Only gradient-based priors used currently.

10. Commonly Used Priors and Related Assumptions prior : Image to be recovered is sparse under some transformation (e.g. wavelet); problem formulation: : inverse wavelet transform; : blur kernel; Other possibilities are DFT, curvelets etc.

11. Commonly Used Priors and Related Assumptions Total Variation: Gradient of image is sparse i.e. image is piecewise smooth (many flat areas, sharp transitions): Content aware prior: Cho et. al. 2010: Spatially varying prior that depends on image content:

12. Commonly Used Priors and Related Assumptions Sparsity under learned filters: Natural Image Statistics and Efficient Coding, Olshausen and Field 1996. Filters are learnt using a neural network; and sparsity is imposed on the coefficients of the image convolved with these filters.

13. Commonly Used Priors and Related Assumptions Examples of such learned filters (from the paper):

14. Comments on Priors These priors work reasonably well for denoising; and other approaches are also available for denoising problem. Not so well for deblurring or blind deconvolution. Cost of prior is higher for unblurred images than for blurry images. This motivates the search for new priors.

15. Optimization Techniques for We consider commonly used techniques for solving problems of the type: Problems arise in deblurring, denoising and compressed sensing. Large size of images makes efficient techniques a necessity. Even an average sized 1000 x 1000 image means ~1 million unknowns. Second order methods such as interior point are infeasible for large number of unknowns.

16. Optimization Techniques for Reference: L1-L2 Optimization in Signal and Image Processing, Zibulevsky and Elad, 2010. Wide variety of techniques are used: mostly first- order or iterative methods. Iterative Shrinkage Methods (ISTA) are of the type: S is a component-wise shrinkage operation. Derived from a quadratic approximation to the original problem.

17. Optimization Techniques for This basic shrinkage was recently accelerated using Nesterov methods: A Method for Solving the Convex Programming Problem with Convergence Rate O(1/k^2), Y. E. Nesterov, 1983. Resulting algorithm is called FISTA:

18. Optimization Techniques for Bregman-type methods: Developed recently by Osher and fellow-researchers at UCLA. Motivation: use a different approximation of the objective function based on Bregman distance. Interestingly, the resulting algorithm actually ends up very similar to Iterative Shrinkage Methods:

19. Optimization Techniques for Now consider TV-regularized problems: Previous methods for can be extended by using splitting techniques. Introduce dummy variables and make a new problem: Then solve alternately for x,w. Problem in x is quadratic – often, can be solved very fast (using FFT’s). Problem in w is of type. The Split Bregman method for L1 Regularized Problems, Goldstein et. al., 2009.

20. Optimization Techniques for Problems of the type: D(x) is identity or gradient operator. Reweighted least squares can be used: slow, but accurate: Iteratively Re-weighted Least Squares Minimization for Sparse Recovery, Daubechies et. al., 2009. Splitting methods can also be used in conjuction with look-up tables to solve the seperable shrinkage problem: very effective in practice.

21. Previous Work

22. Overview Dark Flash Photography: SIGGRAPH 2009. Enables dazzle-free photography in low-light conditions. Fast Image Deconvolution using Hyper Laplacian Priors: NIPS 2009. Fast scheme to solve deconvolution/denoising problems that use nonconvex priors.

23. Overview Dark Flash Photography: SIGGRAPH 2009. Enables dazzle-free photography in low-light conditions. Fast Image Deconvolution using Hyper Laplacian Priors: NIPS 2009. Fast scheme to solve deconvolution/denoising problems that use nonconvex priors.

25. Our Camera & Dark Flash Dark Flash Emits Ultraviolet (UV) and Infrared (IR) light just outside visible wavelength range

26. Dark Flash Photography Dark flash is ~200 times dimmer than conventional Ground truthReconstruction2. Ambient image1. Dark Flash image

27. Key Challenges 1.How to add light to the scene without it being perceived by people. 2.How to obtain an image with correct colors.

28. Human Retina Spectral Sensitivity (Vos 1978)

29. Normal Camera Spectral Sensitivity Spectral response curves for an unmodified camera

30. Modify standard camera: 1. Remove IR-block sensor filter 2. Add lens filter to remove IR >800nm  Some sensitivity in UV & IR just outside visible Dark Flash Camera Spectral Sensitivity IR

31. Comparison with Human Sensitivity Camera’s response is significantly broader Possible to add light without being easily perceived Human Response Curve IR

32. Dark Flash Emission Spectrum Camera Spectral Sensitivity Dark Flash Emission

33. Key Challenges 1.How to add light to the scene without it being perceived by people. 2.How to obtain an image with correct colors.

34. Two Images: Five Spectral Bands In Dark Flash image: – “Blue” channel records UV – “Red” channel records IR Assumptions 1.Little ambient UV and IR light 2.UV/IR flash dominates ambient visible light 2. Ambient RGB 1. Dark Flash IR UV

35. Reconstruction of Red Channel Ambient RedGround Truth Red Intensities Compare to ground truth reference Ambient red has correct intensities but is noisy

36. Intensity Constraint Ambient RedOutput Red Intensities Gradients ?? ≈

37. Spectral Constraint Gradients in Red and IR are similar at most pixels But differ when material changes Ground Truth Gradients in Red Gradients in IR Need to allow occasional disagreement between gradients

38. Spectral Constraint (Also for UV flash) Gradients of Output Red 0.7 - Gradients in IR ? Minimize:

39. Final Cost Function Sum over all pixels. = 0.7. is IR channel; is UV. 3 cost functions – one for each channel. and are weighting terms. - mask to handle specularities/highlights UV Spectral Term IR Spectral Term Likelihood Sparsity

40. Implies that spectral reflectances are the same in UV/IR and visible NOT TRUE Effect of Varying α in Spectral Term Non-convex optimization Models actual distribution Later part of talk focuses on fast solution scheme. Convex optimization Fast: ~2 mins in Matlab for 1.4 megapixel image Can be easily be implemented on GPU Best results Acceptable results Unacceptable results

41. Results

42. Ambient : 1/20 th secReconstructionLong exposure: 4 sec

43. 1/64 th 1/90 th 1/256 th Reconstruction Ambient (Fraction of Normal Illumination)

44. Limitations Rudimentary handling of shadows and highlights – Currently use simple mask (as in existing flash/no-flash work) Need some ambient illumination – Won’t work in complete darkness Color drift & loss of detail at very high noise levels Lack of UV/IR edges causes reconstruction to suffer Taking Photos in Low Light Levels

45. Overview of Talk Dark Flash Photography: SIGGRAPH 2009. Enables dazzle-free photography in low-light conditions. Fast Image Deconvolution using Hyper Laplacian Priors: NIPS 2009. Fast scheme to solve deconvolution problems that use nonconvex priors.

46. Hyper-Laplacian Priors Good models of gradient distributions in natural images. Heavy tails are well modeled. Used in deblurring, denoising, super-resolution etc. Drawback: cause problems to become non-convex.

47. Non-blind Deconvolution We consider the standard non-blind deconvolution problem: Likelihood Prior For >= 1, problem is convex and a global minimum can be found. We consider the solution of this problem when < 1. We want, given blurry/noise image and kernel. are gradient filters.

48. Existing Methods Iterative Reweighted Least Squares/Conjugate Gradients Very slow for large image sizes. IRLS takes ~ 1 hour for a 1 megapixel image. E.g.: Levin et. al. SIGGRAPH 2007. For = 1, Bregman iterations/iterative shrinkage techniques work well. Second-order methods are infeasible.

49. Our Algorithm We use a splitting method. Introduce a new auxiliary variable for each derivative filter. Old cost function: New cost function:

50. Our Algorithm We use a splitting method. Introduce a new auxiliary variable for each derivative filter. Old cost function: New cost function: Split

51. Our Algorithm As, the solution to this problem approaches that of the original problem. Now we can use an alternating mechanism to first minimize for, and then minimize for each. Coupled with a continuation method on.

52. x sub-problem is kept fixed from previous iteration. sub-problem is quadratic: Can be solved using FFTs (assuming circular boundary conditions). Need to solve: For 2 derivative filters, we require only 3 FFT’s at every iteration.

53. w sub-problem sub-problem is component-wise separable; is fixed from previous iteration. For a fixed pixel and filter, it reduces to: Simplified: Can be solved very accurately using Lookup Tables. For the problem can be solved analytically.

54. Overall Algorithm A : Start with a small value of. Alternate between and sub-problems till convergence. Increase the value of by a small amount. GOTO A. Achieves over 2 orders of magnitude speedup over IRLS with similar SNR quality. Timing for de-blurring with a 13 x 13 kernel: Image SizeIRLSOurs 1024 x 10242.34 1281.32.78 3072 x 3072 22.40> 1000024.07

55. Original Blurred SNR=7.31 Ours SNR=18.96 t=1.2 IRLS SNR=19.05 t=483.9

56. Dark Flash Result Long Exposure Ambient 2.5 Lux IRLS ~ 1 hour Ours ~ 1 minute

57. Summary and Drawbacks Fast method for nonconvex nonblind deconvolution Orders of magnitude faster than IRLS. Can be used for other applications. – E.g. super-resolution, dark flash, denoising. FFTs and LUTs are well suited for hardware implementation. Circular boundary conditions cause boundary artefacts – negligible as resolution increases.