Image Enhancement by Regularization Methods Andrey S. Krylov, Andrey V. Nasonov, Alexey S. Lukin Moscow State University Faculty of Computational Mathematics and Cybernetics Laboratory of Mathematical Methods of Image Processing

Introduction Many image processing problems are posed as ill- posed inverse problems. To solve these problems numerically one must introduce some additional information about the solution, such as an assumption on the smoothness or a bound on the norm. This process was theoretically proven by Russian mathematician Andrey N. Tikhonov and it is known as regularization.

Outline Regularization methods Regularization methods Applications Applications –Resampling (interpolation) –Deringing (Gibbs effect reduction) –Super-resolution

Ill-posed Problems Formally, a problem of mathematical physics is called well-posed or well-posed in the sense of Hadamard if it fulfills the following conditions: 1. For all admissible data, a solution exists. 2. For all admissible data, the solution is unique. 3. The solution depends continuously on data.

Ill-posed Problems Many problems can be posed as problems of solution of an equation Many problems can be posed as problems of solution of an equation A is a linear continuous operator, Z and U are Hilbert spaces A is a linear continuous operator, Z and U are Hilbert spaces The problem is ill-posed and the corresponding matrix for operator А in discrete form is ill-conditioned The problem is ill-posed and the corresponding matrix for operator А in discrete form is ill-conditioned

Point Spread Function (PSF) Assume: Point light source PSF =

Convolution Model Notations –L: original image –K: the blur kernel (PSF) –N: sensor noise (white) –B: input blurred image B Generation rule: B = K L + N +

Deblur using Convolution Theorem Convolution Theorem: 8/38

Deblur using Convolution Theorem PSF BlurredImageRecovered 9/38

Noisy case Deconvolution is unstable 10/38

Variational regularization methods Tikhonov methods Tikhonov methods The Residual method (Philips) The Residual method (Philips) The Quasi-solution method (Ivanov) The Quasi-solution method (Ivanov) 11/38

Variational regularization methods Regularization method is determined by: Regularization method is determined by: A) Choice of solution space and of stabilizer B) Choice of C) Method of minimization A and B determine additional information on problem solution we want to use for solution of ill- posed problem to achieve stability A and B determine additional information on problem solution we want to use for solution of ill- posed problem to achieve stability 12/38

Outline Regularization methods Regularization methods Applications Applications –Resampling (interpolation) –Deringing (Gibbs effect reduction) –Super-resolution

Resampling: Introduction Interpolation is also referred to as resampling, resizing or scaling of digital images Interpolation is also referred to as resampling, resizing or scaling of digital images Methods: Methods: Linear non-adaptive (bilinear, bicubic, Lanczos interpolation) Linear non-adaptive (bilinear, bicubic, Lanczos interpolation) Non-linear edge-adaptive (triangulation, gradient methods, NEDI) Non-linear edge-adaptive (triangulation, gradient methods, NEDI) Regularization method is used to construct a non- linear edge-adaptive algorithm Regularization method is used to construct a non- linear edge-adaptive algorithm

Resampling: Linear and non-linear method bilinear interpolation bilinear interpolation non-linear method non-linear method

Resampling: Inverse problem Consider the problem of resampling as Consider the problem of resampling as Problem: operator A is not invertible Problem: operator A is not invertible z is unknown high-resolution image, u is known low-resolution image, A is the downsampling operator which consists of filtering H and decimation D

Resampling: Regularization We use Tikhonov-based regularization method where We use Tikhonov-based regularization method where

Resampling: Regularization Choices of regularizing term (stabilizer) Choices of regularizing term (stabilizer) –Total Variation –Bilateral TV and are shift operators along x and y axes by s and t pixels respectively, p = 1, γ = 0.8

Resampling: Regularization Minimization problem Minimization problem Subgradient method Subgradient method

Resampling: Results Linear method Linear method Regularization-based method Regularization-based method Gibbs phenomenon Gibbs phenomenon

Image Enhancement by Regularization Methods Introduction to regularization Introduction to regularization Applications Applications –Resampling (interpolation) –Deringing (Gibbs effect reduction) –Super-resolution

Total Variation Approach for Deringing Gibbs effect is related to Total Variation Gibbs effect is related to Total Variation High TV, very notable Gibbs effect (ringing) Low TV

Total Variation Regularization methods Tikhonovs approach Tikhonovs approach Rudin, Osher, Fatemi method Rudin, Osher, Fatemi method Ivanovs quasi-solution method Ivanovs quasi-solution method

Global and Local Deringing Two approaches for Deringing Two approaches for Deringing –Global deringing Minimizes TV for entire image Minimizes TV for entire image In this case, we use Tikhonov regularization method In this case, we use Tikhonov regularization method No ways to estimate regularization parameter, details outside edges may be lost No ways to estimate regularization parameter, details outside edges may be lost –Local deringing Used if we have information on TV for small rectangular areas Used if we have information on TV for small rectangular areas In this case, we use Ivanovs quasi-solution method for small overlapping blocks In this case, we use Ivanovs quasi-solution method for small overlapping blocks

Deringing after interpolation Deringing after interpolation Deringing after interpolation –We know information on TV for blocks of initial image to be resampled –We suggest that TV does not change after image interpolation Thus we have real algorithm to find regularization parameter for deringing after image resampling task

Minimization Tikhonov regularization method Tikhonov regularization method –Subgradient method Quasi-solution method Quasi-solution method –1D Conditional gradient method (there is no effective 2D implementation) –In 2D case, we divide an image into a set of rows and process these rows by 1D method, next we do the same with columns and finally we average these results

Minimization Conditional gradient method Conditional gradient method –Conditional gradient method is used to minimize a convex functional on a convex compact set. The key idea of this method is that step directions are chosen among the vertices of the set of constraints, so we do not fall outside this set during minimization process –Conditional gradient method is effective only for small images, so it is used for local deringing only

Resampling + Deringing: PSNR Results After resampling28.38 Global deringing Local deringing Conditional gradient method Subgradient method A set of 100 nature and architecture images with 400x300 resolution (11x11 blocks, 1813 per image) was used to test the methods. We downsampled the images by 2x2 using Gauss blur with radius 0.7 and then upsampled them by our regularization algorithm. Next we applied deringing methods and compared the results with initial images.

Resampling + Deringing: Results regularization-based interpolation regularization-based interpolation application of quasi- solution method application of quasi- solution method

Resampling + Deringing: Results Source image, upsampled by box filterLinear interpolationRegularization-based method Regularization-based interpolation + quasi-solution deringing method

Image Enhancement by Regularization Methods Introduction to regularization Introduction to regularization Applications Applications –Resampling (interpolation) –Deringing (Gibbs effect reduction) –Super-resolution

Super-Resolution: Introduction The problem of super-resolution is to recover a high- resolution image from a set of several degraded low- resolution images The problem of super-resolution is to recover a high- resolution image from a set of several degraded low- resolution images Super-resolution methods Super-resolution methods –Learning-based – single image super-resolution, learning database (matching between low- and high-resolution images) –Reconstruction-based – use only a set of low-resolution images to construct high-resolution image

Super-Resolution: Inverse Problem The problem of super-resolution is posed as error minimization problem The problem of super-resolution is posed as error minimization problem z – reconstructed high-resolution image z – reconstructed high-resolution image v k – k-th low-resolution input image v k – k-th low-resolution input image A k – downsampling operator, it includes motion information A k – downsampling operator, it includes motion information

Super-Resolution: Downsampling operator A k – downsampling operator A k – downsampling operator H cam – camera lens blur (modeled by Gauss filter) H cam – camera lens blur (modeled by Gauss filter) H atm – atmosphere turbulence effect (neglected) H atm – atmosphere turbulence effect (neglected) n – noise (ignored) n – noise (ignored) F k – warping operator – motion deformation F k – warping operator – motion deformation

Super-Resolution: Warping operator Warping operator F k Warping operator F k

Super-Resolution: Regularization The problem is ill-posed, and we use Tikhonov regularization approach (same as in resampling) The problem is ill-posed, and we use Tikhonov regularization approach (same as in resampling) where, Minimization by subgradient method Minimization by subgradient method

Super-Resolution: Results Linear methodPixel replicationNon-linear methodSuper-resolution Face super-resolution for the factor of 4 and 10 input images Source images

Super-Resolution: Results The reconstruction of an image from a sequence examples of input frames (of total 14) linearly interpolated single frame super-resolution result

Super-Resolution for Video Super-Resolution For every frame, we take current frame, 3 previous and 3 next frames. Then we process it by super-resolution. current frame

Super-Resolution: Results for Video Nearest neighbor interpolationSuper-Resolution Super-Resolution for video for a factor of 4

Super-Resolution: Results for Video Bilinear interpolationSuper-Resolution Super-Resolution for video for a factor of 4

Super-Resolution: Results for Video Bicubic interpolationSuper-Resolution Super-Resolution for video for a factor of 4

Conclusion Increasing CPU and GPU power makes regularization methods more and more important in image processing Increasing CPU and GPU power makes regularization methods more and more important in image processing Regularization is a very powerful tool but each specific image processing problem needs its own regularization method Regularization is a very powerful tool but each specific image processing problem needs its own regularization method

Thank you!