1. School of Electronic Engineering, Xidian University, Xi’an, China
Sparsity-based Image Interpolation With Nonlocal Autoregressive Modeling
Weisheng Dong (董伟生)
School of Electronic Engineering, Xidian University, Xi’an, China
Sample codes available:
W. Dong, L. Zhang, R. Lukac, and G. Shi, “Sparse representation based image interpolation with nonlocal autoregressive modeling,” IEEE Trans. Image Process., vol. 22, no. 4, Apr. 2013
11/15/2018

2. Outline Experimental results Conclusions
Background: image interpolation
Edge-directed methods
Sparsity-based methods
Sparse image interpolation with nonlocal autoregressive modeling
The nonlocal autoregressive modeling
The clustering-based sparsity regularization
Experimental results
Conclusions
First, I will briefly introduce the classic image interpolation methods,
Then I will introduce the two key components of the proposed method, i.e., the nonlocal autoregressive modeling, and the clustering-based sparsity regularization.
And report the experimental comparison results.

3. Image Upconversion is Highly Needed
Small images
Display small images / video in HD devices
SD videos
Related applications: deinterlacing:
11/15/2018

4. Challenge
Jagged artifacts
Original HR image
2X upscaling
LR image
Challenge: How can we upconvert the image without annoying jagged artifacts?
11/15/2018

5. Previous work: edge-based methods
Linear interpolators: bilinear, bicubic
Blurring edges, annoying jagged artifacts
Edge directed interpolators (EDI) [TIP’95]
Interpolate along edge directions
Difficult to estimate the edge directions
from LR image
New EDI [TIP’01], Soft-decision adaptive inter. [TIP’08]
Local autoregressive (AR) model of LR image
Ghost artifacts due to Wrong AR models
Smooth variations
As introduced by Yaniv’s, the image interpolation is very challenging.
When there is no low-pass filtering, then the image super-resolution problem will become more challenging. The problem is often called as image interpolation. In this problem, the low-resolution image is obtained by direct downsampling, and there is no smoothing filtering.
Image interpolation problem has also been extensively studied in the past decade. Classic image interpolation methods include linear interpolation methods, e.g., bilinear, bicubic.
For better image interpolation results, edges directed interpolation methods try to interpolate the images along edges, e.g., the well-known NEDI method, and the SAI method. The SAI method is the current state-of-the-art method.
Sparsity-based interpolation method, such as the sparse mixing estimators by Steph Mallat has also been proposed. We have also proposed an effective image interpolation method by using the CSR model.
11/15/2018

6. Artifacts of Edge-based methods
Original
Bicubic
NEDI [TIP’01]
SAI [TIP’08]
11/15/2018

7. Previous works: sparsity-based methods
Dictionary learning based method, [TIP’10]; Nonlocal method, [ICCV’09];
Failed when the LR image is directly downsampled
Bicubic (32.07 dB)
ICCV09 (29.87 dB)
Bicubic (23.48 dB)
ICCV09 (21.88 dB)
As introduced by Yaniv’s, the image interpolation is very challenging.
When there is no low-pass filtering, then the image super-resolution problem will become more challenging. The problem is often called as image interpolation. In this problem, the low-resolution image is obtained by direct downsampling, and there is no smoothing filtering.
Image interpolation problem has also been extensively studied in the past decade. Classic image interpolation methods include linear interpolation methods, e.g., bilinear, bicubic.
For better image interpolation results, edges directed interpolation methods try to interpolate the images along edges, e.g., the well-known NEDI method, and the SAI method. The SAI method is the current state-of-the-art method.
Sparsity-based interpolation method, such as the sparse mixing estimators by Steph Mallat has also been proposed. We have also proposed an effective image interpolation method by using the CSR model.
11/15/2018

8. Previous works: sparsity-based methods
Upscaling via solving a sparse coding problem:
Failed too, ringing and zippers artifacts
Original
DCT (32.49 dB)
PCA (32.50 dB)
This figure shows some results of the interpolated images by the L1-norm sparsity regularization methods. Sub-figure (a) is the original image; (b) is the result recovered using the DCT basis; (c) is the HR image recovered using the local PCA bases.
From this figure we can see that the L1-norm sparsity method fail to reconstruct satisfied HR image. There are many jaggy artifacts in the interpolated images.
11/15/2018

9. Coherence issue of sparsity method
Two fundamental premises of sparsity recovery method:
Incoherence sampling: sampling operator D and the dictionary Φ should be incoherent. The coherence value:
Sparsity: the original image should have sparse representation over Φ
Problem: The downsampling matrix D is often coherent with common dictionaries, e.g., wavelets, K-SVD dictionary
According to the compressive sensing theory, in order for a faithful reconstruction of the original image x from linear measurements y, the following conditions should be satisfied.
First, the degradation matrix, here is the downsampling matrix D, should be incoherent with the dictionary. The coherence value between then can be computed by this formula.
Second, the original image should have sparse representation under dictionary \Phi.
But in image interpolation, the downsampling matrix is usually coherent with common bases or dictionaries, e.g., wavelets, K-SVD dictionary.
Therefore, we cannot directly use the sparse model / CSR model for image interpolation.
D. Donoho and M. Elad, PNAS, 2003.
E. Candes, et al., “An introduction to Compressive sensing,” IEEE SPM, 2008
11/15/2018

10. Contributions
Propose a new image upscaling framework: combining edge-based method and sparsity-based method
Suppress the artifacts of edge-based method using sparse regularization
Overcome the coherence issue by nonlocal autoregressive modeling
Introducing a structured sparse regularization model
Improve the sparse regularization performance
According to the compressive sensing theory, in order for a faithful reconstruction of the original image x from linear measurements y, the following conditions should be satisfied.
First, the degradation matrix, here is the downsampling matrix D, should be incoherent with the dictionary. The coherence value between then can be computed by this formula.
Second, the original image should have sparse representation under dictionary \Phi.
But in image interpolation, the downsampling matrix is usually coherent with common bases or dictionaries, e.g., wavelets, K-SVD dictionary.
Therefore, we cannot directly use the sparse model / CSR model for image interpolation.
11/15/2018

11. Local autoregressive (AR) modeling
Local AR image modeling
Compute the AR model: least-square
Exploiting the local image structure
AR-based image interpolation methods:
NEDI, [TIP 2001] (Over citation)
SAI, [TIP 2008] (Previous state-of-the-art method)
The Autoregressive model is a classic image model, widely used in image processing, e.g., image compression, image interpolation.
In AR modeling, we assume that each pixel can be well predicted by its local neighbors as a linear combination. The regression coefficients can be computed using the least-square method.
The AR model is effective in exploiting the local image structure, and has been successfully used in image interpolation. For example, the NEDI method and the soft-decision adaptive method .
11/15/2018

12. Nonlocal self-similarity
The local AR model is very effective in exploiting the local image structures. On the other hand, it has been well-known that natural images contain rich self-similarity structures.
11/15/2018

13. Nonlocal NAR modeling (NARM)
Nonlocal regression:
Patch matching: nonlocal similar neighbor selection
Regularized least-square:
Nonlocal AR modeling: x = Sx + ex
To exploit both the local and nonlocal redundancies, we extend the local AR model to the nonlocal autoregressive model.
In natural images, for each patch, we can find many local and nonlocal patches similar to it. Then each pixel can be well approximated by its local and nonlocal neighbors, i.e., xi equals to the weighted sum of its nonlocal neighbors. Then the overall image x can be expressed by this formula.
The weights are computed by the regularized Least-square method, and it can be computed efficiently using a conjugate gradient (CG) method.
The nonlocal AR model is used to improve the observation model. And the new observation model is formulated by this formula.
11/15/2018

14. NARM based image interpolation
Improved objective function
S: impose structural constraint on Φ
The benefit of the NARM
The coherence value between new sampling matrix DS and Φ is significantly decreased
Coherence values (8x8 patch): wavelets: 1.20~4.77; Local PCA: 1.05~4.08
By introducing the nonlocal AR model into the observation model, the proposed objective function for image interpolation can be expressed as this equally constrained minimization problem. Here, matrix S acts as a conventional blurring or compressive sensing kernel, but it has different physical meaning. It dependent on the image content, and will be iteratively updated using the recovered images.
With the nonlocal AR model, we found that the coherence values between the new sampling matrix DS and the dictionary is significantly decreased. Thus, the sparse estimation becomes much more stable.
The regularization term used here is the CSR penalty term, and the dictionary here is the local PCA dictionaries.
11/15/2018

15. NARM based image interpolation
Local PCA bases:
(a)
(b)
(c)
(d)
(e)
(a) 1=3.78 vs. 2=1.60; (b) 1=3.28 vs. 2=1.80; (c) 1=3.22 vs. 2=2.08; (d) 1=6.41 vs. 2=1.47; (e) 1=7.50 vs. 2=1.69.
μ1 -- coherence values between D and Φ
μ2 -- coherence values between DS and Φ
11/15/2018

16. Structural sparse regularization
Conventional sparsity regularization
Cannot exploit structural dependencies between nonzero coefficients
Clustering-based sparse representation (CSR) [Dong, Zhang, et al., TIP 2013]
Unify structural clustering and sparse representation into a variational framework
In addition to the coherence issue, designing an appropriate sparsity regularizer is also very important.
Using nonlocal AR model, the L1-norm sparsity-based image interpolation problem can be formulated as this formula.
However, it has been well-known that the L1-norm sparsity regularizer cannot exploit the structural dependencies between nonzero coefficients.
In this work, we propose to use the clustering-based sparse representation to better regularize the solution space.
The regularization term used here is the CSR penalty term, and the dictionary here is the local PCA dictionaries.
11/15/2018

17. Clustering-based Sparse representation
Motivation: Nonzero sparse coefficients are NOT randomly distributed
The motivation behind our work lies in the fact that the nonzero sparse coefficients are NOT randomly distributed, instead they exhibit repetitive structures in the sparse domain.
The fist row shows the original natural images; the second row plots the distributions of the sparse coefficients of the K-SVD dictionary. Only the coefficients associated with the 3rd atoms are plotted.
Form this figure, we can see that the nonzero-coefficients are local structured, and also exhibit nonlocal self-similarity property. Unfortunately, the most of the existing sparse models don’t exploit the structured correlation between the nonzero coefficients.
Since the nonzero coefficients exhibit nonlocal repetitive patterns, the clustering provides a convenient tool for exploiting such nonlinear correlation.
The distribution of the sparse coefficients associated with the 3rd atoms in the K-SVD approach
11/15/2018

18. Clustering-based Sparse representation
The clustering-based regularization
Exploiting the self-similarity:
Unifying the clustering-based sparsity and the learned local sparsity
To exploit the structural information of the nonzero coefficients, we introduce a clustering-based regularization. We use a clustering algorithm to learn the centroids \mu_k from the image patches, and enforce the similarity between the centroid and the corresponding patches. The centroids can also be represented with the dictionary \phi.
To see how the clustering can promote the sparsity, we unify the clustering-based regularization term into the sparse representation model. An intuitive interpretation of the proposed objective function is that the sparse coefficients are en-coded with respect to the centroids.
local sparsity
Clustering-based regularization
11/15/2018

19. Clustering-based Sparse representation
Final CSR objective function
The unitary property of the dictionary
What does the CSR model mean?
Encode the sparse codes with the learned exemplars
Unify dictionary learning and structure clustering into a variational framework
We arrive at the final objective function through the following two steps:
First, we assume that dictionary is unitary and we can then remove the dictionary \phi from the clustering-based regularization term.
Second, inspired by the success of the L1-regularization in compressive sensing, we replace the L2-norm in the clustering-based sparsity regularization term with the L1-norm.
Then we get this double header L1-minimization problem. This is our final objective function of the proposed CSR model.
From the final objective function, we can see that the dictionary learning and the structure clustering are unified in a variational framework, and the sparse codes are encoded with the learned exemplars.
11/15/2018

20. Bayesian interpretation of CSR
The connection between the sparse representation and the Bayesian denoising, e.g., wavelet soft-thresholding
The connection between CSR and the MAP estimator
Laplacian Prior
Gaussian likelihood term
To better understand the proposed CSR model, we give a Bayesian interpretation of the CSR model.
In the past decades, the connection between the sparse representation and the Bayesian denoising has been well established. The sparse representation model is in fact equivalent to this MAP estimator. The first term is the Gaussian likelihood term, and the second term is the Laplaician prior term.
The basic idea behind the CSR model is to treat the centroids of the K clusters as the hidden variables to the sparse coefficients, we can then formulate this MAP estimator. The first term is the Gaussian likelihood term and the second is the joint prior distribution of the sparse codes and the learned centroids.
The joint prior term !
11/15/2018

21. Bayesian interpretation of CSR
The factorization of the joint prior term
where is the prediction residual / noise
assume that and are independent, i.e.,
Generally, the joint prior probability density function (PDF) of \alpha and \beta is very hard to estimate. In this work, we use the structural constraint to approximate this joint prior PDF. The basic idea is to introduce the \gamma, which is the prediction residual of \alpha using \beta. Then we can rewrite the joint PDF as this.
Since this deviation prediction can be viewed as another level of sparse coding, the \gamma is approximately independent from \alpha. In this work, we choose to model \gamma and \alpha by i.i.d Laplaican distribution, and we can write the joint prior model explicitly.
Gaussian
prior
Laplacian
prior
11/15/2018

22. Iterative reweighted CSR
The final MAP estimator
The iterative reweighted CSR
Adaptively estimate λ and η for each local sparse coefficients
Update λ and η iteratively
By substituting the likelihood term and the prior term into the MAP estimator, we obtain the exact objection function of the CSR model. From the MAP estimator, we can write the two regularization parameters, \lambda_1 and \lambda_2 using this formula.
In traditional sparse representation framework, we usually have to manually optimize the sparsity regularization parameters! In contrast, in our work, by using this Bayesian interpretation, we can adaptive compute the two sparsity regularization parameters in an iterative process.
11/15/2018

23. The proposed objective function
Adaptive selection of the dictionary: local PCA
Variable splitting:
βk update:
Since we are using the patch-based sparse model, we introduced an auxiliary variable x into the objective function, and the final objective function is formulated as this formula. The x denotes desired high-resolution whole image; \alpha_i denotes the sparse code of the patch x_i with respective to dictionary \phi_k.
By introducing the auxiliary variable x, we split the objective function into two sub-problems, i.e., the x-subproblem, and the \alpha-subproblem.
The x-subproblem is a equally constrained quadratic optimization problem; and the \alpha-subproblem is a CSR sparse coding problem. Both subproblems can be easily solved.
11/15/2018

24. Alternative optimization algorithm
α-subproblem: for each i
Closed-form solution: bi-variate shrinkage operator
Since we are using the patch-based sparse model, we introduced an auxiliary variable x into the objective function, and the final objective function is formulated as this formula. The x denotes desired high-resolution whole image; \alpha_i denotes the sparse code of the patch x_i with respective to dictionary \phi_k.
By introducing the auxiliary variable x, we split the objective function into two sub-problems, i.e., the x-subproblem, and the \alpha-subproblem.
The x-subproblem is a equally constrained quadratic optimization problem; and the \alpha-subproblem is a CSR sparse coding problem. Both subproblems can be easily solved.
11/15/2018

25. Alternative optimization algorithm
X-subproblem:
Alternative direction method of multiplier (ADMM)
Since we are using the patch-based sparse model, we introduced an auxiliary variable x into the objective function, and the final objective function is formulated as this formula. The x denotes desired high-resolution whole image; \alpha_i denotes the sparse code of the patch x_i with respective to dictionary \phi_k.
By introducing the auxiliary variable x, we split the objective function into two sub-problems, i.e., the x-subproblem, and the \alpha-subproblem.
The x-subproblem is a equally constrained quadratic optimization problem; and the \alpha-subproblem is a CSR sparse coding problem. Both subproblems can be easily solved.
11/15/2018

26. Alternative optimization algorithm
Solved by a conjugate gradient method
11/15/2018

27. Overall interpolation algorithm
11/15/2018

28. Exp. Results (scaling factor = 2)
Original
NEDI (29.36 dB)
SAI (30.76 dB)
Proposed (31.72 dB)
In this figure, we show some image interpolation results by the proposed methods and some other competing methods. The scaling factor in these two examples are 2.
The SAI is the previous state-of-the-art edge-guided image interpolation method. We can see that the proposed method outperforms the previous edge-guided methods. The PSNR gain over SAI can be up to 1 dB.
Original
NEDI (22.97 dB)
SAI (23.78 dB)
Proposed (24.79 dB)
11/15/2018

29. Exp. Results (scaling factor = 2)
Original
NEDI (27.36 dB)
SAI (29.17 dB)
Proposed (30.30 dB)
In this figure, we show some image interpolation results by the proposed methods and some other competing methods. The scaling factor in these two examples are 2.
The SAI is the current state-of-the-art edge-guided image interpolation method. We can see that the proposed method outperforms the previous edge-guided methods. The PSNR gain over SAI can be up to 1 dB.
Original
NEDI (33.85 dB)
SAI (34.13 dB)
Proposed (34.46 dB)
11/15/2018

30. Exp. Results (scaling factor = 3)
Original
bicubic (23.48 dB)
ScSR (23.84 dB)
Proposed (25.57 dB)
This figure shows interpolation results of scaling factor 3. Previous edge-guided methods are in general for scaling factor 2, and then not compared here.
We can see that the proposed method can still produce much sharper edges than other methods when the scaling factor is increased.
Original
bicubic (30.14 dB)
ScSR (30.00 dB)
Proposed (31.16 dB)
11/15/2018

31. Exp. Results (scaling factor = 3)
Original
bicubic (21.85 dB)
ScSR (21.93 dB)
Proposed (23.33 dB)
This figure shows interpolation results of scaling factor 3. Previous edge-guided methods are in general for scaling factor 2, and then not compared here.
We can see that the proposed method can still produce much sharper edges than other methods when the scaling factor is increased.
Original
bicubic (32.07 dB)
ScSR (32.29 dB)
Proposed (34.80 dB)
11/15/2018

32. Conclusions
A new image upconversion framework combining edge- based interpolator with sparse regularization
A nonlocal AR model is proposed for edge-based interpolation
The nonlocal AR model can increase the stability of the sparse reconstruction
Now we draw the conclusions. In our work, the local learned sparse coding and the structural clustering are unified into a variational framework. A Bayesian interpretation is given for the proposed CSR model, and this interpretation extends the CSR into an iterative re-weighted CSR model. To efficiently solved the results double-header L1-minimization problem, we extend the conventional single variate shrinkage operator, and developed a bi-variate shrinkage operator.
The experimental results show that the proposed CSR method performs much better than BM3D on texture images, and the proposed CSR method achieve comparable or even better denoising performance than BM3D on other natural images
Clustering-based sparsity regularization is adopted to exploit the structural dependencies
11/15/2018

33. References
W. Dong, L. Zhang, et al., “Sparse representation based image interpolation with nonlocal autoregressive modeling,” IEEE Trans. Image Process., vol. 22, Apr
Y. Romano, M. Protter, and M. Elad, “Single image interpolation via adaptive non- local sparsity-based modeling,” IEEE Trans. Image Processing, vol. 23, July 2014.
X. Zhang and X. Wu, “Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation”, IEEE Trans. Image Processing, vol. 17, no. 6, 2008.
X. Li and M. Orchard, “New edge-directed interpolation”, IEEE Trans. Image Processing, vol. 10, no
J. Yang, J. Wright, et al., “Image super-resolution via sparse representation,” IEEE Trans. Image Processing, 2010.
W. Dong, X. Li, et al., “Sparsity-based image denoising via dictionary learning and structural clustering,” IEEE CVPR, 2011.
G. Shi, W. Dong, X. Wu and L. Zhang, “Context-based adaptive image resolution upconversion”, Journal of Electronic imaging, vol. 19, 2010.
11/15/2018

34. Thanks for your attention!
Questions?
11/15/2018