Image Denoising using Locally Learned Dictionaries Priyam Chatterjee Peyman Milanfar Dept. of Electrical Engineering University of California, Santa Cruz Computational Imaging VII – 20 Jan, 2009

20 Jan, 2009Image Denoising using Locally Learned Dictionaries2 Overview Data Model Kernel Regression for Denoising Denoising with Locally Learned Dictionaries (K-LLD) Results Conclusions

20 Jan, 2009Image Denoising using Locally Learned Dictionaries3 Data Model Pointwise data model Patchwise model Locally smooth function to be estimated Zero-mean I.I.D. noise Observation denoted as

20 Jan, 2009Image Denoising using Locally Learned Dictionaries4 Steering Kernel Regression (SKR) Optimization problem Solution: Nonlinear filters Polynomial basis Data- dependent weights

20 Jan, 2009Image Denoising using Locally Learned Dictionaries5 SKR Weights Weights based on pixel “self-similarity” in a local patch Covariance matrix takes into account: orientation and strength of edges Gradient Covariance

20 Jan, 2009Image Denoising using Locally Learned Dictionaries6 Steering Kernel Decompose the matrix into three components: Scaling parameter, rotation matrix, and elongation matrix. ElongateRotateScale

20 Jan, 2009Image Denoising using Locally Learned Dictionaries7 SKR Weights Note how the weights adapt to the underlying image structure Noisy Noise-free H. Takeda, S. Farsiu, and P. Milanfar, “Kernel Regression for Image Processing and Reconstruction”, IEEE Trans. on Image Processing, vol. 16, no. 2, pp , February 2007.

20 Jan, 2009Image Denoising using Locally Learned Dictionaries8 Now we extend it ….. is fixed order, everywhere -- not depending on underlying image structure Lower orders fit flat regions, higher order for texture and fine details Global dictionary does not adapt to local image characteristics Dictionary atoms should capture underlying local image structure

20 Jan, 2009Image Denoising using Locally Learned Dictionaries9 Denoising with Locally Learned Dictionaries (K-LLD) Identify dictionary which best captures underlying geometric structure Similar structures will have similar dictionary, similar weights Cluster image based on geometric similarity (K-Means on the SKR weights) Learn dictionary and order of regression for each cluster

20 Jan, 2009Image Denoising using Locally Learned Dictionaries10 K-LLD: Algorithm Outline Calculate weights Learn dictionaries Clustering Iterate Noisy Image Kernel Regression Denoised Image

20 Jan, 2009Image Denoising using Locally Learned Dictionaries11 Class 1 Class K Clustering Stage K-LLD : Algorithm Outline Dictionary Selection Stage Noisy Img Calculate Steering Weights Calculate Steering Weights Coefficient Calculation Stage Coefficient Calculation Stage Denoised Img

20 Jan, 2009Image Denoising using Locally Learned Dictionaries12 Segment Image Segment Image Clustering Stage K-Means Class 1 Class K K-LLD : Algorithm Outline Dictionary Selection Stage Noisy Img Calculate Steering Weights Calculate Steering Weights Coefficient Calculation Stage Coefficient Calculation Stage Denoised Img

20 Jan, 2009Image Denoising using Locally Learned Dictionaries13 Clustering Stage Objective : Cluster image based on geometric similarity of underlying data Feature Selection What features capture data geometry ? Distance Metric What metric captures distance between features ? Clustering Algorithm What algorithm segments the image best ?

20 Jan, 2009Image Denoising using Locally Learned Dictionaries14 K-Means for Clustering Features : normalized steering wts Distance Metric : L 2 Initialization : Randomly initialize cluster centers Run K-Means multiple times and select result that minimizes within-cluster distance

20 Jan, 2009Image Denoising using Locally Learned Dictionaries15 Noise-free Noisy Clustering the noise-free image Clustering the noisy image

20 Jan, 2009Image Denoising using Locally Learned Dictionaries16 Segment Image Segment Image Clustering Stage K-Means Class 1 Class K K-LLD : Algorithm Outline Dictionary Selection Stage Noisy Img Calculate Steering Weights Calculate Steering Weights Coefficient Calculation Stage Coefficient Calculation Stage Denoised Img Dictionary Selection Stage PCA Form Dictionary Form Dictionary

20 Jan, 2009Image Denoising using Locally Learned Dictionaries17 Dictionary Selection Represent each patch in the k th cluster Variable Proj. Solved by PCA Mean patch of k-th cluster Enforce orthonormality

20 Jan, 2009Image Denoising using Locally Learned Dictionaries18 Dictionary Selection PCA in each cluster to form a dictionary Describe data without fitting noise Number of atoms based on cluster geometry Clusters with flat regions need fewer atoms, finer details need more constant Singular values patch size No. of atoms

20 Jan, 2009Image Denoising using Locally Learned Dictionaries19 Example : House Image Dictionary atoms for AWGN of std. dev. 15 Cluster Atom 1 Atom 2 Atom 3

20 Jan, 2009Image Denoising using Locally Learned Dictionaries20 Example : Noise-free case Cluster Atom 1 Atom 2 Atom 3 Atom 4 Few of the atoms in the dictionaries for different clusters

20 Jan, 2009Image Denoising using Locally Learned Dictionaries21 Example : Noise-free clustering Cluster Atom 1 Atom 2 Atom 3 Atom 4 Few of the atoms in the dictionaries for different clusters Brick facade

20 Jan, 2009Image Denoising using Locally Learned Dictionaries22 Data Representation Data represented as For point-wise estimator, local weights should be considered Cluster boundaries not necessarily described well by dictionary Protects against errors in clustering Unknown, to be estimated

20 Jan, 2009Image Denoising using Locally Learned Dictionaries23 Algorithm Outline Segment Image Segment Image Clustering Stage Class 1 Class K K-Means Noisy Img Calculate Steering Weights Calculate Steering Weights Coefficient Calculation Stage Coefficient Calculation Stage Denoised Img Dictionary Selection Stage PCA Form Dictionary Form Dictionary Kernel Regression Denoised Img

20 Jan, 2009Image Denoising using Locally Learned Dictionaries24 Kernel Regression Weighted least squares solution Final estimate Coefficient Calculation center pixel of patch.

20 Jan, 2009Image Denoising using Locally Learned Dictionaries25 Kernel Regression Denoised Img Algorithm Outline Segment Image Segment Image Clustering Stage Class 1 Class K K-Means Noisy Img Calculate Steering Weights Calculate Steering Weights Dictionary Selection Stage PCA Form Dictionary Form Dictionary

20 Jan, 2009Image Denoising using Locally Learned Dictionaries26 Iteration Re-learn weights (features) from denoised image Perform clustering of updated image using new features Learn dictionary from updated image Kernel regression on input noisy image Preserves edges and finer structures

20 Jan, 2009Image Denoising using Locally Learned Dictionaries27 Algorithm Outline Segment Image Segment Image Clustering Stage Class 1 Class K K-Means Noisy Img Calculate Steering Weights Calculate Steering Weights Kernel Regression Dictionary Selection Stage PCA Form Dictionary Form Dictionary

20 Jan, 2009Image Denoising using Locally Learned Dictionaries28 K-LLD: Algorithm Outline Calculate weights Learn dictionaries Clustering Iterate Noisy Image Kernel Regression Denoised Image Original Noisy Image

20 Jan, 2009Image Denoising using Locally Learned Dictionaries29 Performance Gain Performance gain by iterating

Results – AWG noise (std dev 25) K-LLD, MSE SSIM BM3D, MSE SSIM Original Parrot Image K-SVD, MSE SSIM SKR, MSE SSIM Noisy Image

20 Jan, 2009Image Denoising using Locally Learned Dictionaries31 More Results MSE SSIM

Results – Real noise & color ISKRK-LLDBM3D

Color Results ISKR, Order 2 BM3DOriginal Image K-LLD

20 Jan, 2009Image Denoising using Locally Learned Dictionaries34 Conclusions Data adaptive method having multiple degrees of freedom Denoising through a local data representation of images Can be extended to have adaptive patch size based on geometry of data in each cluster

20 Jan, 2009Image Denoising using Locally Learned Dictionaries35 Thank you P. Chatterjee and P. Milanfar, “Clustering-based Denoising with Locally Learned Dictionaries”, Accepted for publication in IEEE Trans. Image Processing Available at:

20 Jan, 2009Image Denoising using Locally Learned Dictionaries36 Iterative Scheme Why iterate ? Weights true to underlying structure in presence of lesser noise Better weights means better clustering Dictionary captures underlying data better when learned on less noisy image