Sparsity Based Poisson Denoising and Inpainting Raja Giryes, Tel Aviv University Joint work with Michael Elad, Technion

Agenda Problem Definition – Poisson Denoising Existing Poisson Denoising Methods Poisson Greedy Algorithm Experimental results Poisson Inpainting

Denoising Problem Original unknown image is a noisy measurement of x. The goal is to recover x from .

Gaussian Denoising Problem where is a zero-mean white Gaussian noise with variance , i.e., each element .

Gaussian Noisy Measurements Another perspective for the Gaussian denoising problem: Look at the measurements as Gaussian distributed with mean equal to the original signal The variance determines the noise power.

Poisson Noisy Measurements The measurements are Poisson Distributed Poisson noise is not an additive noise, unlike the Gaussian case. The noise power is measured by the peak value:

Poisson Denoising Problem Noisy image distribution: is an integer. large large . small small .

Poisson Denoising Problem

Poisson Denoising Problem

Poisson Denoising Applications Tomography – CT, PET and SPECT Astrophysics Fluorescence Microscopy Night Vision Spectral Imaging etc.

Tomography Slices of skeletal SPECT image [Takalo , Hytti and Ihalainen 2011]

Fluorescence Microscopy C. elegans embryo labeled with three fluorescent dyes [Luisier, Vonesch, Blu and Unser 2010]

Astrophysics XMM/Newton image of the Kepler SN1604 supernova [Starck, Donoho and Candès 2003]

Agenda Problem Definition – Poisson Denoising Existing Poisson Denoising Methods Poisson Greedy Algorithm Experimental results Poisson Inpainting

Denoising Methods Many denoising methods exists. However, most of them assume a Gaussian model for the noise. We have two options: Use a transformation that converts the noise to be Gaussian. Work directly with the Poisson model.

The Anscombe Transform The Anscombe transform converts Poisson distributed noise into an approximately Gaussian distributed data with variance 1 using the following formula elementwise [Anscombe, 1948]. Valid only when peak>4

Poisson Log-likelihood We will work directly with the Poisson data. By maximizing the log-likelihood of the Poisson distribution we get the following minimization problem Reminder: In the Gaussian case we had A prior is needed

Sparsity Prior for Poisson Denoising (1) Regular sparsity prior leads to which is a non-negative optimization problem Instead we use that yields the following D is a given dictionary. counts the non-zero elements

Sparsity Prior for Poisson Denoising (2) The minimization problem is likely to be NP-hard. Approximations are needed.

l1 Relaxation One option is to use l1 relaxation is a relaxation parameter. This problem can be solved using the SPIRAL algorithm [Harmany et al., 2012].

Non-local PCA (NLPCA) GMM (Gaussian Mixture Model) based method. Cluster the noisy patches into small number of large groups. For each cluster train a PCA subspace Non-local Sparse PCA (NLSPCA) Uses l1 regularization with NLPCA. Binning Aggregate nearby pixels to improve SNR. Denoise down-sampled image. Interpolate recovered image to return to initial size. [Salmon, Harmany, Deledalle, Willett 2013]

Agenda Problem Definition – Poisson Denoising Existing Poisson Denoising Methods Poisson Greedy Algorithm Experimental results Poisson Inpainting Novel Part

Exponential Sparsity Prior Zero entries Non-zero entries Stress that the classical approaches try to recover z. We focus on x. [Salmon et al. 2012, Giryes and Elad 2012]

Poisson Greedy Algorithm - Summary Divide the image into set of overlapping patches. Cluster (using Gaussian filtering) the noisy patches into large number of small groups. Each group of patches is assumed to have the same non-zero locations (support) in their representations . A global dictionary is used for all groups of patches. Having the reconstructed patches we form the final image by averaging. Put in a table comparing the NLPCA

Dictionary Learning Joint dictionary D and representation with a fixed support learning [Smith, Elad 2013]. after we have the representation of all the patches and their supports we minimize: Global initial dictionary for all images Trained using the following image

Our Algorithm vs. NLPCA Large number of clusters. Small cluster size. Poisson Greedy Algorithm NLPCA Large number of clusters. Small cluster size. Clustering using Gaussian filtering. Global dictionary for all patches. Dictionary learning based approach. Small number of clusters . Large cluster size. Clustering using k-means. Local dictionary for each cluster. GMM based approach.

Algorithm Summary ..… ..… ..… Extracting overlapping patches Applying Poisson greedy algorithm for each group Gaussian filtering Averaging patches Dictionary learning Patch grouping ..…

Poisson Greedy Algorithm-Sparse Coding Input: Group of noisy patches Initialization: While t<k t=t+1 Find new support element and representations: Update the support Form patches estimate:

Boot-strapped Stopping Criterion Ideally we want to select different number of non-zeros for each patch. We want to add elements to the support till the error with respect to the original patch (in the original image) stops decreasing. We do not have access to the original image. Use the patches of the estimated image from the previous iteration.

Agenda Problem Definition – Poisson Denoising Existing Poisson Denoising Methods Poisson Greedy Algorithm Denoising Results Poisson Inpainting Novel Part

Experiment- Parameter Setting Patches of size 20 by 20. Patches clustered to groups of size 50. Initial cardinality of the patches is k=2. 5 dictionary learning iterations. Repeat the process one time with re-clustering based on the recovered image.

Noisy Image Max y value = 7 Peak = 1

Poisson Greedy Algorithm Dictionary learned atoms: Method 22.59db Our 20.37db NLSPCA 19.41db BM3Dbin Peak = 1 [Giryes and Elad 2013].

Poisson Greedy Algorithm Method 22.59db Our 20.37db NLSPCA 19.41db BM3Dbin Peak = 1 [Salmon et al. 2013].

Poisson Greedy Algorithm Method 22.59db Our 20.37db NLSPCA 19.41db BM3Dbin Peak = 1 [Makitalo and Foi 2011]

Original Image

Noisy Image Max y value = 3 Peak = 0.2

Poisson Greedy Algorithm Method 24.16db Our 22.98db NLSPCA 23.16db BM3Dbin Peak = 0.2 [Giryes and Elad 2013].

Poisson Greedy Algorithm Method 24.16db Our 22.98db NLSPCA 23.16db BM3Dbin Peak = 0.2 [Salmon et al. 2013].

Poisson Greedy Algorithm Method 24.16db Our 22.98db NLSPCA 23.16db BM3Dbin Peak = 0.2 [Makitalo and Foi 2011]

Original Image

Noisy Image Max y value = 8 Peak = 2

Poisson Greedy Algorithm Method 24.76db Our 23.23db NLSPCA 24.23db BM3Dbin Peak = 2 [Giryes and Elad 2013].

Poisson Greedy Algorithm Method 24.76db Our 23.23db NLSPCA 24.23db BM3Dbin Peak = 2 [Salmon et al. 2013].

Poisson Greedy Algorithm Method 24.76db Our 23.23db NLSPCA 24.23db BM3Dbin Peak = 2 [Makitalo and Foi 2011]

Original Image

Recovery Results 8 Test images. 6 peak levels (0.1,0.2,0.5,1,2,4). Best average recovery error for 5 out of 6 peak values. Second best for peak =1 (difference of 0.02db). 0.288db better on average than second best

Agenda Problem Definition – Poisson Denoising Existing Poisson Denoising Methods Poisson Greedy Algorithm Denoising Results Poisson Inpainting Novel Part

The Poisson Inpainting Problem Some of the pixels in are occluded. The mask defines missing and given pixels in the measured image + = Noisy Image

Poisson Inpainting Objective The Poisson Inpainting minimization problem We approximate this problem using a greedy algorithm as before.

Noise Estimation for Inpainting Having a recovery , we replace each unknown pixel in with a noisy pixel generated from . We get a noisy image for which we can apply the regular dictionary update steps. +

Inpainting Results 23.86dB Peak = 1, 20% Missing Pixels

Inpainting Results 22.83dB Peak = 1, 40% Missing Pixels

Inpainting Results 21.02dB Peak = 1, 60% Missing Pixels

Inpainting Results Average over four different test images

Inpainting Results 24.34dB Peak = 1

Inpainting Results 23.58dB Peak = 2

Inpainting Results 22.72dB Peak = 2

Inpainting Results 19.76dB Peak = 1

Conclusion Poisson based denoising Sparse representation for Poisson noise Greedy Poisson algorithm State-of-the-art denoising results Poisson inpainting algorithm

Questions?