A LOW-COMPLEXITY, MOTION-ROBUST, SPATIO-TEMPORALLY ADAPTIVE VIDEO DE-NOISER WITH IN-LOOP NOISE ESTIMATION Chirag Jain, Sriram Sethuraman Ittiam Systems (Pvt.) Ltd., Bangalore, India Image Processing, ICIP

Outline Introduction Proposed Algorithm – LLMMSE based spatial filtering – Temporal filtering – Noise estimation Experimental Results Complexity Comparison Conclusions and Future Work

Introduction Many of the successful techniques determine the weights needed to combine the neighborhood of the pixel being de-noised. – e.g., bi-lateral[2], anisotropic diffusions method[3], spatial non local means (NL-means) de-noiser [4]. – Drawback: high computational complexity [2] C. Tomasi, and R. Manduchi.,” Bilateral filtering for gray and color images,” Proceedings of the Sixth International Conference on Computer Vision, pp , [3] P. Bourdon, B. Augereau, C. Olivier, and C. Chatellier, “Noise removal on color image sequences using coupled anisotropic diffusions and noise-robust motion detection,” EUSIPCO'04 -Signal Processing XII, Wien (Austria), pp – Sep [4] A. Buades, B. Coll, and J. M. Morel, “Denoising image sequences does not require motion estimation,” IEEE Int. Conf. on Advanced Video and Signal based Surveillance, 2005.

Introduction We propose a de-noising algorithm using the following principles: – Block-based noise estimation and de-noising; – Use of block motion vectors by a video encoder; – Locally adaptive Linear Minimum Mean Square Error (LLMMSE) methods. – Reliance on Infinite Impulse Response (IIR) temporal filtering. – Spatially adaptive weight selection for the spatial- temporal IIR (ST-IIR) filter.

Proposed Algorithm Case 1: stationary blocks Case 3: well Motion Compensated blocks Case 2: under-compensated blocks

LLMMSE based spatial filtering An 3x3 Gaussian filter is applied to obtain a output L (low pass filtered ). The output H = X - L (X is the noisy image). Spatially filtered output X’(i,j) is computed as: – σ f 2 : variance of noisy signal(X) for every 8x8 block. – σ n 2 : the estimated noise variance of the sequence.

Temporal filtering Let X(n) be the current frame, Y(n-1) be the previous denoised frame. σ fr_diff 2 : the estimated variance of the difference data in stationary areas between X(n-1) and Y(n-2) σ blk_diff 2 : the variance of the difference between cur_blk and its co- located block in Y(n-1) (prev_blk)

Temporal filtering Is block stationary? If (2) is satisfied and at least one of (3) or (4) are true: – Case 1: stationary blocks (CL) Is MC successful? If (5) is satisfied: – Case 3: well Motion Compensated blocks(MC) All remaining blocks belong to case 2: – Case 2: under-compensated blocks

Temporal filtering Case 2 : – Blocks only spatially filtered Case 1&3 : – The ST-IIR filtering is performed at a pixel level by using the CL or MC pixel in Y(n-1): – pixel_diff : the difference between the current pixel and its CL or MC pixel in Y(n-1). – For pixels with pixel_diff > σ fr_diff (n-1), α = e^(-1/k 4 ) σ fr_diff (n-1) : the deviation of stationary (n-1)

Noise estimation If the current frame X(n) has noise variance σ n 2 and Y(n-1) has noise variance σ p 2, the variance σ fr_diff 2 (n) (stationary)will be: Ideally, σ p 2 should be zero, but we should introduce a confidence c for robustness: – c: the average value of α used for ST-IIR.

Noise estimation If the Y(n-1) is not filtered, c = 1, σ n 2 = σ fr_diff 2. The convergence of standard deviation of the noise against frame number for two sequences:

Experimental Results Additive white Gaussian noise of a specified variance was added to clean video sequences. Encoded using a H.264 baseline profile encoder. NL-means[4]: window size of 21x21 and a similarity square neighborhood of 7x7 with (a = 1 and h = n) [4].

Experimental Results

Complexity Comparison The proposed algorithm perform the decisions at a block level and the filtering at a pixel level. The proposed algorithm requires only 38 operations(+,-,*,look-ups) per pixel NL-means has a complexity of 441x49 weighted Euclidean distance calculations and exponential function lookups per pixel.

Conclusions and Future Work The proposed algorithm comes close to the performance of more complex algorithms such as the NL-means method. It is ideally suited for real time embedded implementations. The spatial filter’s output can be used by the encoder to avoid spurious motion vectors.