Image Enhancement [DVT final project] Speaker: Yu-Hsiang Wang Advisor: Prof. Jian-Jung Ding Digital Image and Signal Processing Lab Graduate Institute of Communication Engineering National Taiwan University DISP Lab, Graduate Institute of Communication Engineering, NTU

Outline Target Possible enhancement methods Interpolation Enhancement Adaptive osculatory rational interpolation Enhancement Bilateral Enhancers Non Local Means Conclusion Reference

Target Scale image from SD (Standard-definition) to HD (High-definition). Given: 720x480, YCbCr, 4:2:2, 8 bits per channel. Target: 1920x1080, YCbCr, 4:4:4, 10 bits per channel.

Possible enhancement methods Sharpness Noise reduction Edge smoothness Skin-tone enhancement Texture enhancement Super resolution (SR) Multi-Image SR [1] Single-Image Multi-Patch SR [1]

Interpolation: Bilinear 1. Interpolate R1: 2. Interpolate R2: 3. Interpolate P: [2]

Interpolation: Adaptive osculatory rational interpolation The interpolation kernel function of adaptive osculatory rational interpolation (AORI) is more accurately approximate to the ideal interpolation not only in space domain but also in frequency domain. Apply 4 points to interpolate 1 point in one direction. 會選用此方法的原因是相較於bilinear bicubic 等方法，他更能在space domain以及frequency domain上近似ideal interpolation

Interpolation: Adaptive osculatory rational interpolation The interpolation function where r(x) is the interpolated value, g(xk) are the sample values, RI(x) is the interpolation kernel.

Interpolation: Adaptive osculatory rational interpolation The interpolation kernel [M. Hu and J. Q. Tan, ”Adaptive osculatory rational interpolation for image processing,” Journal of Computational and Applied Mathematics, vol. 195, pp. 46-53, 2006.] 因為interpolation kernel是對稱的，所以Kernel的推導主要分成兩部分，分別是區間0~1 以及 1~2

Interpolation: Adaptive osculatory rational interpolation The original 1920x1080 image:

Interpolation: Adaptive osculatory rational interpolation The interpolated image by AORI:

Interpolation: Adaptive osculatory rational interpolation The original 1920x1080 image:

Interpolation: Adaptive osculatory rational interpolation The interpolated image by AORI:

Enhancement: Bilateral Enhancers Extended from the bilateral filter (BF). BF: A nonlinear filter adopts a low-pass Gaussian filter for both the domain filter and the range filter. Smooth the noise while preserving edges. 包含了對於domain filter與range filter的Gaussian filter

Enhancement: Bilateral Enhancers Bilateral filter (BF) with the normalization factor where r is the input image, h is the output image, Ω(x) is a subset of the input image r with x and ξ are the pixels coordinates. (row, column) [C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proc. ICCV, pp. 839–846, 1998.] ξ:克希，Ω: omega

Enhancement: Bilateral Enhancers Bilateral filter (BF) with the normalization factor The function c operates on the spatial domain designed as The s operates on the range domain designed as Function c是作用在空間域上的Gaussian function，position差越大(或是說離mask中心點越遠的pixel)其weighting就越小

Enhancement: Bilateral Enhancers Problem of Bilateral filter: Only edge-preserving and de-noising. Do not enhance the sharpness of an image. 據說用Bilateral filter會有模糊化的感覺

Enhancement: Bilateral Enhancers Bilateral enhancers (sharpness + smoothness) The c function is set as the same as in BF. (Gaussian function on spatial domain) [C. Gatta and P. Radeva, “Bilateral enhancers,” IEEE ICIP, pp. 3161-3164, 2009.] vs Bilateral enhancer改善並強化Bilateral filter，除了做bilateral filter會做的smoothness外，還會做sharpness銳利化的動作

Enhancement: Bilateral Enhancers Bilateral enhancers (sharpness + smoothness) The p composes of two parts (p = ps + pe): the edge-preserving smoothing (ps) the selective sharpening (pe) vs Function p 是由兩個function所組成，分別是edge-preserving smoothing 和 selective sharpening ，這兩個function所得到的值相加起來即是p function的值

Enhancement: Bilateral Enhancers The edge-preserving smoothing (ps) ηs: adjusts the intensity of the blurring. σs: controls how strong should be an edge to be preserved from the blurring. If the intensity difference is small, Gaussian smoothing is performed. Smooth noise and preserve edge ，η: eta(艾塔) 是在做調整blurring的程度，這個值越大就越模糊，但太小呢，也有可能讓圖變的太銳利，σ(sigma)則是控制edge經過blurring後被保存的強度。所以如果pixel value的差異越小，這個Gaussian Smoothing動作就越強，因為他辨別此處不是edge，所以可以做smooth的動作

Enhancement: Bilateral Enhancers The selective sharpening (pe) ηe and σe have similar meaning as for the edge-preserving smoothing. If the intensity difference is small, no enhancement is performed. Eta和sigma的道理和edge-preserving smoothing很像，都是在控制blurring以及edge-preserving的程度。差別在於，selective sharpening是如果pixel value差異小則越不會去做enhance the noise的動作

Enhancement: Bilateral Enhancers The interpolated image by AORI:

Enhancement: Bilateral Enhancers Enhanced the previous page’s image by BE.

Enhancement: Bilateral Enhancers The original 1920x1080 image:

Enhancement: Bilateral Enhancers The interpolated image by AORI:

Enhancement: Bilateral Enhancers Enhanced the previous page’s image by BE.

Framework of System

Enhancement: Non Local Means Purpose: Noise reduction. Given an interpolated image The estimated value W is a search window of fixed size (we choose 5x5 here) centered at pixel i. The weights depend on the similarity between the pixel i and j. [A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” IEEE CVPR, Jun. 2005.]

Enhancement: Non Local Means The weighted Euclidean distance Nk denotes a square neighborhood of fixed size (we choose 3x3) and centered at a pixel k. α is the standard deviation of the Gaussian kernel. G(N) is the Gaussian kernel of the same size as Nk. Weighted 歐幾里得距離的算法，其中Nk代表了一個Mask(在這邊我們選用3*3)，這個mask的中心點即是k這個點，所以這個距離公式即是算i j兩個pixel所各自蓋到的3*3mask，兩mask的差，在乘上一個也是3*3的Gaussian kernel，而這個Gaussian的標準差為阿法。

Enhancement: Non Local Means The weights are defined as Z(i) is the normalizing constant h denotes a degree of filtering. (It controls the decay of the weights function of the Euclidean distances.) (h = 2.5) 這個weight會受到i j 兩點的相似度以及h這個參數的影響，其中h代表了filter的強度，他可以控制這個weight decay的程度(在這裡h設為定值)，所以此weight最主要是和pixel的similarity有關，如果兩點越相似，則weighting function給的weight就越大

Enhancement: Non Local Means Scheme of NL-means strategy. [6]

Enhancement: Non Local Means Advantage: NL-means compares the gray level of pixels. Compare the geometrical configuration in a whole neighborhood. Disadvantage: Do not perform sharpness. Blur some edges.

Enhancement: Non Local Means The interpolated image by AORI:

Enhancement: Non Local Means De-noise the previous image by NL-means.

Enhancement: Non Local Means The interpolated image by AORI:

Enhancement: Non Local Means De-noise the previous image by NL-means.

Enhancement: Non Local Means Enhanced the previous page’s image by BE.

Conclusion Adaptive osculatory rational interpolation is more accurately approximate to the ideal interpolation. Bilateral enhancers performs well at sharpness and smoothness. Non local means is mainly used for noise reduction.

Reference [1] D. Glasner, S. Bagon, and M. Irani, “Super-resolution from a single image,” ICCV, pp. 349-356, Sep. 2009. [2]http://en.wikipedia.org/wiki/Bilinear_interpolation [3] M. Hu and J. Q. Tan, ”Adaptive osculatory rational interpolation for image processing,” Journal of Computational and Applied Mathematics, vol. 195, pp. 46-53, 2006. [4] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proc. ICCV, pp. 839–846, 1998. [5] C. Gatta and P. Radeva, “Bilateral enhancers,” IEEE ICIP, pp. 3161-3164, 2009. [6] A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” IEEE CVPR, Jun. 2005.