1. Image Enhancement [DVT final project]
Speaker: Yu-Hsiang Wang
Advisor: Prof. Jian-Jung Ding
Digital Image and Signal Processing Lab
Graduate Institute of Communication Engineering
National Taiwan University
DISP Lab, Graduate Institute of Communication Engineering, NTU

2. Outline Target Possible enhancement methods Interpolation Enhancement
Adaptive osculatory rational interpolation
Enhancement
Bilateral Enhancers
Non Local Means
Conclusion
Reference

3. Target
Scale image from SD (Standard-definition) to HD (High-definition).
Given: 720x480, YCbCr, 4:2:2, 8 bits per channel.
Target: 1920x1080, YCbCr, 4:4:4, 10 bits per channel.

4. Possible enhancement methods
Sharpness
Noise reduction
Edge smoothness
Skin-tone enhancement
Texture enhancement
Super resolution (SR)
Multi-Image SR [1]
Single-Image Multi-Patch SR [1]

5. Interpolation: Bilinear
1. Interpolate R1:
2. Interpolate R2:
3. Interpolate P:
[2]

6. Interpolation: Adaptive osculatory rational interpolation
The interpolation kernel function of adaptive osculatory rational interpolation (AORI) is more accurately approximate to the ideal interpolation not only in space domain but also in frequency domain.
Apply 4 points to interpolate 1 point in one direction.
會選用此方法的原因是相較於bilinear bicubic 等方法，他更能在space domain以及frequency domain上近似ideal interpolation

7. Interpolation: Adaptive osculatory rational interpolation
The interpolation function
where r(x) is the interpolated value,
g(xk) are the sample values,
RI(x) is the interpolation kernel.

8. Interpolation: Adaptive osculatory rational interpolation
The interpolation kernel
[M. Hu and J. Q. Tan, ”Adaptive osculatory rational interpolation for image processing,” Journal of Computational and Applied Mathematics, vol. 195, pp , 2006.]
因為interpolation kernel是對稱的，所以Kernel的推導主要分成兩部分，分別是區間0~1 以及 1~2

9. Interpolation: Adaptive osculatory rational interpolation
The original 1920x1080 image:

10. Interpolation: Adaptive osculatory rational interpolation
The interpolated image by AORI:

11. Interpolation: Adaptive osculatory rational interpolation
The original 1920x1080 image:

12. Interpolation: Adaptive osculatory rational interpolation
The interpolated image by AORI:

13. Enhancement: Bilateral Enhancers
Extended from the bilateral filter (BF).
BF:
A nonlinear filter adopts a low-pass Gaussian filter for both the domain filter and the range filter.
Smooth the noise while preserving edges.
包含了對於domain filter與range filter的Gaussian filter

14. Enhancement: Bilateral Enhancers
Bilateral filter (BF)
with the normalization factor
where r is the input image,
h is the output image,
Ω(x) is a subset of the input image r with
x and ξ are the pixels coordinates. (row, column)
[C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proc. ICCV, pp. 839–846, 1998.]
ξ:克希，Ω: omega

15. Enhancement: Bilateral Enhancers
Bilateral filter (BF)
with the normalization factor
The function c operates on the spatial domain designed as
The s operates on the range domain designed as
Function c是作用在空間域上的Gaussian function，position差越大(或是說離mask中心點越遠的pixel)其weighting就越小

16. Enhancement: Bilateral Enhancers
Problem of Bilateral filter:
Only edge-preserving and de-noising.
Do not enhance the sharpness of an image.
據說用Bilateral filter會有模糊化的感覺

17. Enhancement: Bilateral Enhancers
Bilateral enhancers (sharpness + smoothness)
The c function is set as
the same as in BF. (Gaussian function on spatial domain)
[C. Gatta and P. Radeva, “Bilateral enhancers,” IEEE ICIP, pp , 2009.] vs
Bilateral enhancer改善並強化Bilateral filter，除了做bilateral filter會做的smoothness外，還會做sharpness銳利化的動作

18. Enhancement: Bilateral Enhancers
Bilateral enhancers (sharpness + smoothness)
The p composes of two parts (p = ps + pe):
the edge-preserving smoothing (ps)
the selective sharpening (pe) vs
Function p 是由兩個function所組成，分別是edge-preserving smoothing 和 selective sharpening ，這兩個function所得到的值相加起來即是p function的值

19. Enhancement: Bilateral Enhancers
The edge-preserving smoothing (ps)
ηs: adjusts the intensity of the blurring.
σs: controls how strong should be an edge to be preserved from the blurring.
If the intensity difference is small, Gaussian smoothing is performed.
Smooth noise and preserve edge ，η: eta(艾塔) 是在做調整blurring的程度，這個值越大就越模糊，但太小呢，也有可能讓圖變的太銳利，σ(sigma)則是控制edge經過blurring後被保存的強度。所以如果pixel value的差異越小，這個Gaussian Smoothing動作就越強，因為他辨別此處不是edge，所以可以做smooth的動作

20. Enhancement: Bilateral Enhancers
The selective sharpening (pe)
ηe and σe have similar meaning as for the edge-preserving smoothing.
If the intensity difference is small, no enhancement is performed.
Eta和sigma的道理和edge-preserving smoothing很像，都是在控制blurring以及edge-preserving的程度。差別在於，selective sharpening是如果pixel value差異小則越不會去做enhance the noise的動作

21. Enhancement: Bilateral Enhancers
The interpolated image by AORI:

22. Enhancement: Bilateral Enhancers
Enhanced the previous page’s image by BE.

23. Enhancement: Bilateral Enhancers
The original 1920x1080 image:

24. Enhancement: Bilateral Enhancers
The interpolated image by AORI:

25. Enhancement: Bilateral Enhancers
Enhanced the previous page’s image by BE.

26. Framework of System

27. Enhancement: Non Local Means
Purpose: Noise reduction.
Given an interpolated image
The estimated value
W is a search window of fixed size (we choose 5x5 here) centered at pixel i.
The weights depend on the similarity between the pixel i and j.
[A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” IEEE CVPR, Jun ]

28. Enhancement: Non Local Means
The weighted Euclidean distance
Nk denotes a square neighborhood of fixed size (we choose 3x3) and centered at a pixel k.
α is the standard deviation of the Gaussian kernel.
G(N) is the Gaussian kernel of the same size as Nk.
Weighted 歐幾里得距離的算法，其中Nk代表了一個Mask(在這邊我們選用3*3)，這個mask的中心點即是k這個點，所以這個距離公式即是算i j兩個pixel所各自蓋到的3*3mask，兩mask的差，在乘上一個也是3*3的Gaussian kernel，而這個Gaussian的標準差為阿法。

29. Enhancement: Non Local Means
The weights are defined as
Z(i) is the normalizing constant
h denotes a degree of filtering. (It controls the decay of the weights function of the Euclidean distances.) (h = 2.5)
這個weight會受到i j 兩點的相似度以及h這個參數的影響，其中h代表了filter的強度，他可以控制這個weight decay的程度(在這裡h設為定值)，所以此weight最主要是和pixel的similarity有關，如果兩點越相似，則weighting function給的weight就越大

30. Enhancement: Non Local Means
Scheme of NL-means strategy. [6]

31. Enhancement: Non Local Means
Advantage:
NL-means compares the gray level of pixels.
Compare the geometrical configuration in a whole neighborhood.
Disadvantage:
Do not perform sharpness.
Blur some edges.

32. Enhancement: Non Local Means
The interpolated image by AORI:

33. Enhancement: Non Local Means
De-noise the previous image by NL-means.

34. Enhancement: Non Local Means
The interpolated image by AORI:

35. Enhancement: Non Local Means
De-noise the previous image by NL-means.

36. Enhancement: Non Local Means
Enhanced the previous page’s image by BE.

37. Conclusion
Adaptive osculatory rational interpolation is more accurately approximate to the ideal interpolation.
Bilateral enhancers performs well at sharpness and smoothness.
Non local means is mainly used for noise reduction.

38. Reference
[1] D. Glasner, S. Bagon, and M. Irani, “Super-resolution from a single image,” ICCV, pp , Sep
[2]
[3] M. Hu and J. Q. Tan, ”Adaptive osculatory rational interpolation for image processing,” Journal of Computational and Applied Mathematics, vol. 195, pp , 2006.
[4] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proc. ICCV, pp. 839–846,
[5] C. Gatta and P. Radeva, “Bilateral enhancers,” IEEE ICIP, pp , 2009.
[6] A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” IEEE CVPR, Jun