1. HALO-FREE DESIGN FOR RETINEX BASED REAL-TIME VIDEO ENHANCEMENT SYSTEM
~PDCN 2013~
February 11 – 13, 2013
Innsbruck, Austria
Yuecheng Li1, Hong Zhang1, and Mingui Sun2
1Image Processing Center, Beihang University, Beijing, China
2Department of Neurosurgery, University of Pittsburgh, PA, USA

2. Outline 1 Introduction and Problems 2 Strategies 3 Conclusions
2.1 Preventing Traditional Halo Artifacts
2.2 Preventing Motion Caused Halo Artifacts
3 Conclusions

3. Outline 1 Introduction and Problems 2 Strategies 3 Conclusions
2.1 Preventing Traditional Halo Artifacts
2.2 Preventing Motion Caused Halo Artifacts
3 Conclusions

4. Retinex Theory Observed Image Reflectance Illumination Retinex Theory:
1 Introduction and Problems
Retinex Theory
Retinex Theory:
Derived from Color Constancy phenomenon.
A prevailing theory in the field of image enhancement.
At each pixel (x, y) in the image domain:
Observed Image Reflectance Illumination
Mathematically ill-posed problem.

5. Retinex Theory Solution:
1 Introduction and Problems
Retinex Theory
Solution:
Assumption: The reflectance image corresponds to the sharp details in the image while the illumination image is expected to be spatially smooth.
Center/Surround mode (SSR, MSR, and MSRCR) : Gaussian Convolution.

6. Traditional Halo Artifacts
1 Introduction and Problems
Traditional Halo Artifacts
Due to the isotropy of Gaussian filters.
Improvements: Bilateral Filtering, mean-shift, variational approaches, etc.
Original image
Enhanced by using traditional retinex-based method

7. Motion Caused Halo Artifacts
1 Introduction and Problems
Motion Caused Halo Artifacts
Our work: MSR is realized on FPGA
FPGA based video enhancement system

8. Motion Caused Halo Artifacts
1 Introduction and Problems
Motion Caused Halo Artifacts
Other applications:
NASA
Digital Signal Processors (DSP);
real-time Enhanced Vision System (EVS).
Tsutsui et al.:
VLSI architecture;
variation mode.
consensus design:
A pipeline structure, where current frame’s reflectance is derived from previous frame’s illumination, can save frame buffer and system delay.
Motion caused halo artifacts
Due to inter-frame motion, motion causes mismatch between previous frame’s illumination and current frame.
Video: Two kinds of halo artifacts

9. Outline 1 Introduction and Problems 2 Strategies 3 Conclusions
2.1 Preventing Traditional Halo Artifacts
2.2 Preventing Motion Caused Halo Artifacts
3 Conclusions

10. Computation optimization Improved pipeline structure
2 Strategies
Strategies
Methods to prevent halo artifacts:
Traditional halo artifacts:
Motion caused halo artifacts:
Bilateral Filtering
Computation optimization
Improved pipeline structure

11. Outline 1 Introduction and Problems 2 Strategies 3 Conclusions
2.1 Preventing Traditional Halo Artifacts
2.2 Preventing Motion Caused Halo Artifacts
3 Conclusions

12. Preventing Traditional Halo Artifacts
2 Strategies
Preventing Traditional Halo Artifacts
Bilateral filtering definition
Supposing I is a given gray-scale image and p = (px, py) the 2-D pixel position, BF[·] is defined by:
Ω is a neighboring domain;
Wp is a normalization factor;
Gs is a spatial Gaussian decreasing the influence of distant pixels;
Gr is a range Gaussian decreasing the influence of pixels q with an intensity value different from Ip.

13. Preventing Traditional Halo Artifacts
2 Strategies
Preventing Traditional Halo Artifacts
BF Based Retinex
Supposing L is the illumination image and R the reflectance image, BF based Retinex model in logarithmic form can be expressed as:
The computation is complex and heavy even for a single scale BF.
Multi-scale is necessary.

14. Preventing Traditional Halo Artifacts
2 Strategies
Preventing Traditional Halo Artifacts
Optimized separable BF based illumination estimator
Separable BF
Spatial Gaussian filter: three scales
Range Gaussian filter: single scale
Scales combining
Single scale

15. Preventing Traditional Halo Artifacts
2 Strategies
Preventing Traditional Halo Artifacts
Combing the three scales for spatial Gaussian filter
Computation burden decreases to 1/3 without obvious performance sacrifice.
a. Reflectance image by independent three scales
b. Reflectance image by the combined scale

16. Preventing Traditional Halo Artifacts
2 Strategies
Preventing Traditional Halo Artifacts
Performance comparison
a. Source image
b. Result by Traditional
MSR
c. Result by Optimized Separable BF
Based Retinex

17. Outline 1 Introduction and Problems 2 Strategies 3 Conclusions
2.1 Preventing Traditional Halo Artifacts
2.2 Preventing Motion Caused Halo Artifacts
3 Conclusions

18. Preventing Motion Caused Halo Artifacts
2 Strategies
Preventing Motion Caused Halo Artifacts
Motion Caused Halo Artifacts
Inter frame motion will produce spatial mismatch between previous frame’s illumination and current frame
thus produce the motion caused halo artifacts.
Commonly adopted pipeline structure

19. Preventing Motion Caused Halo Artifacts
2 Strategies
Preventing Motion Caused Halo Artifacts a b c d e f

20. Preventing Motion Caused Halo Artifacts
2 Strategies
Preventing Motion Caused Halo Artifacts
Inspired
In the scaled Retinex domain, image contents are generally illumination free.
Hoi-Kok Cheung et al. promote H.264 with Retinex.
motion estimation
motion compensation
prediction error coding
Retinex
Images

21. Preventing Motion Caused Halo Artifacts
2 Strategies
Preventing Motion Caused Halo Artifacts
Proposed pipeline structure
Rn-1 Rn
gray-scale projection
Cross correlation
Motion vectors
Motion compensation on Ln

22. Preventing Motion Caused Halo Artifacts
2 Strategies
Preventing Motion Caused Halo Artifacts
ME Algorithm
Efficient Gray-scale Projection algorithm:
where the Grayn(j) and Grayn(i) correspond to the gray projection value of column j and row i respectively.

23. Preventing Motion Caused Halo Artifacts
2 Strategies
Preventing Motion Caused Halo Artifacts
Effectiveness of gray-scale projection algorithm.
Samples from images set for simulation

24. Preventing Motion Caused Halo Artifacts
2 Strategies
Preventing Motion Caused Halo Artifacts
Both the source images and their reflectance are manually translated with vertical 10 pixels and horizontal 20 pixels respectively.
Gray-scale projection algorithm is performed with the original images and their translated images to estimate the preset inter-frame motion.
Average accuracy of gray-scale projection algorithm
Being conducted on source images
Being conducted On reflectance images
Horizontal
68%
77%
Vertical
69%
94%

25. Preventing Motion Caused Halo Artifacts
2 Strategies
Preventing Motion Caused Halo Artifacts
Problem on boundary
Results under traditional pipeline
structure
Results under the proposed pipeline
structure with ME and MC

26. Outline 1 Introduction and Problems 2 Strategies 3 Conclusions
2.1 Preventing Traditional Halo Artifacts
2.2 Preventing Motion Caused Halo Artifacts
3 Conclusions

27. Conclusions Conclusions: Future work
Based on two ‘concepts: edge preserving and illumination-free reflectance
Traditional halo artifacts is restrained by optimized separable BF
Motion caused halo artifacts is alleviated by a modified pipeline structure with traditional ME algorithm
Future work
Solving the problem on image boundary; Further necessary optimization.

28. Q/A