1. Automatic Estimation and Removal of Noise from a Single Image
Accepted for publication by IEEE Transactions on Pattern Analysis and Machine Intelligence
Automatic Estimation and Removal of Noise from a Single Image
Ce Liu* Richard Szeliski† Sing Bing Kang†
C. Lawrence Zitnick† William T. Freeman*
*CSAIL, MIT
†Microsoft Research

2. The criteria of image denoising
Perceptually flat regions should be flat
Image boundaries should be preserved (neither blurred or sharpened)
Texture details should not be lost
Global contrast should be preserved
No artifacts should be generated

3. Denoising is hard… Different source and type of noise
How strong is the noise?
Locally, it is hard to distinguish
Texture vs. noise
Object boundary vs. structural noise
Speed? Quality?

4. Related work Wavelets (coring, GSM) Anisotropic diffusion (PDEs)
FRAME & FOE
Bilateral filtering
Nonlocal methods
Conditional Random Fields

5. Segmentation-based piecewise smooth image models
An image can be reconstructed by
Over-segmentation (your favorite method)
Affine reconstruction for each segment (RGB is an affine function of xy)
Boundary blur estimation
Piecewise smooth image reconstruction

6. Piecewise smooth reconstruction
(a) Original image
(b) Segmentation
(c) Per-segment affine reconstruction
(d) Affine reconstruction + boundary blur

7. Important properties of piecewise smooth image model
Piecewise smooth image model is consistent with sparse image prior
horizontal
vertical
Piecewise smooth reconstruction
Filter responses

8. Important properties of piecewise smooth image model
The color distribution per segment can be well approximated by a line segment
Sorted eigenvalues in RGB space for each segment. The reddish picture indicates that the largest eigenvalue is significantly bigger than the second and third
RGB values projected onto the eigenvector w.r.t. the largest eigenvalue. It is almost identical to the original image.

9. Important properties of piecewise smooth image model
The standard deviation of residual per each segment is the upper bound of the noise level in that segment s I
Brightness
Residual std. dev.
Assume brightness mean I is accurate estimate
Standard deviation s is an over-estimate: (may contain signal)
The lower envelope is the upper bound of noise level function (NLF)

10. Insert the CVPR slides on noise estimation

11. Segmentation-based noise reduction
The noise level (std dev) for each segment, each RGB channel (from the inferred NLF)
0th-order model
Assume the pixel is independent of the neighbors (of course this is not true)
Bayesian MAP inference for each pixel (trivial)

12. The results of 0th order model

13. 1st order Gaussian CRF Of course the neighboring pixels are dependent
A Gaussian conditional random field (GCRF) is formed to incorporate
Distance to noise
Distance to signal
Image-dependent regularization
Inference by conjugate gradient method

14. Experimental results
17 images are randomly selected from Berkeley image segmentation database
Compare to bilateral filtering, curvature preserving PDE and wavelet GSM.

15. We win the PSNR battle

16. Visual inspection
10% AWGN
PDE
Wavelet GSM
Ours
Original

17. Close-up view
10% AWGN
PDE
Wavelet GSM
Ours
Original

18. Visual inspection
10% AWGN
PDE
Wavelet GSM
Ours
Original

19. Close-up view
10% AWGN
PDE
Wavelet GSM
Ours
Original

20. Apply to real CCD noise
Estimated NLF

21. Close-up view
Since wavelet GSM only takes one constant noise level, it either over-smoothes (as s =15%) or under-smoothes (s =10%). We estimate signal-dependent noise level (NLF), and our approach handles both high and low noise level well.
Estimated NLF

22. A very challenging example
Real digital camera noise
(low light, high ISO)
Denoised by wavelet GSM
Denoised by our model
Estimated NLF

23. Conclusion
Noise estimation and removal using one framework: piecewise smooth image model
The chrominance component of the noise can be significantly reduced by simple projection
Computer vision algorithms can be automated by estimating the peripheral parameters such as noise level, blur level, lighting, …