1. Introduction of Saliency Map
Presenter: Chien-Chi Chen
Advisor: Jian-Jiun Ding

2. Outline Introduction of saliency map Button-up approach
L. Itti’s approach
Frequency-tuned
Multi-scale contrast
Depth of field
Spectral Residual approach
Global contrast based
Top-down approach
Context-aware
Information maximum
Measuring visual saliency by site entropy rate

3. Outline Introduction of saliency map Button-up approach
L. Itti’s approach
Frequency-tuned
Multi-scale contrast
Depth of field
Spectral Residual approach
Global contrast based
Top-down approach
Context-aware
Information maximum
Measuring visual saliency by site entropy rate

4. Introduction of saliency map
Low-level(contrast)
Color
Orientation
Size
Motion
Depth
High-level
People
Context
Important!
Low-level
With face detection
Judd et al, 2009

5. Outline Introduction of saliency map Button-up approach
L. Itti’s approach
Frequency-tuned
Multi-scale contrast
Depth of field
Spectral Residual approach
Global contrast based
Top-down approach
Context-aware
Information maximum
Measuring visual saliency by site entropy rate

6. L. Itti’s approach Architecture: Gaussian Pyramids R,G,B,Y
Gabor pyramids for q = {0º, 45º, 90º, 135º}
我們先將彩色影像分成RGBY ,I,o 再做{0~8}個scale的Gaussian pyramids，
再分別對每個作cs
再做asc在合併

7. L. Itti’s approach Center-surround Difference
Achieve center-surround difference through across-scale difference
Operated denoted by Q: Interpolation to finer scale and point-to-point subtraction
One pyramid for each channel: I(s), R(s), G(s), B(s), Y(s) where s Î [0..8] is the scale

8. L. Itti’s approach Center-surround Difference I(c, s) = | I(c) Q I(s)|
Intensity Feature Maps
I(c, s) = | I(c) Q I(s)|
c Î {2, 3, 4}
s = c + d where d Î {3, 4}
So I(2, 5) = | I(2) Q I(5)| I(2, 6) = | I(2) Q I(6)| I(3, 6) = | I(3) Q I(6)| …
 6 Feature Maps

9. L. Itti’s approach Red-Green and Yellow-Blue
Center-surround Difference
Color Feature Maps
Center-surround Difference
Orientation Feature Maps
Red-Green and Yellow-Blue
Same c and s as with intensity
+B-Y
+R-G
+G-R
+B-Y
+Y-B
+G-R
+R-G
+Y-B
+B-Y
RG(c, s) = | (R(c) - G(c)) Q (G(s) - R(s)) |
BY(c, s) = | (B(c) - Y(c)) Q (Y(s) - B(s)) |

10. L. Itti’s approach Normalization Operator
Promotes maps with few strong peaks
Surpresses maps with many comparable peaks
Normalization of map to range [0…M]
Compute average m of all local maxima
Find the global maximum M
Multiply the map by (M – m)2

11. L. Itti’s approach
Example of Operation:
Inhibition of return

12. Frequency-tuned
Image Average
Goal: color contrast
Gaussian blur

13. Multi-scale contrast Saliency algorithm Image Saliency map
Center-surround histogram
Conditional
Random
Field
Image
Saliency map
[learning to detect a salient object] Liu
Local Color contrast and 分散度
Color spatial-distribution

14. Center-surround histogram
Multi-scale contrast
Multi-scale contrast
Center-surround histogram
Local summation of laplacian pyramid
Distance between histograms of RGB color:
Chi square histogram
5 種情況的長寬比
且找出5種情況的maximun，並且對這五種情況的值做距離的weight

15. Multi-scale contrast Color spatial-distribution GMM Image(RGB)
The variance of Coordinate of pixel x and y
Image(RGB)
GMM
Distance from pixel x to image center
1/18 *coordinate

16. Multi-scale contrast
Energy term:
Saliency object:
Pairwise feature:

17. Multi-scale contrast CRF:
The derivative of the log-likelihood with respect to

18. Depth of field
As the spread of single lens reflex camera, more and more low depth of field(DOF) images are captured.
However, current saliency detection methods work poorly for the low DOF images.

19. Depth of field
Algorithm:

20. Depth of field Classification: Focal Point: In a low DOF image
Rectangle with the highest edge density, and center is initial focal point
DOG
Composition Analysis:
segmentation
Region

21. Spectral Residual Approach
First scaling image to 64x64.
Then we smoothed the saliency map with a gaussian filter g(x) ( = 8).

22. Global contrast-based
Histogram based contrast(Lab):
Quantization of Lab
In lab
將m個顏色與接近的saliency直做平均，做為C的salient
Each channel to have 12 different value 85

23. Global contrast-based
Region based contrast
Segment the Image
[Efficient graph-based image segmentation]

24. Outline Introduction of saliency map Button-up approach
L. Itti’s approach
Frequency-tuned
Center-surround
Depth of field
Spectral Residual approach
Global contrast based
Top-down approach
Context-aware
Information maximum
Measuring visual saliency by site entropy rate

25. Context-Aware
Goal: Convey the image content
Liu et al, 2007

26. Context-Aware
Distance between a pair of patches:
High
salient

27. Context-Aware Saliency Distance between a pair of patches:
K most similar patches at scale r
High for K most similar
Saliency

28. Context-Aware Salient at: Multiple scales  foreground
Few scales  background
Scale 1
Scale 4

29. Context-Aware
Foci =
Include distance map X

30. Outline Introduction of saliency map Button-up approach
L. Itti’s approach
Frequency-tuned
Center-surround
Depth of field
Spectral Residual approach
Global contrast based
Top-down approach
Context-aware
Information maximum
Measuring visual saliency by site entropy rate

31. Measuring visual saliency by site entropy rate1
Ak為每個base在每個點代表的權重，亦為feature weight

32. Measuring visual saliency by site entropy rate2
A fully-connected graph representation is adopted for each

33. Sub-band graph representation
K: base number，i: node number

34. Sub-band graph representation

35. Measuring visual saliency by site entropy rate3
A random walk is adopted on each sub-band graph. And Site entropy rate(SER) is measured the average information from a node to the other

36. The site entropy rate

37. Conclusion Image processing is funny
Unusual in its neighborhood will correspond to high saliency weight
Contrast is the key of saliency
The point is how to set it

38. Reference
[1] R. Achanta, F. Estrada, P. Wils, and S. S¨usstrunk. Salient region detection and segmentation. In ICVS, pages 66–75. Springer, , 412, 414 [2] R. Achanta, S. Hemami, F. Estrada, and S. S¨usstrunk. Frequency-tuned salient region detection. In CVPR, pages 1597–1604, , 410, 412, 413, 414, 415 [3] L. Itti, C. Koch, and E. Niebur. A model of saliency based visual attention for rapid scene analysis. IEEE TPAMI, 20(11):1254–1259, , 410, 412, 414 [4] X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In CVPR, pages 1–8, , 412, 413, 414 [5] S. Goferman, L. Zelnik-Manor, and A. Tal. Context-aware saliency detection. In CVPR, , 412, 413, 414, 415 [6] MM Cheng, GX Zhang, N. J. Mitra, X. Huang, S.M. Hu. Global Contrast based Salient Region Detect. In CVPR, [7] T. Liu, Z. Yuan, J. Sun, J.Wang, N. Zheng, T. X., and S. H.Y. Learning to detect a salient object. IEEE TPAMI, 33(2):353–367, [8] W. Wang, Y. Wang, Q. Huang, W. Gao, Measuring Visaul Saliency by Site Entropy Rate, In CVPR, 2010.