Introduction of Saliency Map Presenter: Chien-Chi Chen Advisor: Jian-Jiun Ding

Outline Introduction of saliency map Button-up approach L. Itti’s approach Frequency-tuned Multi-scale contrast Depth of field Spectral Residual approach Global contrast based Top-down approach Context-aware Information maximum Measuring visual saliency by site entropy rate

Outline Introduction of saliency map Button-up approach L. Itti’s approach Frequency-tuned Multi-scale contrast Depth of field Spectral Residual approach Global contrast based Top-down approach Context-aware Information maximum Measuring visual saliency by site entropy rate

Introduction of saliency map Low-level(contrast) Color Orientation Size Motion Depth High-level People Context Important! Low-level With face detection Judd et al, 2009

Outline Introduction of saliency map Button-up approach L. Itti’s approach Frequency-tuned Multi-scale contrast Depth of field Spectral Residual approach Global contrast based Top-down approach Context-aware Information maximum Measuring visual saliency by site entropy rate

L. Itti’s approach Architecture: Gaussian Pyramids R,G,B,Y Gabor pyramids for q = {0º, 45º, 90º, 135º} 我們先將彩色影像分成RGBY ,I,o 再做{0~8}個scale的Gaussian pyramids， 再分別對每個作cs 再做asc在合併

L. Itti’s approach Center-surround Difference Achieve center-surround difference through across-scale difference Operated denoted by Q: Interpolation to finer scale and point-to-point subtraction One pyramid for each channel: I(s), R(s), G(s), B(s), Y(s) where s Î [0..8] is the scale

L. Itti’s approach Center-surround Difference I(c, s) = | I(c) Q I(s)| Intensity Feature Maps I(c, s) = | I(c) Q I(s)| c Î {2, 3, 4} s = c + d where d Î {3, 4} So I(2, 5) = | I(2) Q I(5)| I(2, 6) = | I(2) Q I(6)| I(3, 6) = | I(3) Q I(6)| …  6 Feature Maps

L. Itti’s approach Red-Green and Yellow-Blue Center-surround Difference Color Feature Maps Center-surround Difference Orientation Feature Maps Red-Green and Yellow-Blue Same c and s as with intensity +B-Y +R-G +G-R +B-Y +Y-B +G-R +R-G +Y-B +B-Y RG(c, s) = | (R(c) - G(c)) Q (G(s) - R(s)) | BY(c, s) = | (B(c) - Y(c)) Q (Y(s) - B(s)) |

L. Itti’s approach Normalization Operator Promotes maps with few strong peaks Surpresses maps with many comparable peaks Normalization of map to range [0…M] Compute average m of all local maxima Find the global maximum M Multiply the map by (M – m)2

L. Itti’s approach Example of Operation: Inhibition of return

Frequency-tuned Image Average Goal: color contrast Gaussian blur

Multi-scale contrast Saliency algorithm Image Saliency map Center-surround histogram Conditional Random Field Image Saliency map [learning to detect a salient object] Liu Local Color contrast and 分散度 Color spatial-distribution

Center-surround histogram Multi-scale contrast Multi-scale contrast Center-surround histogram Local summation of laplacian pyramid Distance between histograms of RGB color: Chi square histogram 5 種情況的長寬比 且找出5種情況的maximun，並且對這五種情況的值做距離的weight

Multi-scale contrast Color spatial-distribution GMM Image(RGB) The variance of Coordinate of pixel x and y Image(RGB) GMM Distance from pixel x to image center 1/18 *coordinate

Multi-scale contrast Energy term: Saliency object: Pairwise feature:

Multi-scale contrast CRF: The derivative of the log-likelihood with respect to

Depth of field As the spread of single lens reflex camera, more and more low depth of field(DOF) images are captured. However, current saliency detection methods work poorly for the low DOF images.

Depth of field Algorithm:

Depth of field Classification: Focal Point: In a low DOF image Rectangle with the highest edge density, and center is initial focal point DOG Composition Analysis: segmentation Region

Spectral Residual Approach First scaling image to 64x64. Then we smoothed the saliency map with a gaussian filter g(x) ( = 8).

Global contrast-based Histogram based contrast(Lab): Quantization of Lab In lab 將m個顏色與接近的saliency直做平均，做為C的salient Each channel to have 12 different value 85

Global contrast-based Region based contrast Segment the Image [Efficient graph-based image segmentation]

Outline Introduction of saliency map Button-up approach L. Itti’s approach Frequency-tuned Center-surround Depth of field Spectral Residual approach Global contrast based Top-down approach Context-aware Information maximum Measuring visual saliency by site entropy rate

Context-Aware Goal: Convey the image content Liu et al, 2007

Context-Aware Distance between a pair of patches: High salient

Context-Aware Saliency Distance between a pair of patches: K most similar patches at scale r High for K most similar Saliency

Context-Aware Salient at: Multiple scales  foreground Few scales  background Scale 1 Scale 4

Context-Aware Foci = Include distance map X

Outline Introduction of saliency map Button-up approach L. Itti’s approach Frequency-tuned Center-surround Depth of field Spectral Residual approach Global contrast based Top-down approach Context-aware Information maximum Measuring visual saliency by site entropy rate

Measuring visual saliency by site entropy rate 1 Ak為每個base在每個點代表的權重，亦為feature weight

Measuring visual saliency by site entropy rate 2 A fully-connected graph representation is adopted for each

Sub-band graph representation K: base number，i: node number

Sub-band graph representation

Measuring visual saliency by site entropy rate 3 A random walk is adopted on each sub-band graph. And Site entropy rate(SER) is measured the average information from a node to the other

The site entropy rate

Conclusion Image processing is funny Unusual in its neighborhood will correspond to high saliency weight Contrast is the key of saliency The point is how to set it

Reference [1] R. Achanta, F. Estrada, P. Wils, and S. S¨usstrunk. Salient region detection and segmentation. In ICVS, pages 66–75. Springer, 2008. 410, 412, 414 [2] R. Achanta, S. Hemami, F. Estrada, and S. S¨usstrunk. Frequency-tuned salient region detection. In CVPR, pages 1597–1604, 2009. 409, 410, 412, 413, 414, 415 [3] L. Itti, C. Koch, and E. Niebur. A model of saliency based visual attention for rapid scene analysis. IEEE TPAMI, 20(11):1254–1259, 1998. 409, 410, 412, 414 [4] X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In CVPR, pages 1–8, 2007. 410, 412, 413, 414 [5] S. Goferman, L. Zelnik-Manor, and A. Tal. Context-aware saliency detection. In CVPR, 2010. 410, 412, 413, 414, 415 [6] MM Cheng, GX Zhang, N. J. Mitra, X. Huang, S.M. Hu. Global Contrast based Salient Region Detect. In CVPR, 2011 . [7] T. Liu, Z. Yuan, J. Sun, J.Wang, N. Zheng, T. X., and S. H.Y. Learning to detect a salient object. IEEE TPAMI, 33(2):353–367, 2011. 410 [8] W. Wang, Y. Wang, Q. Huang, W. Gao, Measuring Visaul Saliency by Site Entropy Rate, In CVPR, 2010.