1. Global Contrast based Salient Region Detection
Ming-Ming Cheng, Niloy J. Mitra, Xiaolei Huang, Philip H. S. Torr, and Shi-Min Hu
speaker:Jianhong Yuan
date:2017/6/20

2. salient object and saliency
In these images, salient object are labeled with red circle.
it's the most noticeable foreground object in the scene.
Saliency is often attributed to variations in image properties such as color, gradient, edges, and boundaries.
However, it is difficult to compute the recognition of the salient object region consistent with the results of human labeling.

3. related work (a) original (b)Zhai and Shah (c)Achanta
Y. Zhai and M. Shah, 2006; luminance information
R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk, 2009; average image color
humans tend to focus attention in image regions with high spatial contrast and local variance in pixel correlation.
The first related work define pixel-level saliency based on a pixel's contrast to all other pixels. However, for efficiency they use only luminance information, thus ignoring distinctiveness clues in other channels.
The second related work that directly defines pixel saliency using a pixel's color difference from the average image color.The contrast of salient object and background is not very clear.
these methods also ignore spatial relationships across image parts, which can be critical for reliable saliency detection.
(a) original
(b)Zhai and Shah
(c)Achanta

4. Key work Histogram-based contrast(HC) Region-based contrast(RC)
the aothor propose two medthods to measure saliency.Then I will introduce these two algorithms separately.
(a) original
(b) HC map
(c) RC map

5. RC: high contrast ; spatial relationship ;
Human attention
spatial relationship
HC: high contrast
RC: high contrast ; spatial relationship ;
improvement of HC
People will pay more attention to the image with the surrounding objects have a very strong contrast of the region.
In addition to contrast, spatial relations also play an important role in human attention.
As an improvement of HC ,author try to introduce a new contrast analysis method named RC integrate spatial relationships into region-level contrast computation.

6. HC define saliency value time complexity:O(N)+O(n2)
color distance matric
We can get a HC-map through the HC method.
HC-map assign pixel-wise saliency values based simply on color separation from all other image pixels.Formula in the letter D represents the color distance matrix between pixels in the lab space
This formula can expend as fellowing.
In this definition, the pixels of the same color have the same significance value, so the author rearrange the formulas,
and the pixels of the same color are grouped together to obtain the saliency values for each color.
cl is the color of the pixel Ik
fj is the probability that Cj appears in all the colors of the image.
The time complexity is expensive,so they want to speed up the method by reducing the total number of pixel colors in the image.
time complexity:O(N)+O(n2)
the number of pixel colors in the image

7. Histogram speed up (a) input image (b) color histogram of (a)
In order to reduce the number of colors in the image,
First quantify each color channel into 12 copies, the color will be reduced to 12 * 12 * 12=1728,
then discard the low frequency color , keep the coverage more than 95% of the color ,replace the image pixels fewer than 5% with closest color in the histogram.
The quantized image is less colored, but still guarantee the visual quality required for salient detection.
(a) input image
(b) color histogram of (a)
(c) quantized image

8. Color space smooth before after
saliency value
before
after
The quantization itself will cause artifacts. The similar color may be quantized to different values led to randomness of the saliency value.
The author employed a smoothing procedure to solve this problem.
The saliency value of each color defined by the weighted average of the saliency values of similar colors.
And assign larger weights to those colors closer to it in the color feature space.
In this photo,shows the saliency value of some similar colors were assigned different values before color space smoothing .
But after color space smoothing, similar colors is more likely to be assigned similar saliency values.
Redefine saliency value of each color defined by the weighted average of the saliency values of similar colors

9. region-level contrast
improvement of HC
pixel-level contrast
spatial relationship
high computational cost
region-level contrast
In addition to contrast, spatial relationship also play an important role in human attention.
The cost of using the spatial relationship while calculating pixel-level contrast is very large, consequently,introduce region contrast which integrate spatial relationships into region-level contrast computation.
region contrast(RC)

10. Algorithm flow (RC) image segment create color histogram
initial saliency value
In region contrast,the first thing they do is to segment the image into regions.Nextly,they build the color histogram for each region.
Then compute the saliency value by measuring its color contrast to other regions in the image.
At last,they have introduced a spatial weighting term to increase the effect of closer regions.
after refinement,will get the RC-map like this.
refinement
image regions
RC-map

11. Saliency value of region
define the initial saliency for each region
color distance matric between the two pixels
HC:
color distance matric between the two regions
RC:
weight of the region ri.
In this part,we introduce how to compute the saliency value for each region.
For a region such as rk,its saliency value can be computed through measuring its color contrast to other regions in the image.
w(ri) is the weight of the region ri. The author use the number of pixels in region ri to represent it.So that they can highlight the color contrast of large regions.
the color distance between two regions can be calculated by this formula.
f(ck，i) is the probability of the i-th color ck，i among all nk colors in the rk region .

12. Refinement of RC Spatial weighted region contrast (initial saliency)
In order to increase the spatial impact of the region,they introduced spatial weighting term to compute saliency value of a region.
//后面补上 Ws（rk）的具体说明
This weighting term is related to a average distance between pixels in region rk and the center of the image.So the region close to the center of the image will get a high value.
Ds(rk,ri) is the spatial distance between regions rk and ri.And is defined as Euclidean distance between regions' centroids.
The parameter σ controls the strength of spatial distance weighting,the larger values of σ reduce the effect of spatial weighting,it will cause contrast in father regions contribute more to the saliency of the current region.In the implementation,the author fixed the square of σ is 0.4.
spatial prior weighting term

13. Application SaliencyCut Sketch based image retrival
Here are two applications with Rc method.

14. GrabCut (a) input (b) output
Before introduce SaliencyCut, we may begin with GrabCut,because it will be used iteratively in our SaliencyCut.
In grabCut, make use of humans annotate,for example, select a rectangle region to specify background region then through the iteration medthod of graphCut extract a precise image mask.
But,the saliencyCut can automatic extract salient region without any human labels.
(a) input
(b) output

15. SaliencyCut SaliencyCut candidate region background
Firstly,they use the incomplete trimap to do initial segmentation.They devide the image into two parts according to the a fixed threshold Tb. If the saliency value of image pixels is larger than Tb, then the largest connected region is considered as initial canditate region of the most dominate salient object. The candidate region is regard as unknown part of the trimap,the other region is labeled as background.Through the uknown regions can train foreground color models that used to distinguish the foreground pixels.This is the saliencyCut result,it uniformly highlight salient object.
SaliencyCut

16. SaliencyCut Segmentation by iterative fitting red --> foreground
original
initial segment
trimap
foreground
Different from GrabCut, SaliencyCut is a itreative process rather than one-pass.
第二幅图：At first,set the gray area to background,while others regarded as unknown region.
After first iteration,expand and erode the current image to get a new trimap as show in third image.By now,we can already obtain foreground pixels labeling with red.The region outside dilated region is set to background,while the remaining areas that unchange its color is unknown region in the trimap.
Repeat the iteration atmost four times,finally get a segmentation like this(at the bottom of the middle).
The last image is labeled by human.From this example we can see,SaliencyCut has successfully segment salient region with an accurate result.
background
second trimap
final segment
manual labeled
red --> foreground
green --> background

17. Sketch based image retrival
Salient object
outlines
input sketch
comparing consistency
Shape-based sorting
Sketch-based image retrieval enables users to quickly find the desired graph on a large number of image database.
But for mostly users cannot express fine details exactly through drawings,and the accuracy of image detection is relatively low.
Matching object shapes with a clean background is a more mature area which can achieve over 90% retrival rates.
So we want to extract the shapes of objects to support image retrival. Combine the SaliencyCut and shape matching algorithm to improve the sketch based image retrival rate.
First, the author uses a keyword to retrieve a set of candidate images,then compute its saliency map,perform saliencyCut on these image, the sorting result are ordered by consistency between input sketch and salient object outlines from saliencyCut.
retrival result

18. Sketch based image retrival
first row shows images downloaded from Flickr using keyword ‘giraffe’,
second row shows our retrieval results obtained by comparing user input sketch with SaliencyCut result using shape context measure
third row shows corresponding sketch based retrieval results using other sate-of-the-art method.
Sketch based image comparison

19. object of interest image segmatation
Evaluation
segmentation by fixed thresholding
object of interest image segmatation
The true effectiveness of a saliency detection method depends on the applications.The first evaluation is compared our RC and HC with other saliency detection method. Next one evaluate the saliencyCut method.

20. Segmentation by fixed thresholding
This is the chart reflect precision and zorerecall curves.And we make use of the salient maps from different methods to get a binary segmentation of the salient objects based on a fixed thresholding. The chart clearly show that RC method performed better than other methods.
The red dot in the figure indicates the maximum recall rate, and the author set the threshold is equal to zero.Which means all the pixels are considered to be foreground,because their saliency value is bigger than zero.
Here the precision achieve 0.2,indicate that there are about 20% image pixels belonging to salient region.
(i) Achanta et al.dataset
(ii) MSRA10K dataset

21. Object of interest image segmatation
After evaluating the saliency detection result, next we want to compare the SaliencyCut method with other state-of-the-art algorithm. But all of the segmentations are based on the RC-map.These saliency segmentation methods were given best parameters while using the two largest public availble dataset.The author compared these methods through three parts,average precision,recall and F-measure.
F-measure is the weighted average of Precision and Recall, As the Precision and Recall indicators sometimes appear contradictory situation, so need the F-measure to comprehensive evaluation. In this paper β² is set as 0.3 to emphasizing the impact of precision.
From this comparson,ours SaliencyCut using RC-saliency maps performs better than other methods.
(i) Achanta et al.dataset
(ii) MSRA10K dataset

22. Conclusion
present global contrast based saliency computation methods —— Histogram based contrast(HC)
spatial information-enhanced Region based Contrast (RC).
introduce a novel unsupervised segmentation algorithm —— SaliencyCut
In this paper ,proposed two global contrast based saliency conputation methods and one introduced spatial information.
Using these two method to produce the saliency map we can automatically segment the most noticeble object in a image.
Compared with the HC method, the RC method can obtain a higher quality saliency image, but the computational efficiency is lower.

23. limitations
images with multiple objects may not get all the salient objects.
if the objects occlude each other,cannot generate good results.
Althrough saliencyCut worked well, still have some limitations .images with multiple objects may not get all the salient objects.
And if the objects occlude each other,cannot generate good results .

24. Thank you

25. questions Yanghao
Why choose lab color space instead of RGB color model?
Lab color space is based on our physiological characteristics,and contain more wider color gamut.
Althrough saliencyCut worked well, still have some limitations .images with multiple objects may not get all the salient objects.
And if the objects occlude each other,cannot generate good results .

26. questions Xie kaixi while the quantification based on RGB model?
Actually, the author tried to quantify the RGB and LAB space, choose the effect of a better one
Althrough saliencyCut worked well, still have some limitations .images with multiple objects may not get all the salient objects.
And if the objects occlude each other,cannot generate good results .

27. questions An weizhi
You mentioned that the method you are using does not require an iterative process compared with GrabCut?
I mean compared to GrabCut,we don't need any manual annotation to segment background region,it's a iterative process of grabCut.
How to segment image into regions?
We use the fixed threshold to divide image into two parts
Althrough saliencyCut worked well, still have some limitations .images with multiple objects may not get all the salient objects.
And if the objects occlude each other,cannot generate good results .