1. Saliency detection with background model
Donghun Yeo
CV Lab.

2. Contents Definition of Saliency Detection
Saliency Detection via Dense and Sparse Reconstruction
Saliency Detection via Absorbing Markov Chain

3. Definition of Saliency Detection
Locate important and interesting regions or objects in an image.
Generally the output is saliency value for each pixel.
input image GT
Result of [Mai et al. 2013]
input image
GT of saliency value
[Mai et al. 2013] Saliency Aggregation: A Data-driven Approach, CVPR 2013

4. Saliency Detection via Dense and Sparse Reconstruction
Xiaohui Li* (DUT, China), Huchuan Lu (DUT,China), Lihe Zhang (DUT, China), Xiang Ruan (OMRON Corporation), Ming-Hsuan Yang (UC Merced, USA)
Saliency Detection via Absorbing Markov Chain
Bowen Jiang* (DUT), Lihe Zhang (DUT, China), huchuan Lu (DUT,China), Chuan Yang, Ming-Hsuan Yang (UC Merced, USA).
Image boundary pixels are background!

5. Reconstruction using background templates
Saliency Detection via Dense and Sparse Reconstruction
Xiaohui Li* (DUT, China), Huchuan Lu (DUT,China), Lihe Zhang (DUT, China), Xiang Ruan (OMRON Corporation), Ming-Hsuan Yang (UC Merced, USA)

6. Image Representation
Generate superpixels (simple linear iterative clustering*)
With multiple scales
Feature for each segment
Mean colors and coordinates
An image : a set of segments, where N is number of superpixels
* R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Susstrunk. Slic superpixels. Technical Report , EPFL, 2010

7. Background Templates
Gathering segments at the image boundary for each scale.
To reconstruct saliency segments with background templates brings larger error!

8. Dense Reconstruction Error
Via PCA
is largest D’ eigenvectors of normalized covariance matrix of
(a) Original Image (b) Dense Reconstruction Error

9. Sparse Reconstruction Error
Spare reconstruction with background templates directly
(a) Original Image (b) Dense RE (c) Sparse RE (d) GT

10. Characteristics of two reconstruction
Dense reconstruction error
More accurate to handle the salient object segments at image boundaries
Sparse reconstruction error
Good to eliminate backgrounds
(a) Original Image (b) Dense RE (c) Sparse RE (d) GT

11. Context-Based Error Propagation
Apply K-means clustering to N image segments
Propagate the saliency to the segments in the same cluster.
The propagations are processed sequentially in the order of their reconstruction errors.
Initial

12. Context-Based Error Propagation
(a) Original Image (b) GT (c) Dense RE (d) DRE Propagated (e) Sparse RE (f) SRE Propagated

13. Pixel Level Saliency (Merging Multiple Scales)
For a pixel, integrate multi-scale reconstruction errors of segments that containing the pixel
Feature of the pixel z
A feature of segment n(s) that containing z in scale s

14. Object-Biased Gaussian Refinement
Gaussian filtering with estimated object center

15. Bayesian Integration of Saliency Maps
Bayes formula

16. Bayesian Integration of Saliency Maps
Two saliency map 𝑆 𝑖 and 𝑆 𝑗
Segment the image with 𝑆 𝑖 by its mean saliency value
𝐹 𝑖 (Foreground pixels), 𝐵 𝑖 (Background pixels)
Make a saliency histogram with 𝑆 𝑖
𝑁 𝑏 𝐹 𝑖 𝑠 : # of pixels of foreground which are in same saliency bin of 𝑠

17. Experiment Evaluation Metric Precision-Recall curve F-measure,
Mean Precision , Mean Recall, Mean F-measure

18. Effect of Each Process

19. Comparison

20. Results

21. Absorbing Markov Chain
A state is absorbing when which has no outgoing edge.
A Markov chain is absorbing if it has at least one absorbing state.
Saliency Detection via Absorbing Markov Chain
Bowen Jiang* (DUT), Lihe Zhang (DUT, China), huchuan Lu (DUT,China), Chuan Yang, Ming-Hsuan Yang (UC Merced, USA).
Make boundary segments as absorbing states,
Use absorbing time to compute the saliency

22. Principles of Markov Chain
A set of states
Transition matrix : 𝑚×𝑚
𝑝 𝑖𝑗 : the probability of moving from state 𝑠 𝑖 to state 𝑠 𝑗

23. Principles of Markov Chain
Absorbing Markov chain
Absorbing state 𝑠 𝑖 : 𝑝 𝑖𝑖 =1
Transient state : not absorbing
A Markov chain with 𝑟 absorbing states and 𝑡 transient states
By renumbering the states so that the transient states come first,
Expected number of times that a chain spends in the transient states 𝑗 given that the chain starts in the transient state 𝑖 : 𝑛 𝑖𝑗
Expected number of times that a chain starting in transient state 𝑖 spends before absorption : , where c is l dim vector all of whose elements 1

24. Graph Construction Generate superpixels
Duplicate the boundary superpixels around the image borders
The duplicated superpixels are absorbing states
Others are transient states
Each node is connected to the transient nodes which neighbor it and transient nodes which share boundaries with its neighboring nodes.
The nodes in the boundaries are fully connected with each other
: mean color similarity
: Affinity 𝑨
: Transition matrix P

25. Saliency Detection
Saliency : Expected number of times that a chain starting in transient state 𝑖 spends before absorption
The normalized 𝑦(𝑖) into [0,1]

26. Two problems!
When the salient region appears at the boundaries : robust to the kind of situation (not always)
When the background region is far from the absorbing states : need to handle
Check the number of regions with the middle level saliency

27. Update Process
When the number of regions with the middle level saliency is large,
Using the mean recurrent time ℎ 𝑖 (when a chain starts from 𝑠 𝑖 , expected number of times to return to state 𝑠 𝑖 )
Saliency as a weighted absorbed time

28. Experiments Evaluation Metric Precision-Recall curve F-measure,
Mean Precision , Mean Recall, Mean F-measure

29. Experiments - ASD, SED

30. Experiments – MSRA Dataset

31. Effect of Update Processing

32. Results

33. ExcutionTime