Learning to Detect A Salient Object Reporter: 鄭綱 (3/2)

Outline  Introduction  CRF  Formulation of Static Salient Object  Strong Contrast  Center-Surround Histogram  Color Spatial-Distribution  Learning & Inference for the Model  Formulation of Dynamic Salient Object  Results  Conclusion  References

Introduction  Image Labeling Problem Sky Building Lawn Plane Tree

Introduction  Which kinds of information can be used for labeling? Features from individual sites Intensity, color, texture, … Interactions with neighboring sites Contextual information Vegetation Sky or Building?

Introduction  Contextual information: 2 types of interactions Interaction with neighboring labels (Spatial smoothness of labels) neighboring sites tend to have similar labels(except at the discontinuities) Interactions with neighboring observed data Building Sky

Introduction  Let be the label of the node of the image set S, and N i be the neighboring nodes of node i.  Three kinds of information for image labeling  Features from local node  Interaction with neighboring labels  Interaction with neighboring observed data node iS-{i}NiNi

Introduction  General formulation: where and are called association potential and interaction potential.

Introduction Labels in Spatial Data are NOT independent! – spatially adjacent labels are often the same (Markov Random Fields and Conditional Random Fields) – spatially adjacent elements that have similar features often receive the same label (Conditional Random Fields) – spatially adjacent elements that have different features may not have correlated labels (Conditional Random Fields)

Salient Object? Salience Map?

Formulation of Static Salient Object Detection  CRF model (static image): Salient object featurePairwise feature Strong contrast Center- surround histogram Color spatial- distribution Maximize!! Minimize!!

Strong Contrast  Generate contrast map for each level of 6- level Gaussian pyramid. Then do linear combination. Input imageLevel 1 Level 4

Center-Surround Histogram  Salient object usually has a “huge” difference from local area. ……. More different between 2 rectangles !

Center-Surround Histogram where …... ??

Color Spatial-Distribution  The wider a color is distributed in the image, the less possible a salient object contains this color.  Each pixel is assigned to a color component with the probability:

Color Spatial-Distribution  Then the feature can be defined as a weighted sum:

Formulation of Static Salient Object Detection  CRF model (static image): Salient object featurePairwise feature Strong contrast Center- surround histogram Color spatial- distribution Maximize!! Minimize!!

Learning & Inference for the Model  The goal of CRF learning is to estimate the linear weights. Gradient descent Training images

Learning & Inference for the Model A training image for example: where An training image Label inferred this iterate Possible label Label per pixel

Learning & Inference for the Model Gradient descent: How to use ground-truth information? Where is the labeled ground-truth. t: iteration

Learning & Inference for the Model  Situations: Ground-truth mistake!!!

Ground-Truth Mistake  Solution: apply Gaussian function to weight every pixel in the rectangle.

Inference  We should find the most probable labeling to maximize in training & detection.  BP – Max-product Belief Propagation [Pearl ‘86] + Can be applied to any energy function – In vision results are usually worse than that of graph cuts – Does not always converge  TRW - Max-product Tree-reweighted Message Passing [Wainwright, Jaakkola, Willsky ‘02], [Kolmogorov ‘05] + Can be applied to any energy function + Convergence guarantees for the algorithm in [Kolmogorov ’05]

Formulation of Dynamic Salient Object Detection  Similar as static salient object detection! Static Salient feature Maximize!! Contrast of motion Center- surround histogram Spatial- distribution of motion Penalty term of motion

Results From left to right: input image, multi-scale contrast, center- surround histogram, color spatial distribution, and binary mask by CRF.

Results From left to right: Fuzzy growing based method Salience map Their approach Ground-truth

Results 1. multi-scale contrast 2. center-surround histogram 3. color spatial distribution 4. combination all Image set AImage set B 1. FG 2. SM 3. their approach

Conclusion  They model the salient object detection by CRF, where a group of salient features are combined through CRF learning.  It’s a set of novel local, regional & global salient features to define a generic salient object.  Multi-object & no object cases are left as future work.

References (paper & book)  “Learning to Detect A Salient Object”, CVPR 2007, PAMI   “A Model of Saliency-Based Visual Attention for Rapid Scene Analysis”, PAMI,   “Pattern Recognition and Machine Learning”, C. M. Bishop. 

References (about CRF)  “Discriminative Random Fields”, IJCV,  i= &rep=rep1&type=pdf i= &rep=rep1&type=pdf  “Conditional Random Fields: An Introduction”, H. M. Wallach.  i= &rep=rep1&type=pdf i= &rep=rep1&type=pdf  “Log-linear models & conditional random fields”, C. Elkan,  l.pdf l.pdf

References (about TRW)  “MAP estimation via agreement on (hyper) trees: Message-passing and linear programming approaches”, IEEE transaction on Information Theory,   “Convergent Tree-Reweighted Message Passing for Energy Minimization”, PAMI,  &rep=rep1&type=pdf &rep=rep1&type=pdf  “A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors”, PAMI,  &rep=rep1&type=pdf &rep=rep1&type=pdf