Presentation is loading. Please wait.

Presentation is loading. Please wait.

Scene Labeling Using Sparse Precision Matrix

Similar presentations


Presentation on theme: "Scene Labeling Using Sparse Precision Matrix"โ€” Presentation transcript:

1 Scene Labeling Using Sparse Precision Matrix
Nasim Souly and Dr. Mubarak Shah Center for Research in Computer Vision University of Central Florida CVPR 2016

2 Introduction Assigning a semantic label to each pixel of an image.

3 Typical Approach Segment an image into super-pixels (segments)
Compute local features for each segment and label these using classifiers Smooth labeling such that neighboring segments receive the same labels Limitations unable to incorporate long-range connections not be able to model the contextual relationships among labels Change to full screen

4 Our Approach Model interaction between labels and segments
An energy minimization over a Graph: Whose structure is captured by Inverse of Covariance matrix (Precision Matrix) Which encodes only significant interactions Which avoids a fully connected graph We use local and global information.

5 Background Covariance Correlation Precision Matrix
ฮฃ=๐‘๐‘œ๐‘ฃ ๐‘‹,๐‘Œ =๐ธ( ๐‘‹โˆ’๐ธ ๐‘‹ ๐‘Œโˆ’๐ธ ๐‘Œ ) Correlation ๐ถ ๐‘‹,๐‘Œ = ๐‘๐‘œ๐‘ฃ(๐‘‹,๐‘Œ) ๐‘ ๐‘‘ ๐‘‹ ๐‘ ๐‘‘(๐‘Œ) Precision Matrix ฮฉ= ฮฃ โˆ’1 Partial Correlation ๐œŒ ๐‘‹,๐‘Œ ๐‘ ๐‘…={๐œŒ ๐‘—๐‘˜ } matrix of partial correlations. ๐‘… ๐‘—,๐‘˜ = โˆ’ฮฉ ๐‘—๐‘˜ ฮฉ ๐‘—๐‘— ฮฉ ๐‘˜๐‘˜

6 Precision Matrix and Graphical model
Partial correlation is zero if and only if X is conditionally independent of Y given Z, under the Gaussian assumption Covariance Precision ฮฃ =T/k ฮฉ= ฮฃ โˆ’1 =k/T Example from David MacKay's talk on Gaussian Process Basics

7 Graphical Lasso Given sample covariance matrix S we want to estimate precision matrix ( ฮฉ= ฮฃ โˆ’1 ) using the penalized log-likelihood: argmax ฮฉ logdet ฮฉ โˆ’๐‘ก๐‘Ÿ ๐‘†ฮฉ โˆ’๐œ† ฮฉ 1 S is the empirical Covariance By estimating a sparse precision matrix the structure of the dependency graph between variables is obtained.

8 Graphical model and Sparse Precision Matrix
A sample graphical model Blue indicates positive interaction and red negative interaction precision matrix of the ground truth learnt structure from data Images from โ€œJean Honorio, Luis Ortiz and Dimitris Samaras, Sparse and Locally Constant Gaussian Graphical Models , NIPS 2009โ€

9 Proposed Method Segmentation Local Classifiers Global Retrieval
Divide image into coherent segments. Local Classifiers Compute a features including SIFT, color histogram and etc for each segment . Use random forest classifiers to classify each segment. Global Retrieval Retrieve a subset of the nearest neighbors of the query image from the training data. Modify Local Classifiers scores leveraging the global GIST features extracted from the data.

10 Build label graph using correlations
Training Segment training samples Find features of all segments in dataset Train local classifiers on super-pixels using Random Forest Find label graph and pairwise costs between labels for inference Y s are Labels Sparse Inverse of Covariance Data matrix ๐‘‹ 1 (1) โ‹ฏ ๐‘‹ ๐‘› (1) ๐‘‹ 1 (2) โ‹ฎ โ‹ฏ โ‹ฑ ๐‘‹ ๐‘› (2) โ‹ฎ ๐‘‹ 1 (๐‘™) โ‹ฏ ๐‘‹ ๐‘› (๐‘™) Graphical lasso Build label graph using correlations โ‹ฎ n images l Labels

11 Inference Given a test image find its segments and compute the features Find the interactions between super-pixels Obtain unary term using classifier scores and global retrieval Compute pair-wise cost between selected connections Optimize the energy function Pair-wise terms by label correlations and Image features Structure of Graph by Glasso segmentation Optimized Solution Unary terms by Classifier (RF) and Retrieval Set

12 Scene Graph Structure Capture the structure of the graph for the image segments Each super-pixel is treated as a random variable Use graphical lasso and find the partial correlation graph, where the zero indicates no edge Dependency between super-pixels are obtained Empirical Precision (๐‘บ โˆ’๐Ÿ ) Estimated Sparse Precision ๐œด

13 Energy Function Optimization
Unary Pairwise โˆ€๐‘–,๐‘—โˆˆฮฉ ๐‘คโ„Ž๐‘’๐‘Ÿ๐‘’ ฮฉ ๐‘–,๐‘— โ‰ 0 Confidence from classifier ๐œŒ ๐‘™,๐‘˜ =โˆ’ฮฉ ๐‘™๐‘˜ ฮฉ ๐‘™๐‘™ ฮฉ ๐‘˜๐‘˜ relevancy of two super-pixels based on their correlations

14 Experiments and Results
Label Graph for SIFTFlow data set Using Empirical Precision matrix Using the sparse partial correlation matrix

15 Experiments and Results
Stanford-background data set Method Avg Accuracy Local Classifiers 72.8 Ours (Local Classifiers + Global) 78.9 Ours (Local + Global + Spatia smoothing 82.2 Ours Final (sparse structure) 84.6 Farabet natural [3] 81.4 Gould [9] 77.1 Shauai [21] 80.1

16 Long distance connections
Image Classifier output Spatial Smoothing Our results Ground truth Meaningful long connection refine the label

17 Long distance connections
Image Classifier output Spatial Smoothing Our results Ground truth Meaningful long connection refine the label

18 Experiments and Results
SIFT Flow dataset Method Avg Accuracy Local Classifiers 71.2 Ours (Local Classifiers + Global) 75.3 Ours (Local + Global + Spatial smoothing) 77.7 Ours Final (sparse structure) 80.6 Farabet [3] 78.5 Tighe [26] 78.6 Shauai [21] 80.1 Animation on rows

19 Relations Between Labels make a difference
Image Classifier output Ground truth Spatial Smoothing Our results Mountain-Road negative correlation Higher cost Building-Door positive correlation Lower-cost Sea-Car Negative correlation Higher cost

20 Summary We find dependency and interactions between labels as well as super pixels using sparse precision matrix. Incorporate global information Taking into account long range relationship Avoid over smoothing and fully connected graphs Promising results on different datasets

21 Thank You!


Download ppt "Scene Labeling Using Sparse Precision Matrix"

Similar presentations


Ads by Google