Download presentation
Presentation is loading. Please wait.
1
Scene Labeling Using Sparse Precision Matrix
Nasim Souly and Dr. Mubarak Shah Center for Research in Computer Vision University of Central Florida CVPR 2016
2
Introduction Assigning a semantic label to each pixel of an image.
3
Typical Approach Segment an image into super-pixels (segments)
Compute local features for each segment and label these using classifiers Smooth labeling such that neighboring segments receive the same labels Limitations unable to incorporate long-range connections not be able to model the contextual relationships among labels Change to full screen
4
Our Approach Model interaction between labels and segments
An energy minimization over a Graph: Whose structure is captured by Inverse of Covariance matrix (Precision Matrix) Which encodes only significant interactions Which avoids a fully connected graph We use local and global information.
5
Background Covariance Correlation Precision Matrix
ฮฃ=๐๐๐ฃ ๐,๐ =๐ธ( ๐โ๐ธ ๐ ๐โ๐ธ ๐ ) Correlation ๐ถ ๐,๐ = ๐๐๐ฃ(๐,๐) ๐ ๐ ๐ ๐ ๐(๐) Precision Matrix ฮฉ= ฮฃ โ1 Partial Correlation ๐ ๐,๐ ๐ ๐
={๐ ๐๐ } matrix of partial correlations. ๐
๐,๐ = โฮฉ ๐๐ ฮฉ ๐๐ ฮฉ ๐๐
6
Precision Matrix and Graphical model
Partial correlation is zero if and only if X is conditionally independent of Y given Z, under the Gaussian assumption Covariance Precision ฮฃ =T/k ฮฉ= ฮฃ โ1 =k/T Example from David MacKay's talk on Gaussian Process Basics
7
Graphical Lasso Given sample covariance matrix S we want to estimate precision matrix ( ฮฉ= ฮฃ โ1 ) using the penalized log-likelihood: argmax ฮฉ logdet ฮฉ โ๐ก๐ ๐ฮฉ โ๐ ฮฉ 1 S is the empirical Covariance By estimating a sparse precision matrix the structure of the dependency graph between variables is obtained.
8
Graphical model and Sparse Precision Matrix
A sample graphical model Blue indicates positive interaction and red negative interaction precision matrix of the ground truth learnt structure from data Images from โJean Honorio, Luis Ortiz and Dimitris Samaras, Sparse and Locally Constant Gaussian Graphical Models , NIPS 2009โ
9
Proposed Method Segmentation Local Classifiers Global Retrieval
Divide image into coherent segments. Local Classifiers Compute a features including SIFT, color histogram and etc for each segment . Use random forest classifiers to classify each segment. Global Retrieval Retrieve a subset of the nearest neighbors of the query image from the training data. Modify Local Classifiers scores leveraging the global GIST features extracted from the data.
10
Build label graph using correlations
Training Segment training samples Find features of all segments in dataset Train local classifiers on super-pixels using Random Forest Find label graph and pairwise costs between labels for inference Y s are Labels Sparse Inverse of Covariance Data matrix ๐ 1 (1) โฏ ๐ ๐ (1) ๐ 1 (2) โฎ โฏ โฑ ๐ ๐ (2) โฎ ๐ 1 (๐) โฏ ๐ ๐ (๐) Graphical lasso Build label graph using correlations โฎ n images l Labels
11
Inference Given a test image find its segments and compute the features Find the interactions between super-pixels Obtain unary term using classifier scores and global retrieval Compute pair-wise cost between selected connections Optimize the energy function Pair-wise terms by label correlations and Image features Structure of Graph by Glasso segmentation Optimized Solution Unary terms by Classifier (RF) and Retrieval Set
12
Scene Graph Structure Capture the structure of the graph for the image segments Each super-pixel is treated as a random variable Use graphical lasso and find the partial correlation graph, where the zero indicates no edge Dependency between super-pixels are obtained Empirical Precision (๐บ โ๐ ) Estimated Sparse Precision ๐ด
13
Energy Function Optimization
Unary Pairwise โ๐,๐โฮฉ ๐คโ๐๐๐ ฮฉ ๐,๐ โ 0 Confidence from classifier ๐ ๐,๐ =โฮฉ ๐๐ ฮฉ ๐๐ ฮฉ ๐๐ relevancy of two super-pixels based on their correlations
14
Experiments and Results
Label Graph for SIFTFlow data set Using Empirical Precision matrix Using the sparse partial correlation matrix
15
Experiments and Results
Stanford-background data set Method Avg Accuracy Local Classifiers 72.8 Ours (Local Classifiers + Global) 78.9 Ours (Local + Global + Spatia smoothing 82.2 Ours Final (sparse structure) 84.6 Farabet natural [3] 81.4 Gould [9] 77.1 Shauai [21] 80.1
16
Long distance connections
Image Classifier output Spatial Smoothing Our results Ground truth Meaningful long connection refine the label
17
Long distance connections
Image Classifier output Spatial Smoothing Our results Ground truth Meaningful long connection refine the label
18
Experiments and Results
SIFT Flow dataset Method Avg Accuracy Local Classifiers 71.2 Ours (Local Classifiers + Global) 75.3 Ours (Local + Global + Spatial smoothing) 77.7 Ours Final (sparse structure) 80.6 Farabet [3] 78.5 Tighe [26] 78.6 Shauai [21] 80.1 Animation on rows
19
Relations Between Labels make a difference
Image Classifier output Ground truth Spatial Smoothing Our results Mountain-Road negative correlation Higher cost Building-Door positive correlation Lower-cost Sea-Car Negative correlation Higher cost
20
Summary We find dependency and interactions between labels as well as super pixels using sparse precision matrix. Incorporate global information Taking into account long range relationship Avoid over smoothing and fully connected graphs Promising results on different datasets
21
Thank You!
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.