1. Scene Labeling Using Sparse Precision Matrix
Nasim Souly and Dr. Mubarak Shah
Center for Research in Computer Vision
University of Central Florida
CVPR 2016

2. Introduction
Assigning a semantic label to each pixel of an image.

3. Typical Approach Segment an image into super-pixels (segments)
Compute local features for each segment and label these using classifiers
Smooth labeling such that neighboring segments receive the same labels
Limitations
unable to incorporate long-range connections
not be able to model the contextual relationships among labels
Change to full screen

4. Our Approach Model interaction between labels and segments
An energy minimization over a Graph:
Whose structure is captured by Inverse of Covariance matrix (Precision Matrix)
Which encodes only significant interactions
Which avoids a fully connected graph
We use
local and
global information.

5. Background Covariance Correlation Precision Matrix
Σ=𝑐𝑜𝑣 𝑋,𝑌 =𝐸( 𝑋−𝐸 𝑋 𝑌−𝐸 𝑌 )
Correlation
𝐶 𝑋,𝑌 = 𝑐𝑜𝑣(𝑋,𝑌) 𝑠𝑑 𝑋 𝑠𝑑(𝑌)
Precision Matrix
Ω= Σ −1
Partial Correlation 𝜌 𝑋,𝑌 𝑍
𝑅={𝜌 𝑗𝑘 } matrix of partial correlations.
𝑅 𝑗,𝑘 = −Ω 𝑗𝑘 Ω 𝑗𝑗 Ω 𝑘𝑘

6. Precision Matrix and Graphical model
Partial correlation is zero if and only if X is conditionally independent of Y given Z, under the Gaussian assumption
Covariance
Precision
Σ =T/k
Ω= Σ −1 =k/T
Example from David MacKay's talk on Gaussian Process Basics

7. Graphical Lasso
Given sample covariance matrix S we want to estimate precision matrix ( Ω= Σ −1 ) using the penalized log-likelihood:
argmax Ω logdet Ω −𝑡𝑟 𝑆Ω −𝜆 Ω 1
S is the empirical Covariance
By estimating a sparse precision matrix the structure of the dependency graph between variables is obtained.

8. Graphical model and Sparse Precision Matrix
A sample graphical model
Blue indicates positive interaction and red negative interaction
precision matrix of the ground truth
learnt structure from data
Images from “Jean Honorio, Luis Ortiz and Dimitris Samaras, Sparse and Locally Constant Gaussian Graphical Models , NIPS 2009”

9. Proposed Method Segmentation Local Classifiers Global Retrieval
Divide image into coherent segments.
Local Classifiers
Compute a features including SIFT, color histogram and etc for each segment .
Use random forest classifiers to classify each segment.
Global Retrieval
Retrieve a subset of the nearest neighbors of the query image from the training data.
Modify Local Classifiers scores leveraging the global GIST features extracted from the data.

10. Build label graph using correlations
Training
Segment training samples
Find features of all segments in dataset
Train local classifiers on super-pixels using Random Forest
Find label graph and pairwise costs between labels for inference
Y s are Labels
Sparse Inverse of Covariance
Data matrix
𝑋 1 (1) ⋯ 𝑋 𝑛 (1) 𝑋 1 (2) ⋮ ⋯ ⋱ 𝑋 𝑛 (2) ⋮ 𝑋 1 (𝑙) ⋯ 𝑋 𝑛 (𝑙)
Graphical lasso
Build label graph using correlations ⋮
n images
l Labels

11. Inference
Given a test image find its segments and compute the features
Find the interactions between super-pixels
Obtain unary term using classifier scores and global retrieval
Compute pair-wise cost between selected connections
Optimize the energy function
Pair-wise terms by label correlations and Image features
Structure of Graph by Glasso
segmentation
Optimized Solution
Unary terms by
Classifier (RF) and Retrieval Set

12. Scene Graph Structure
Capture the structure of the graph for the image segments
Each super-pixel is treated as a random variable
Use graphical lasso and find the partial correlation graph, where the zero indicates no edge
Dependency between super-pixels are obtained
Empirical Precision (𝑺 −𝟏 )
Estimated Sparse Precision 𝜴

13. Energy Function Optimization
Unary
Pairwise
∀𝑖,𝑗∈Ω 𝑤ℎ𝑒𝑟𝑒 Ω 𝑖,𝑗 ≠0
Confidence from classifier
𝜌 𝑙,𝑘 =−Ω 𝑙𝑘 Ω 𝑙𝑙 Ω 𝑘𝑘
relevancy of two super-pixels based on their correlations

14. Experiments and Results
Label Graph for SIFTFlow data set
Using Empirical Precision matrix
Using the sparse partial correlation matrix

15. Experiments and Results
Stanford-background data set
Method
Avg Accuracy
Local Classifiers
72.8
Ours (Local Classifiers + Global)
78.9
Ours (Local + Global + Spatia smoothing
82.2
Ours Final (sparse structure)
84.6
Farabet natural [3]
81.4
Gould [9]
77.1
Shauai [21]
80.1

16. Long distance connections
Image
Classifier output
Spatial Smoothing
Our results
Ground truth
Meaningful long connection refine the label

17. Long distance connections
Image
Classifier output
Spatial Smoothing
Our results
Ground truth
Meaningful long connection refine the label

18. Experiments and Results
SIFT Flow dataset
Method
Avg Accuracy
Local Classifiers
71.2
Ours (Local Classifiers + Global)
75.3
Ours (Local + Global + Spatial smoothing)
77.7
Ours Final (sparse structure)
80.6
Farabet [3]
78.5
Tighe [26]
78.6
Shauai [21]
80.1
Animation on rows

19. Relations Between Labels make a difference
Image
Classifier output
Ground truth
Spatial Smoothing
Our results
Mountain-Road
negative correlation
Higher cost
Building-Door
positive correlation
Lower-cost
Sea-Car
Negative correlation
Higher cost

20. Summary
We find dependency and interactions between labels as well as super pixels using sparse precision matrix.
Incorporate global information
Taking into account long range relationship
Avoid over smoothing and fully connected graphs
Promising results on different datasets

21. Thank You!