Creating Consistent Scene Graphs Using a Probabilistic Grammar Tianqiang LiuSiddhartha ChaudhuriVladimir G. Kim Qi-Xing HuangNiloy J. MitraThomas Funkhouser 11,23 3, Princeton University Cornell University Stanford University TTICUCL

Motivation Growing number of 3D scenes online. Google 3D warehouse

Motivation Synthesis [Fisher et al 2012, Xu et al 2013] Understanding [Xu et al 2014, Song et al 2014]

Goal Input: A scene from Trimble 3D Warehouse

Goal Output 1: Semantic segmentations

Goal Output 2: Category labels. Door Nightstand Window Mattress Bed frame Pillow Dresser Mirror Heater Artifact

Goal Output 2: Category labels at different levels. Bed & supported Dresser & supported

Goal Sleeping area Vanity area Door set Output 2: Category labels at different levels.

Challenges Shape is not distinctive. Night table Console table

Challenges Contextual information Meeting chair Meeting table Study desk Study chair

Challenges All-pair contextual information is not distinctive. #pairs feet #pairs feet

Challenges

All-pair contextual information is not distinctive. #pairs feet #pairs feet

Key Idea Semantic groups Semantic hierarchy

Key Idea #pairs feet #pairs feet

Pipeline Probabilistic grammar

Related Work Van Kaick et al. 2013

Related Work Van Kaick et al. 2013Boulch et al. 2013

Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

Probabilistic Grammar Labels Rules Probabilities

Labels bed, night table, sleeping area Examples:

Rules sleeping areabed, night table Example:

Probabilities Derivation probabilities Cardinality probabilities Geometry probabilities Spatial probabilities

Derivation probability bedbed frame, mattress

Cardinality probability sleeping areabed, night table

Geometry probability

Spatial probability

Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

Pipeline Probabilistic grammar

Learning a Probabilistic Grammar Identify objects Speed X 5

Learning a Probabilistic Grammar Label objects Speed X 5

Learning a Probabilistic Grammar Group objects Speed X 5

Learning a Probabilistic Grammar Grammar generation Labels all unique labels Rules Probabilities

Learning a Probabilistic Grammar Grammar generation Labels Rules concatenating all children for each label Probabilities Nightstand & supported NightstandTable lamp Nightstand & supported NightstandPlant Nightstand & supported NightstandTable lamp Plant Training example 1:Training example 2:

Learning a Probabilistic Grammar Grammar generation Labels Rules Probabilities : learning from occurrence statistics : estimating Gaussian kernels : kernel density estimation

Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

Pipeline Probabilistic grammar

Pipeline Probabilistic grammar

Scene parsing Objective function is the unknown hierarchy is the input scene is the probabilistic grammar

Scene parsing After applying Bayes’ rule

Scene parsing After applying Bayes’ rule Prior of hierarchy

Scene parsing After applying Bayes’ rule Prior of hierarchy : probability of a single derivation formulated using

Scene parsing After applying Bayes’ rule Prior of hierarchy compensates for decreasing probability as has more internal nodes.

Scene parsing After applying Bayes’ rule Likelihood of scene

Scene parsing After applying Bayes’ rule Likelihood of scene : geometry probability

Scene parsing After applying Bayes’ rule Likelihood of scene : sum of all pairwise spatial probabilities

Scene parsing We work in the negative logarithm space

Scene parsing Rewrite the objective function recursively where is the root of, is the energy of a sub-tree. XX

Scene parsing The search space is prohibitively large … Problem 1: #possible groups is exponential. Problem 2: #label assignments is exponential.

Scene parsing Problem 1: #possible groups is exponential. ……… ………

Solution: proposing candidate groups. Scene parsing ……… ………

Rule: a a c c d d b b a a d d b b c c b b d d a a c c c c a a d d b b … Problem 2: #label assignments is exponential.

Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization … … where is partial label of

Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization … where … Now #rules and #assignments are both polynomial. The problem can be solved by dynamic programming. is partial label of

Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization Convert the result to a parse tree of the original grammar

Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

Benefit of hierarchy Meeting table Shape only Study chair

Benefit of hierarchy Shape only Flat grammar Meeting chair Meeting table Study chair

Benefit of hierarchy Shape only Flat grammar Study chair Study desk Meeting chair Meeting table Ours Meeting table Study chair

Benefit of hierarchy Shape only Flat grammar Meeting chair Meeting table Ours Study area Meeting area Meeting table Study chair

Benefit of hierarchy Shape only Chair Mattress Console table

Benefit of hierarchy Shape onlyFlat grammar Chair Nightstand Console table Mattress Console table

Benefit of hierarchy Shape onlyFlat grammarOurs Chair Nightstand Console table Mattress Console table Chair Desk

Benefit of hierarchy Shape onlyFlat grammarOurs Chair Nightstand Console table Mattress Console table Study area Sleep area

Datasets 77 bedrooms30 classrooms8 libraries 17 small bedrooms8 small libraries

Benefit of hierarchy Object classification

Generalization of our method Parsing Sketch2Scene data set

Take-away message Modeling hierarchy improves scene understanding.

Limitation and Future Work Modeling correlation in probabilistic grammar.

Limitation and Future Work Modeling correlation in probabilistic grammar. Grammar learning from noisy data. Sleep area bed Nightstand & supported Nightstand Table lamp Input scene graphsGrammar

Limitation and Future Work Applications in other fields. Modeling correlation in probabilistic grammar. Grammar learning from noisy data. Modeling from RGB-D data [Chen et al. 2014]

Acknowledgement Data Kun Xu Discussion Christiane Fellbaum, Stephen DiVerdi Funding NSF, ERC Starting Grant, Intel, Google, Adobe

Code and Data

Back up slides

Learning a Probabilistic Grammar Creating a training set manually 1.Identify leaf-level objects. 2.Provide a label for each object in a scene. 3.Group objects and provide group labels. 4.Maintain consistency. Sleeping area Nightstand & supported NightstandTable lamp Sleeping area Nightstand & supported Nightstand … …

Sensitivity analysis Impact of training set size Impact of energy terms

Benefit of hierarchy Object classification Object grouping