Download presentation

Presentation is loading. Please wait.

Published byAbraham Diggle Modified about 1 year ago

1
Creating Consistent Scene Graphs Using a Probabilistic Grammar Tianqiang LiuSiddhartha ChaudhuriVladimir G. Kim Qi-Xing HuangNiloy J. MitraThomas Funkhouser 11,23 3, Princeton University Cornell University Stanford University TTICUCL

2
Motivation Growing number of 3D scenes online. Google 3D warehouse

3
Motivation Synthesis [Fisher et al 2012, Xu et al 2013] Understanding [Xu et al 2014, Song et al 2014]

4
Goal Input: A scene from Trimble 3D Warehouse

5
Goal Output 1: Semantic segmentations

6
Goal Output 2: Category labels. Door Nightstand Window Mattress Bed frame Pillow Dresser Mirror Heater Artifact

7
Goal Output 2: Category labels at different levels. Bed & supported Dresser & supported

8
Goal Sleeping area Vanity area Door set Output 2: Category labels at different levels.

9
Challenges Shape is not distinctive. Night table Console table

10
Challenges Contextual information Meeting chair Meeting table Study desk Study chair

11
Challenges All-pair contextual information is not distinctive. #pairs feet #pairs feet

12
Challenges

13

14

15
All-pair contextual information is not distinctive. #pairs feet #pairs feet

16
Key Idea Semantic groups Semantic hierarchy

17
Key Idea #pairs feet #pairs feet

18
Pipeline Probabilistic grammar

19
Related Work Van Kaick et al. 2013

20
Related Work Van Kaick et al. 2013Boulch et al. 2013

21
Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

22
Probabilistic Grammar Labels Rules Probabilities

23
Labels bed, night table, sleeping area Examples:

24
Rules sleeping areabed, night table Example:

25
Probabilities Derivation probabilities Cardinality probabilities Geometry probabilities Spatial probabilities

26
Derivation probability bedbed frame, mattress

27
Cardinality probability sleeping areabed, night table

28
Geometry probability

29
Spatial probability

30
Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

31
Pipeline Probabilistic grammar

32
Learning a Probabilistic Grammar Identify objects Speed X 5

33
Learning a Probabilistic Grammar Label objects Speed X 5

34
Learning a Probabilistic Grammar Group objects Speed X 5

35
Learning a Probabilistic Grammar Grammar generation Labels all unique labels Rules Probabilities

36
Learning a Probabilistic Grammar Grammar generation Labels Rules concatenating all children for each label Probabilities Nightstand & supported NightstandTable lamp Nightstand & supported NightstandPlant Nightstand & supported NightstandTable lamp Plant Training example 1:Training example 2:

37
Learning a Probabilistic Grammar Grammar generation Labels Rules Probabilities : learning from occurrence statistics : estimating Gaussian kernels : kernel density estimation

38
Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

39
Pipeline Probabilistic grammar

40
Pipeline Probabilistic grammar

41
Scene parsing Objective function is the unknown hierarchy is the input scene is the probabilistic grammar

42
Scene parsing After applying Bayes’ rule

43
Scene parsing After applying Bayes’ rule Prior of hierarchy

44
Scene parsing After applying Bayes’ rule Prior of hierarchy : probability of a single derivation formulated using

45
Scene parsing After applying Bayes’ rule Prior of hierarchy compensates for decreasing probability as has more internal nodes.

46
Scene parsing After applying Bayes’ rule Likelihood of scene

47
Scene parsing After applying Bayes’ rule Likelihood of scene : geometry probability

48
Scene parsing After applying Bayes’ rule Likelihood of scene : sum of all pairwise spatial probabilities

49
Scene parsing We work in the negative logarithm space

50
Scene parsing Rewrite the objective function recursively where is the root of, is the energy of a sub-tree. XX

51
Scene parsing The search space is prohibitively large … Problem 1: #possible groups is exponential. Problem 2: #label assignments is exponential.

52
Scene parsing Problem 1: #possible groups is exponential. ……… ………

53
Solution: proposing candidate groups. Scene parsing ……… ………

54
Rule: a a c c d d b b a a d d b b c c b b d d a a c c c c a a d d b b … Problem 2: #label assignments is exponential.

55
Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization … … where is partial label of

56
Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization … where … Now #rules and #assignments are both polynomial. The problem can be solved by dynamic programming. is partial label of

57
Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization Convert the result to a parse tree of the original grammar

58
Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

59
Benefit of hierarchy Meeting table Shape only Study chair

60
Benefit of hierarchy Shape only Flat grammar Meeting chair Meeting table Study chair

61
Benefit of hierarchy Shape only Flat grammar Study chair Study desk Meeting chair Meeting table Ours Meeting table Study chair

62
Benefit of hierarchy Shape only Flat grammar Meeting chair Meeting table Ours Study area Meeting area Meeting table Study chair

63
Benefit of hierarchy Shape only Chair Mattress Console table

64
Benefit of hierarchy Shape onlyFlat grammar Chair Nightstand Console table Mattress Console table

65
Benefit of hierarchy Shape onlyFlat grammarOurs Chair Nightstand Console table Mattress Console table Chair Desk

66
Benefit of hierarchy Shape onlyFlat grammarOurs Chair Nightstand Console table Mattress Console table Study area Sleep area

67
Datasets 77 bedrooms30 classrooms8 libraries 17 small bedrooms8 small libraries

68
Benefit of hierarchy Object classification

69
Generalization of our method Parsing Sketch2Scene data set

70
Take-away message Modeling hierarchy improves scene understanding.

71
Limitation and Future Work Modeling correlation in probabilistic grammar.

72
Limitation and Future Work Modeling correlation in probabilistic grammar. Grammar learning from noisy data. Sleep area bed Nightstand & supported Nightstand Table lamp Input scene graphsGrammar

73
Limitation and Future Work Applications in other fields. Modeling correlation in probabilistic grammar. Grammar learning from noisy data. Modeling from RGB-D data [Chen et al. 2014]

74
Acknowledgement Data Kun Xu Discussion Christiane Fellbaum, Stephen DiVerdi Funding NSF, ERC Starting Grant, Intel, Google, Adobe

75
Code and Data

76
Back up slides

77
Learning a Probabilistic Grammar Creating a training set manually 1.Identify leaf-level objects. 2.Provide a label for each object in a scene. 3.Group objects and provide group labels. 4.Maintain consistency. Sleeping area Nightstand & supported NightstandTable lamp Sleeping area Nightstand & supported Nightstand … …

78
Sensitivity analysis Impact of training set size Impact of energy terms

79
Benefit of hierarchy Object classification Object grouping

Similar presentations

© 2016 SlidePlayer.com Inc.

All rights reserved.

Ads by Google