1. Creating Consistent Scene Graphs Using a Probabilistic Grammar Tianqiang LiuSiddhartha ChaudhuriVladimir G. Kim Qi-Xing HuangNiloy J. MitraThomas Funkhouser 11,23 3,451 123 4 5 Princeton University Cornell University Stanford University TTICUCL

2. Motivation Growing number of 3D scenes online. Google 3D warehouse

3. Motivation Synthesis [Fisher et al 2012, Xu et al 2013] Understanding [Xu et al 2014, Song et al 2014]

4. Goal Input: A scene from Trimble 3D Warehouse

5. Goal Output 1: Semantic segmentations

6. Goal Output 2: Category labels. Door Nightstand Window Mattress Bed frame Pillow Dresser Mirror Heater Artifact

7. Goal Output 2: Category labels at different levels. Bed & supported Dresser & supported

8. Goal Sleeping area Vanity area Door set Output 2: Category labels at different levels.

9. Challenges Shape is not distinctive. Night table Console table

10. Challenges Contextual information Meeting chair Meeting table Study desk Study chair

11. Challenges All-pair contextual information is not distinctive. #pairs feet #pairs feet

12. Challenges

15. All-pair contextual information is not distinctive. #pairs feet #pairs feet

16. Key Idea Semantic groups Semantic hierarchy

17. Key Idea #pairs feet #pairs feet

18. Pipeline Probabilistic grammar

19. Related Work Van Kaick et al. 2013

20. Related Work Van Kaick et al. 2013Boulch et al. 2013

21. Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

22. Probabilistic Grammar Labels Rules Probabilities

23. Labels bed, night table, sleeping area Examples:

24. Rules sleeping areabed, night table Example:

25. Probabilities Derivation probabilities Cardinality probabilities Geometry probabilities Spatial probabilities

26. Derivation probability bedbed frame, mattress

27. Cardinality probability sleeping areabed, night table

28. Geometry probability

29. Spatial probability

30. Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

31. Pipeline Probabilistic grammar

32. Learning a Probabilistic Grammar Identify objects Speed X 5

33. Learning a Probabilistic Grammar Label objects Speed X 5

34. Learning a Probabilistic Grammar Group objects Speed X 5

35. Learning a Probabilistic Grammar Grammar generation Labels all unique labels Rules Probabilities

36. Learning a Probabilistic Grammar Grammar generation Labels Rules concatenating all children for each label Probabilities Nightstand & supported NightstandTable lamp Nightstand & supported NightstandPlant Nightstand & supported NightstandTable lamp Plant Training example 1:Training example 2:

37. Learning a Probabilistic Grammar Grammar generation Labels Rules Probabilities : learning from occurrence statistics : estimating Gaussian kernels : kernel density estimation

38. Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

39. Pipeline Probabilistic grammar

40. Pipeline Probabilistic grammar

41. Scene parsing Objective function is the unknown hierarchy is the input scene is the probabilistic grammar

42. Scene parsing After applying Bayes’ rule

43. Scene parsing After applying Bayes’ rule Prior of hierarchy

44. Scene parsing After applying Bayes’ rule Prior of hierarchy : probability of a single derivation formulated using

45. Scene parsing After applying Bayes’ rule Prior of hierarchy compensates for decreasing probability as has more internal nodes.

46. Scene parsing After applying Bayes’ rule Likelihood of scene

47. Scene parsing After applying Bayes’ rule Likelihood of scene : geometry probability

48. Scene parsing After applying Bayes’ rule Likelihood of scene : sum of all pairwise spatial probabilities

49. Scene parsing We work in the negative logarithm space

50. Scene parsing Rewrite the objective function recursively where is the root of, is the energy of a sub-tree. XX

51. Scene parsing The search space is prohibitively large … Problem 1: #possible groups is exponential. Problem 2: #label assignments is exponential.

52. Scene parsing Problem 1: #possible groups is exponential. ……… ………

53. Solution: proposing candidate groups. Scene parsing ……… ………

54. Rule: a a c c d d b b a a d d b b c c b b d d a a c c c c a a d d b b … Problem 2: #label assignments is exponential.

55. Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization … … where is partial label of

56. Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization … where … Now #rules and #assignments are both polynomial. The problem can be solved by dynamic programming. is partial label of

57. Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization Convert the result to a parse tree of the original grammar

58. Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results

59. Benefit of hierarchy Meeting table Shape only Study chair

60. Benefit of hierarchy Shape only Flat grammar Meeting chair Meeting table Study chair

61. Benefit of hierarchy Shape only Flat grammar Study chair Study desk Meeting chair Meeting table Ours Meeting table Study chair

62. Benefit of hierarchy Shape only Flat grammar Meeting chair Meeting table Ours Study area Meeting area Meeting table Study chair

63. Benefit of hierarchy Shape only Chair Mattress Console table

64. Benefit of hierarchy Shape onlyFlat grammar Chair Nightstand Console table Mattress Console table

65. Benefit of hierarchy Shape onlyFlat grammarOurs Chair Nightstand Console table Mattress Console table Chair Desk

66. Benefit of hierarchy Shape onlyFlat grammarOurs Chair Nightstand Console table Mattress Console table Study area Sleep area

67. Datasets 77 bedrooms30 classrooms8 libraries 17 small bedrooms8 small libraries

68. Benefit of hierarchy Object classification

69. Generalization of our method Parsing Sketch2Scene data set

70. Take-away message Modeling hierarchy improves scene understanding.

71. Limitation and Future Work Modeling correlation in probabilistic grammar.

72. Limitation and Future Work Modeling correlation in probabilistic grammar. Grammar learning from noisy data. Sleep area bed Nightstand & supported Nightstand Table lamp Input scene graphsGrammar

73. Limitation and Future Work Applications in other fields. Modeling correlation in probabilistic grammar. Grammar learning from noisy data. Modeling from RGB-D data [Chen et al. 2014]

74. Acknowledgement Data Kun Xu Discussion Christiane Fellbaum, Stephen DiVerdi Funding NSF, ERC Starting Grant, Intel, Google, Adobe

75. Code and Data http://www.cs.princeton.edu/~tianqian/projects/hierarchy/

76. Back up slides

77. Learning a Probabilistic Grammar Creating a training set manually 1.Identify leaf-level objects. 2.Provide a label for each object in a scene. 3.Group objects and provide group labels. 4.Maintain consistency. Sleeping area Nightstand & supported NightstandTable lamp Sleeping area Nightstand & supported Nightstand … …

78. Sensitivity analysis Impact of training set size Impact of energy terms

79. Benefit of hierarchy Object classification Object grouping