1. A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel

2. Digitalization of indoor scenes Indoor scenes from Google 3D Warehouse

3. Acquisition of indoor scenes

4. Goal Scene understanding

5. Challenges Clutter –Densely populated –Arbitrary arrangements Partial representation –Occlusions Complex geometry

6. Classification & Segmentation Two interleaved problems –What are the objects? –Where are the objects? Chicken-egg problem –Classification needs segmentation –Segmentation needs a prior

7. Our solution Search –Propagate / accumulate patches Classify –Query classifier to detect object

8. Related Work Indoor scenes ( This Session ) – [Fisher et al. 2012] [Shao et al. 2012] [Kim et al. 2012] Semantic relationship – [Fisher et al. 2010, 2011] Recognition using depth + texture (RGB-D) – [Quigley et al.2009], [Lai and Fox 2010] Outdoor classification – [Golovinskiy et al. 2009] Semantic labeling – [Koppula et al. 2011] Controlled region growing process

9. Our search-classify idea 0.60.8 0.920.94

10. Method overview Training Search-Classify

11. Point cloud features –Height-size ratio of BBox –Aspect ratio of each layer –Bottom-top, mid-top size ratio –Change in COM along horizontal slabs BhBh BdBd BwBw

12. Classifier Handle missing data –Occlusion Random decision forest –Efficient multi-class classifier Trained with both scanned and synthetic data –Manually segmented and labeled –510 chairs –250 tables –110 cabinets –40 monitors etc. [Shotton et al. 2008, 2011]

13. Search-Classify Starts from seeds –Random patch triplets –Remove seeds with low confidence Accumulating neighbor patches –Highest classification confidence Stop condition –Steep decrease in classification confidence 0.650.920.930.88 Seed

14. Segmented - classified objects problems –Overlap, outliers, ambiguities etc. Refinement –Outliers = patches with large distance Segmentation refinement by template fitting

15. Template deformation Different styles for each class Predefined scalable parts Templates can deform [Xu et al. 2010]

16. Template deformation Different styles for each class Predefined scalable parts Templates can deform [Xu et al. 2010]

17. Fitting via template deformation ConfidenceFitting errorBest fitting Best matching template –One-side Euclidean distance from points to template

18. Results and discussion

20. Scalability test with varied object density 0 (25) 1 (45) 5 (60)

21. Results and discussion Comparison Lai et al. 2011 Ours

22. Limitation Upward assumption –Features –Template fitting

23. Future work Contextual information

24. Thank you