A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel

Digitalization of indoor scenes Indoor scenes from Google 3D Warehouse

Acquisition of indoor scenes

Goal Scene understanding

Challenges Clutter –Densely populated –Arbitrary arrangements Partial representation –Occlusions Complex geometry

Classification & Segmentation Two interleaved problems –What are the objects? –Where are the objects? Chicken-egg problem –Classification needs segmentation –Segmentation needs a prior

Our solution Search –Propagate / accumulate patches Classify –Query classifier to detect object

Related Work Indoor scenes ( This Session ) – [Fisher et al. 2012] [Shao et al. 2012] [Kim et al. 2012] Semantic relationship – [Fisher et al. 2010, 2011] Recognition using depth + texture (RGB-D) – [Quigley et al.2009], [Lai and Fox 2010] Outdoor classification – [Golovinskiy et al. 2009] Semantic labeling – [Koppula et al. 2011] Controlled region growing process

Our search-classify idea

Method overview Training Search-Classify

Point cloud features –Height-size ratio of BBox –Aspect ratio of each layer –Bottom-top, mid-top size ratio –Change in COM along horizontal slabs BhBh BdBd BwBw

Classifier Handle missing data –Occlusion Random decision forest –Efficient multi-class classifier Trained with both scanned and synthetic data –Manually segmented and labeled –510 chairs –250 tables –110 cabinets –40 monitors etc. [Shotton et al. 2008, 2011]

Search-Classify Starts from seeds –Random patch triplets –Remove seeds with low confidence Accumulating neighbor patches –Highest classification confidence Stop condition –Steep decrease in classification confidence Seed

Segmented - classified objects problems –Overlap, outliers, ambiguities etc. Refinement –Outliers = patches with large distance Segmentation refinement by template fitting

Template deformation Different styles for each class Predefined scalable parts Templates can deform [Xu et al. 2010]

Template deformation Different styles for each class Predefined scalable parts Templates can deform [Xu et al. 2010]

Fitting via template deformation ConfidenceFitting errorBest fitting Best matching template –One-side Euclidean distance from points to template

Results and discussion

Scalability test with varied object density 0 (25) 1 (45) 5 (60)

Results and discussion Comparison Lai et al Ours

Limitation Upward assumption –Features –Template fitting

Future work Contextual information

Thank you