1. SPONSORED BY SA2014.SIGGRAPH.ORG Annotating RGBD Images of Indoor Scenes Yu-Shiang Wong and Hung-Kuo Chu National Tsing Hua University CGV LAB

2. SA2014.SIGGRAPH.ORG SPONSORED BY Outline Motivation Related Works Annotation Procedure User Study

3. SA2014.SIGGRAPH.ORG SPONSORED BY Motivation Scene understanding is a popular topic. RGBD dataset with high quality semantic annotations are valuable: Learning Evaluations Two fundamental problems Data Acquisition and Annotation

4. SA2014.SIGGRAPH.ORG SPONSORED BY Motivation Scene understanding is a popular topic. RGBD dataset with high quality semantic annotations are valuable: Learning Evaluations Two fundamental problems Data Acquisition and Annotation

5. SA2014.SIGGRAPH.ORG SPONSORED BY RGBD Indoor Datasets Cornell-RGBD (2011-12) : 24 labeled office scenes NYU2 (2011-12) : 1449 labeled indoor scenes –408,000+ RGBD videos frames (unlabeled) SUN 3D (2013) : 415+ full captured room –10+ room is full labeled, annotations are propagated through video. UZH & ETH 3D Scanned Point Datasets (2014) : 42 x full captured room –high quality point clouds (unlabeled) Object Detection and Classification from Large-Scale Cluttered Indoor Scans (EG 2014) …

6. SA2014.SIGGRAPH.ORG SPONSORED BY Data annotation is a painstaking and time- consuming task Motivation OMG! So many data need to be annotated

7. SA2014.SIGGRAPH.ORG SPONSORED BY Data annotation is a painstaking and time- consuming task Interactive tool for annotating RGBD indoor scenes Motivation We need a good tool!

8. SA2014.SIGGRAPH.ORG SPONSORED BY Data annotation is a tedious and time- consuming task Interactive tool for annotating RGBD indoor scenes Leverage both the cognitive ability of human and computational power of machine. Motivation

9. SA2014.SIGGRAPH.ORG SPONSORED BY RELATED WORKS

10. SA2014.SIGGRAPH.ORG SPONSORED BY Image Annotation LabelMe: a database and web-based tool for image annotation. Russell et. al., IJCV 2007 SUN3D: A Database of Big Spaces Reconstructed using SfM and Object Labels, Xiao et.al. ICCV 2013 Cheaper by the Dozen: Group Annotation of 3D Data, Boyko et. al., UIST 2014

11. SA2014.SIGGRAPH.ORG SPONSORED BY Scene Understanding using RGBD Data Image-based Indoor segmentation and support inference from RGBD images. Silberman et.al. ECCV 2012. RGB-(D) scene labeling: Features and algorithms. Ren et. al. CVPR. 2012 Proxy-based Imagining the unseen: Stability- based cuboid arrangements for understanding cluttered indoor scenes. Shao et. al., SIGGRAPH Asia 2014 PanoContext: A whole-room 3d context model for panoramic scene understanding. Zhang et. al., ECCV 2014 Holistic scene understanding for 3D object detection with rgbd cameras., Lin et. al., ICCV 2013 3D- based reasoning with blocks, support, and stability. Xiao et. al. CVPR 2013

12. SA2014.SIGGRAPH.ORG SPONSORED BY Annotation Procedure: Overview Input : RGB-D image Output: Seg., Label, Box proxy, Support structure User Å Input Machine Output

13. SA2014.SIGGRAPH.ORG SPONSORED BY Input RGB-D Image Output Annotated 3D Structure Annotation Procedure: Overview Draw Scribbles Draw Scribbles Estimate Boxes Estimate Boxes Extract Room Extract Room Annotate Label and Structure Annotate Label and Structure User Session Machine Session

14. SA2014.SIGGRAPH.ORG SPONSORED BY Preprocessing Estimate normal Perform over-segmentation using both color and normal map. Efficient graph based image segmentation [Felzenszwalb et.al. 2004] The coarser segmentation is used for room estimation. The finer segmentation is used for user- assisted object segmentation. Annotation Procedure:

15. SA2014.SIGGRAPH.ORG SPONSORED BY Extracting Room Layout Input RGB-D Image Output Annotated 3D Structure Draw Scribbles Draw Scribbles Estimate Boxes Estimate Boxes Extract Room Extract Room Annotate Label and Structure Annotate Label and Structure Annotation Procedure: [Silberman 2012]

16. SA2014.SIGGRAPH.ORG SPONSORED BY User Scribbles Check floor and walls hypotheses If the hypotheses fail, user clicks the segment to identify floor and walls. User draws scribbles to extract the object segments Output Annotated 3D Structure Draw Scribbles Draw Scribbles Estimate Boxes Estimate Boxes Extract Room Extract Room Annotate Label and Structure Annotate Label and Structure User Annotation Procedure: Input RGB-D Image

17. SA2014.SIGGRAPH.ORG SPONSORED BY Estimating Boxes Box orientation = Find out an orthogonal basis in 3D domain (3 unknowns direction) We assume one direction of box is parallel to the normal of floor (1 unknowns direction, 1 by cross product) Box Fitting Method : 1.Filtering point cloud by KNN 2.Project point cloud of a box to floor plane 3.Fit a line in 2D domain to extract a major direction 4.Using cross product to extract last direction. Output Annotated 3D Structure Draw Scribbles Draw Scribbles Estimate Boxes Estimate Boxes Extract Room Extract Room Annotate Label and Structure Annotate Label and Structure Annotation Procedure: Input RGB-D Image

18. SA2014.SIGGRAPH.ORG SPONSORED BY Output Annotated 3D Structure Annotate Label and 3D Structure Draw Scribbles Draw Scribbles Estimate Boxes Estimate Boxes Extract Room Extract Room Annotate Label and Structure Annotate Label and Structure User Tasks : 1.Type in the object label 2.Drag an arrow to specify the support relationships User Annotation Procedure: Input RGB-D Image

19. SA2014.SIGGRAPH.ORG SPONSORED BY Output Annotated 3D Structure Box Quality Refinement (Optional) Draw Scribbles Draw Scribbles Estimate Boxes Estimate Boxes Extract Room Extract Room Annotate Label and Structure Annotate Label and Structure User Tasks : 1.Adjust the orientation of boxes 2.Adjust the size of boxes User Annotation Procedure: Input RGB-D Image

20. SA2014.SIGGRAPH.ORG SPONSORED BY USER STUDY

21. SA2014.SIGGRAPH.ORG SPONSORED BY User Study : Settings Select 50 x scenes across 7 scene class from NYU2 Recruit 2 users, Each user is requested to annotate 50 x scenes Target class : 24 merged object classes List : bed, chair, cabinet, dresser, television, night stand, table, sofa, picture, pillow, … Each scene contains 3-6 objects

22. SA2014.SIGGRAPH.ORG SPONSORED BY User Study : Results Task Type Mean time per box Mean time per scene Total Time Check Room--1.6 sec1.3 min Draw Scribbles16 sec1 min51 min Type Labels4 sec17 sec13 min Drag Supports2 sec9 sec7.5 min Boxes Adjustment 11 sec35 sec29 min System Process Time: calculate normal, fitting planes and boxes: < 3 sec [in C++] Annotation Time: ( 50 x Scenes ) ( Accuracy = 64 %) TOTAL = 101 min

23. SA2014.SIGGRAPH.ORG SPONSORED BY Demo

24. SA2014.SIGGRAPH.ORG SPONSORED BY Conclusion An interactive system to facilitate annotating RGBD indoor scenes. Generating high quality ground truth data with rich annotations Object segments Object labels 3D geometry 3D structure

25. SA2014.SIGGRAPH.ORG SPONSORED BY On Going Work The major bottleneck lie in manual operations: Drawing scribbles Refine box proxy Typing labels Specify structure Incorporate inferring algorithm and 3D structure analysis to reduce the manual burden from the user.

26. SA2014.SIGGRAPH.ORG SPONSORED BY THANKS YOU !