Download presentation

Presentation is loading. Please wait.

Published byJessica Money Modified over 3 years ago

1
**Indoor Scene Segmentation using a Structured Light Sensor**

Nathan Silberman and Rob Fergus ICCV 2011 Workshop on 3D Representation and Recognition Courant Institute

2
**Overview Indoor Scene Recognition using the Kinect**

Introduce new Indoor Scene Depth Dataset Describe CRF-based model Explore the use of rgb/depth cues

3
**Motivation Indoor Scene recognition is hard**

Far less texture than outdoor scenes More geometric structure

4
**Motivation Indoor Scene recognition is hard**

Far less texture than outdoor scenes More geometric structure Kinect gives us depth map (and RGB) Direct access to shape and geometry information

5
**Overview Indoor Scene Recognition using the Kinect**

Introduce new Indoor Scene Depth Dataset Describe CRF-based model Explore the use of rgb/depth cues

6
Capturing our Dataset We wanted a dataset that could be used for scene understanding rather than just detection Kinect normally has AC Adapter Replaced with battery Open Source Drivers used with mouse to record data

7
**Statistics of the Dataset**

Scene Type Number of Scenes Frames Labeled Frames * Bathroom 6 5,588 76 Bedroom 17 22,764 480 Bookstore 3 27,173 784 Cafe 1 1,933 48 Kitchen 10 12,643 285 Living Room 13 19,262 355 Office 14 19,254 319 Total 64 108,617 2,347 Fix the names of columns. Talk them through the columns. DENSELY LABELED!!!! - Mention labelme Most of these are collected from friends apts in new york Each image is 480x640 Each pixel is a 5-tuple: RGB (3) Depth (1) Label (1) * Labels obtained via LabelMe

8
Dataset Examples Living Room Explain the noise RGB Raw Depth Labels

9
**Dataset Examples Living Room RGB Depth* Labels**

Add note about bilateral filtering use to fill in holes RGB Depth* Labels * Bilateral Filtering used to clean up raw depth image

10
**Dataset Examples Bathroom RGB Depth Labels**

WHEN WE FILL IN THE HOLES, ADD SOME TEXT ABOUT PROJECTION + BILATERAL FILTER RGB Depth Labels

11
Dataset Examples Bedroom RGB Depth Labels

12
**Existing Depth Datasets**

RGB-D Dataset [1] Most depth datasets are very small, a few images, but there are a few bigger ones RGBD for small objects only, similar to COIL Uses kinect Make3d is not densely labeled and is all outdoors B3do Uses kinect, is really for detection, not scene understanding has 849 labeled frames, 75 scenes, around 50 objects [1] K. Lai, L. Bo, X. Ren, and D. Fox. A Large-Scale Hierarchical Multi-View RGB-D Object Dataset. ICRA 2011 [2] B. Liu, S. Gould and D. Koller. Single Image Depth Estimation from Predicted Semantic Labels. CVPR 2010 Stanford Make3d [2]

13
**Existing Depth Datasets**

PCD has 52 scenes Point Cloud Data [1] B3DO [2] [1] Abhishek Anand, Hema Swetha Koppula, Thorsten Joachims, Ashutosh Saxena. Semantic Labeling of 3D Point Clouds for Indoor Scenes. NIPS, 2011 [2] A. Janoch, S. Karayev, Y. Jia, J. T. Barron, M. Fritz, K. Saenko, T. Darrell. A Category-Level 3-D Object Dataset: Putting the Kinect to Work. ICCV Workshop on Consumer Depth Cameras for Computer Vision. 2011

14
**Dataset Freely Available**

Images - code Labels Splits Emphasis the raw data is the raw data streams

15
**Overview Indoor Scene Recognition using the Kinect**

Introduce new Indoor Scene Depth Dataset Describe CRF-based model Explore the use of rgb/depth cues

16
**Segmentation using CRF Model**

Cost(labels) = Local Terms(label i) + Spatial Smoothness (label i, label j) Use phrase cost function (energy) Standard CRF, talk about costs. Now we’re going to go into detail with regard to each term, the alternative choices we can make for each term, and we’ll talk about evaluation last. Solving it using standard graph cuts solver The interesting thing is how we incorporate depth using the local appearance, and the spatial smoothness Standard CRF formulation Optimized via graph cuts Discrete label set (~12 classes)

17
**Model Cost(labels) = Local Terms(label i) +**

Spatial Smoothness (label i, label j) = Appearance(label i | descriptor i) Location(i) Solving it using standard graph cuts solver The interesting thing is how we incorporate depth using the local appearance, and the spatial smoothness

18
**Model Cost(labels) = Local Terms(label i) +**

Spatial Smoothness (label i, label j) = Appearance(label i | descriptor i) Location(i) Solving it using standard graph cuts solver The interesting thing is how we incorporate depth using the local appearance, and the spatial smoothness

19
**Appearance Term Appearance(label i | descriptor i)**

Several Descriptor Types to choose from: RGB-SIFT Depth-SIFT Depth-SPIN RGBD-SIFT RGB-SIFT/D-SPIN We’ll evaluate these alternatives later.

20
**Descriptor Type: RGB-SIFT**

RGB image from the Kinect 128 D Say STANDARD SIFT Extracted Over Discrete Grid

21
**Descriptor Type: Depth-SIFT**

Depth image from kinect with linear scaling 128 D Pixel intensity is proportional to depth Talk about why depth sift is a good idea -large scale shape information -small magnitude directional gradient information – essentially surface normals Extracted Over Discrete Grid

22
**Descriptor Type: Depth-SPIN**

Depth image from kinect with linear scaling Radius 50 D Same sift features, but done over linear scaled depth Talk about why depth sift is a good idea -large scale shape information -small magnitude directional gradient information – essentially surface normals Depth Extracted Over Discrete Grid A. E. Johnson and M. Hebert. Using spin images for efficient object recognition in cluttered 3d scenes. IEEE PAMI, 21(5):433–449, 1999

23
**Descriptor Type: RGBD-SIFT**

RGB image from the Kinect Concatenate Depth image from kinect with linear scaling 256 D

24
**Descriptor Type: RGD-SIFT, D-SPIN**

RGB image from the Kinect Concatenate Depth image from kinect with linear scaling 178 D

25
**Descriptor at each location**

Appearance Model Appearance(label i | descriptor i) - Modeled by a Neural Network with a single hidden layer Talk through this in animated way. Last animation is backprop with curly arrow going down Descriptor at each location

26
**Descriptor at each location**

Appearance Model Appearance(label i | descriptor i) Softmax output layer 13 Classes Talk through this in animated way. Last animation is backprop with curly arrow going down 1000-D Hidden Layer 128/178/256-D Input Descriptor at each location

27
**Appearance Model Appearance(label i | descriptor i) Interpreted as**

p(label | descriptor) Probability Distribution over classes 13 Classes Talk through this in animated way. Last animation is backprop with curly arrow going down 1000-D Hidden Layer 128/178/256-D Input Descriptor at each location

28
**Appearance Model Appearance(label i | descriptor i)**

Probability Distribution over classes 13 Classes Trained with backpropagation Talk through this in animated way. Last animation is backprop with curly arrow going down 1000-D Hidden Layer 128/178/256-D Input Descriptor at each location

29
**Model Cost(labels) = Local Terms(label i) +**

Spatial Smoothness (label i, label j) = Appearance(label i | descriptor i) Location(i) Solving it using standard graph cuts solver The interesting thing is how we incorporate depth using the local appearance, and the spatial smoothness

30
**Model Cost(labels) = Local Terms(label i) +**

Spatial Smoothness (label i, label j) = Appearance(label i | descriptor i) Location(i) Solving it using standard graph cuts solver The interesting thing is how we incorporate depth using the local appearance, and the spatial smoothness

31
**Model Cost(labels) = Local Terms(label i) +**

Spatial Smoothness (label i, label j) = Appearance(label i | descriptor i) Location(i) Solving it using standard graph cuts solver The interesting thing is how we incorporate depth using the local appearance, and the spatial smoothness 2D Priors 3D Priors

32
**Location Priors: 2D 2D Priors are histograms of P(class, location)**

Smoothed to avoid image-specific artifacts Check on the math!!!

33
**Motivation: 3D Location Priors**

2D Priors don’t capture 3d geomety 3D Priors can be built from depth data Rooms are of different shapes and sizes, how do we align them? The idea of a 3d prior is mean to capture high degree of regularity…

34
**Motivation: 3D Location Priors**

To align rooms, we’ll use a normalized cylindrical coordinate system: Show that we are dividing each scanline by its max depth to give unit distance Band of maximum depths along each vertical scanline

35
**Relative Depth Distributions**

Table Television Bed Wall Talk through the images better Explain that the cylinder gives relative depth Density 1 1 Relative Depth

36
**Location Priors: 3D Each column is a bin of relative depth**

3 dimensions: Explain binning! - Log Depth Explain Height and angle

37
**Model Cost(labels) = Local Terms(label i) +**

Spatial Smoothness (label i, label j) = Appearance(label i | descriptor i) Location(i) Solving it using standard graph cuts solver The interesting thing is how we incorporate depth using the local appearance, and the spatial smoothness 2D Priors 3D Priors

38
**Model Cost(labels) = Local Terms(label i) +**

Spatial Smoothness (label i, label j) Solving it using standard graph cuts solver The interesting thing is how we incorporate depth using the local appearance, and the spatial smoothness Penalty for adjacent labels disagreeing (Standard Potts Model)

39
**Spatial Modulation of Smoothness**

Model Cost(labels) = Local Terms(label i) + Spatial Smoothness (label i, label j) Solving it using standard graph cuts solver The interesting thing is how we incorporate depth using the local appearance, and the spatial smoothness In practice they don’t make a huge diff Spatial Modulation of Smoothness None RGB Edge Depth Edges RGB + Depth Edges Superpixel Edges Superpixel + RGB Edges Superpixel + Depth Edges

40
**Experimental Setup 60% Train (~1408 images) 40% Test (~939 images)**

10 fold cross validation Images of the same scene cannot appear apart Performance criteria is pixel-level classification (mean diagonal of confusion matrix) 12 most common classes, 1 background class (from the rest)

41
**Evaluating Descriptors**

Percent 5% gain using RGBD over RGB, 7% over just depth Talk over what the blue bars are and what the red bars ar 2D Descriptors 3D Descriptors

42
**Evaluating Location Priors**

Percent Talk over the actual percentage boosts 2D Descriptors 3D Descriptors

43
**2nd column is standard RGB, 3rd is leveraging the depth cues**

44
Point out the failure on top, still somewhat unreliable, still doesn’t understand 3d structure of objects. Absolute numbers may be low but many, many objects in indoor scene,

45
**Conclusion Kinect Depth signal helps scene parsing**

Still a long way from great performance Shown standard approaches on RGB-D data. Lots of potential for more sophisticated methods. No complicated geometric reasoning

46
**Preprocessing the Data**

We use open source calibration software [1] to infer: Parameters of RGB & Depth cameras Homography between cameras. Depth and RGB are not aligned Missing pixels due to Shadow casued by displacement between infrared emmitter and camera Dark/specular surfaces Random noise Depth values are between 0 and 65,536 We need to invert the depth [1] N. Burrus. Kinect RGB Demo v Feb. 2011

47
**Preprocessing the data**

Bilateral filter used to diffuse depth across regions of similar RGB intensity Naïve GPU implementation runs in ~100 ms

48
Motivation Results from Spatial Pyramid-based classification [1] using 5 indoor scene types. Contrast this with the 81% received by [1] on a 13-class (mostly outdoor) scene dataset. They note similar confusion within indoor scenes. [1] Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene CategoriesS. Lazebnik, C. Schmid, and J. Ponce, CVPR 2006

Similar presentations

OK

Parsing Natural Scenes and Natural Language with Recursive Neural Networks INTERNATIONAL CONFERENCE ON MACHINE LEARNING (ICML 2011) RICHARD SOCHER CLIFF.

Parsing Natural Scenes and Natural Language with Recursive Neural Networks INTERNATIONAL CONFERENCE ON MACHINE LEARNING (ICML 2011) RICHARD SOCHER CLIFF.

© 2018 SlidePlayer.com Inc.

All rights reserved.

Ads by Google

Ppt on exterior angle theorem Ppt on switching devices on netflix Ppt on l&t finance holding Ppt on summary writing Convert pdf to ppt online without email Ppt on 9 11 attack on world trade center and effects Ppt on operating system security Atoms for kids ppt on batteries Ppt on object oriented programming Ppt on swami vivekananda thoughts