1. RGB-D Mapping: Using Depth Cameras for Dense 3D Modeling of Indoor Environments
Peter Henry1, Michael Krainin1, Evan Herbst1,
Xiaofeng Ren2, and Dieter Fox1,2
1University of Washington Computer Science and Engineering
2Intel Labs Seattle

2. RGB-D Data Recently Microsoft Kinect has been in the news ($150)
We have been using a PrimeSense reference design (over the past year)
640 x 480 color and dense depth at 30 hz
Projecting IR texture onto scene to obtain dense depth values even in textureless areas (as opposed to passive stereo)
Limitations:
FOV (60 deg)
Depth range (5 m, up to 10 with less accuracy)
Depth accuracy (vary over ~3cm range at 3m)
Resolution 640x480 for images is low
Motion blur / rolling shutter / depth-to-image registration

3. Here is a flythrough of one of our maps
Applications
Robotic navigation and manipulation
Semantic mapping / object recognition
Remote telepresense / search + rescue
Architectural models and remodeling
How did we get this map?

4. Related Work SLAM Loop Closure Detection Photo collections
[Davison et al, PAMI 2007] (monocular)
[Konolige et al, IJR 2010] (stereo)
[Pollefeys et al, IJCV 2007] (multi-view stereo)
[Borrmann et al, RAS 2008] (3D laser)
[May et al, JFR 2009] (ToF sensor)
Loop Closure Detection
[Nister et al, CVPR 2006]
[Paul et al, ICRA 2010]
Photo collections
[Snavely et al, SIGGRAPH 2006]
[Furukawa et al, ICCV 2009]
Todo: No titles
Todo: Pictures per section
Monocular SLAM
Scale ambiguity
(We need large scale environments)
Robust to lack of visual features
Resulting map dense
Stereo SLAM
We have lower resolution
Robust to frames with no visual features
3D Laser slam
We have limited field of view, limited depth range, limited depth accuracy
Want to take advantage of visual features
Loop Closure Detection
Useful for global consistency
We want a metrically accurate map (need to correct after detection)
Photo collections
Sparse features
Filling in with multi-view stereo doesn’t work in textureless areas
Computationally expensive
“MonoSLAM: real-time single camera SLAM” [Davison et al, PAMI 2007]
“View-Based Maps” [Konolige et al, IJR 2010
“Detailed Real-Time Urban 3D Reconstruction from Video” [Pollefeys et al, IJCV 2007]
“Globally consistent 3D mapping with scan matching” [Borrmann et al, RAS 2008]
“Three-dimensional mapping with time-of-flight cameras” [May et al, JFR 2009]
“Scalable Recognition with a Vocabulary Tree” [Nister et al, CVPR 2006]
“FAB-MAP 3D: Topological Mapping with Spatial and Visual Appearance” [Paul et al, ICRA 2010]
“Photo Tourism: Exploring Photo Collections in 3D” [Snavely et al, SIGGRAPH 2006]
“Reconstructing Building Interiors from Images” [Furukawa et al, ICCV 2009]

5. System Overview Frame-to-frame alignment
Loop closure detection and global consistency
Map representation
- Implemented in ROS (Willow Garage), c++
An outline of our system
Alignment
RANSAC (visual)
ICP (shape)
Loop Closure
Detect
Correct
Surfel Maps
Fuse all measurements
Examine components in more detail…

6. Alignment (3D SIFT RANSAC)
SIFT features (from image) in 3D (from depth)
RANSAC alignment requires no initial estimate
Requires sufficient matched features
- Use SiftGPU (150 ms per frame for features, 80ms for RANSAC)
Also SURF and FAST / Calonder implemented

7. RANSAC Failure Case

8. Alignment (ICP) Does not need visual features or color
Requires sufficient geometry and initial estimate
-Say iterative closest point
Limited FOV, needs geometry, needs good initial estimate, slower

9. ICP Failure Case

10. Joint Optimization (RGBD-ICP)
Fixed feature matches
Dense point-to-plane
-Details
Point-to-plane requires Levenburg-Marquardt but fewer iterations
KD-Tree for nearest neighbor
Downsample source cloud + target cloud (10% still works)
Discard percentage of outliers / and those over max distance
Discard boundary points

11. Loop Closure Sequential alignments accumulate error
Revisiting a previous location results in an inconsistent map

12. Loop Closure Detection
Detect by running RANSAC against previous frames
Pre-filter options (for efficiency):
Only a subset of frames (keyframes)
Every nth frame
New Keyframe when RANSAC fails directly against previous keyframe
Only keyframes with similar estimated 3D pose
Place recognition using vocabulary tree
Place recognition is since paper publication

13. Loop Closure Correction
TORO [Grisetti 2007]:
Constraints between camera locations in pose graph
Maximum likelihood global camera poses
Put pose graph picture here (eliminate SBA). Move SBA to the end or an end animation.
Title: Global Alignment. Global optimization. Global consistency.

16. Map Representation: Surfels
Surface Elements [Pfister 2000, Weise 2009, Krainin 2010]
Circular surface patches
Accumulate color / orientation / size information
Incremental, independent updates
Incorporate occlusion reasoning
750 million points reduced to 9 million surfels
Mention: as a post processing for final representation
Pfister: Rendering
Weise: In human hand modeling
Krainin: In-robot hand modeling, ICRA workshops, International Journal of Robotics Research (to appear)
Each frame contains up to 300,000 3d points
We have TOO MUCH data
Alternatives:
Downsample: Lose data
Mesh:
incremental updates difficult.
Overlap makes updates potentially inefficient
Not parallel (GPU)
Volumetric: Memory difficult to obtain detail at the size of our environments

18. Experiment 1: RGBD-ICP 16 locations marked in environment (~7m apart)
Camera on tripod carried between and placed at markers
Error between ground truth distance and the accumulated distance from frame-to-frame alignment
α weight
0.0 (ICP)
0.2
0.5
0.8
1.0 (RANSAC)
Mean error (m)
0.22
0.19
0.16
0.18
1.29

19. Mean Translational Error (m) Mean Angular Error (deg)
Experiment 2: Robot Arm
Camera held in WAM arm for ground truth pose
Method
Mean Translational Error (m)
Mean Angular Error (deg)
RANSAC
.168
4.5
ICP
.144
3.9
RGBD-ICP (α=0.5)
.123
3.3
RGBD-ICP + TORO
.069
2.8

20. Map Accuracy: Architectural Drawing
114m long way
2130 frames

21. Map Accuracy: 2D Laser Map
A second building
Walls are truly curved
70m sides
2625 frames

22. Conclusion
Kinect-style depth cameras have recently become available as consumer products
RGB-D Mapping can generate rich 3D maps using these cameras
RGBD-ICP combines visual and shape information for robust frame-to-frame alignment
Global consistency achieved via loop closure detection and optimization (RANSAC, TORO, SBA)
Surfels provide a compact map representation

23. Open Questions Which are the best features to use?
How to find more loop closure constraints between frames?
What is the right representation (point clouds, surfels, meshes, volumetric, geometric primitives, objects)?
How to generate increasingly photorealistic maps?
Autonomous exploration for map completeness?
Can we use these rich 3D maps for semantic mapping?

24. Thank You!