1. Xiaofeng Ren: Intel Labs Seattle Dieter Fox: UW and Intel Labs Seattle Kurt Konolige: Willow Garage Jana Kosecka: George Mason University

2.  9:00am - 09:10amWelcome  09:10am - 09:50amOverview and RGB-D Research at Intel Labs and UW D. Fox and X. Ren; University of Washington, Intel Labs Seattle  09:50am - 10:30amInvited Talk (Vision and Graphics) C. Theobalt; Max Planck Institute  10:30am - 11:00amSemantic Parsing in Indoor and Outdoor Scenes J. Kosecka; George Mason University  11:00am - 11:30amCoffee Break  11:30am - 11:50am3D Pose Estimation, Tracking and Model Learning of Articulated Objects from Dense Depth Video using Projected Texture Stereo J. Sturm, K. Konolige, C. Stachniss, W. Burgard; Univ. of Freiburg and Willow Garage  11:50am - 12:10pmLearning Deformable Object Models for Mobile Robot Navigation using Depth Cameras and a Manipulation Robot B. Frank, R. Schmedding, C. Stachniss, M. Teschner, W. Burgard; Univ. of Freiburg  12:10pm - 12:30pm3D Indoor Mapping for Micro-UAVs Using Hybrid Range Finders and Multi-Volume Occupancy Grids W. Morris, I. Dryanovski, J. Xiao; City College of New York  12:30pm - 01:40pmLunch Break  01:40pm - 02:20pmInvited Talk (Robotics and Vision) P. Newman; Oxford University  02:20pm - 03:00pm3D Modeling and Object Recognition at Willow Garage R. Rusu, K. Konolige; Willow Garage  03:00pm - 04:00pmPoster Session and Wrap-Up RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington2

3. Dieter Fox Xiaofeng Ren Intel Labs Seattle University of Washington Department of Computer Science & Engineering

4. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington4  RGB-D: adding depth to color  Dense 3D mapping  Object recognition and modeling  Discussion

5.  Panning 2D scanner, Velodyne, time of flight cameras, stereo  Still very expensive, substantial engineering effort, not dense RSS RGB-D Workshop5Fox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington

6.  Soon we’ll have cheap depth cameras with high resolution and accuracy (>640x480, 30 Hz)  Key industry drivers: Gaming, entertainment  Two main techniques:  Structured light with stereo  Time of flight RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington6

7. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington7 Microsoft Natal promo video

8. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington8

9. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington9  RGB-D: adding depth to color  Dense 3D mapping  Object recognition and modeling  Discussion

10.  Visual odometry via frame to frame matching  Loop closure detection via 3D feature matching  Optimization via TORO, SBA RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington10 [Henry-Herbst- Krainin-Ren-F]

11. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington11

12.  Standard point cloud ICP not robust enough  Limited FOV, lack of features for data association  Add sparse visual features (SIFT, Canny edges)  Improved data association, might fail in dark areas RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington12 Point-to-plane Point-to-point Point-to-edge

13. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington13

14. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington14 Data processing: 4 frames / sec

15. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington15

16. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington16

17. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington17

18.  “Surface Elements” – circular disks representing local surface patches  Introduced by graphics community [Pfister ‘00], [Habbecke ‘07], [Weise ‘09] 18RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington

19. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington19

20. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington20  RGB-D: adding depth to color  Dense 3D mapping  Object recognition and modeling  Discussion

21.  Enable robots to autonomously learn new objects  Robot picks up objects and builds models of them  Models can be shared among robots  Models can contain meta data (where to find, how to grasp, material, what to do with it …) RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington21 [Krainin-Henry-Lai-Ren-F]

22.  Commonly used but requires high accuracy e.g. [Sato ‘97], [Kraft ’08] 22RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington

23. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington23

24. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington24  Builds object model on-the-fly  Jointly tracks hand and object  ICP incorporates dense points, SIFT features, and color gradients

25. 25RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington

26. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington26

27.  Switching Kalman filter  Examining object  Moving to or from table  Grasping or releasing  Between grasps  Second grasp should be computed from partial model 27RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington

28. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington28 159 objects 31 classes 12,554 video frames Shape based segmentation [Lai-Bo-Ren-F: RSS-09, IJRR-10]

29.  Learn local distance function for each object  Sparsification via regularization RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington29

30. RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington30  RGB-D: adding depth to color  Dense 3D mapping  Object recognition and modeling  Discussion

31.  New breed of depth camera systems can have substantial impact on  mapping (3D, semantic, …)  navigation (collision avoidance, 3D path planning)  manipulation (grasping, object recognition)  human robot interaction (detect humans, gestures, …)  Currently mostly constrained to indoors, but outdoors possible too RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington31

32.  Which problems become easy?  Gesture recognition? Grasping? Segmentation? 3D mapping? Object modeling?  Which problems become (more) tractable?  Dense 3D mapping? Object recognition?  What are the new research areas / opportunities generated by RGB-D?  Graphics, visualization, tele-presence  HRI, activity recognition RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington32

33.  What’s the best way to combine shape and color?  depth just an additional dimension?  interest points, feature descriptors, segmentation  How to take advantage of geometric info?  on top of, next to, supports, …  Is depth always necessary?  vision often seems more efficient  can we use RGB-D to train fast RGB systems? RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington33

34.  Hardware: What can we expect in the near future?  Real-time dense 3D reconstruction / mapping  Representation: planes, meshes, surfels, geometric primitives, texture, articulation  Registration: 3D points vs. visual features  Semantic mapping / object recognition  What does 3D add: interest points, feature descriptors, segmentation, spatial information  Humans  Detection, tracking, pose estimation  Gesture and activity recognition RSS RGB-D WorkshopFox / Ren: RGB-D at Intel Labs Seattle and Univ. of Washington34

35. Brian Ferris, Peter Henry, Evan Herbst, Jonathan Ko, Michael Krainin, Kevin Lai, Cynthia Matuszek Post-docs: Liefeng Bo, Marc Deisenroth Intel research: Matthai Philipose, Xiaofeng Ren, Josh Smith