Download presentation

1
**DTAM: Dense Tracking and Mapping in Real-Time**

Newcombe, Lovegrove & Davison ICCV11 Amaury Dame Active Vision Lab Oxford Robotics Research Group

2
**Introduction Input : Objective : Single hand held RGB camera**

Dense mapping Dense tracking Typical aims of surveillance: Detection of targets Wide field of view beneficial Tracking of targets, Identification of targets Small field of view beneficial Collaborative sensing Input image 3D dense map Amaury Dame Active Vision Lab Oxford Robotics Research Group 2 Slide 2

3
System overview Plan : Depth map estimation : Notations, photometric error energy, TV optimisation Dense tracking Amaury Dame Active Vision Lab Oxford Robotics Research Group 3 Slide 3

4
**Depth map estimation Principle: Formulation:**

S depth hypothesis are considered for each pixel of the reference image Ir Each corresponding 3D point is projected onto a bundle of images Im Keep the depth hypothesis that best respects the color consistency from the reference to the bundle of images Typical aims of surveillance: Detection of targets Wide field of view beneficial Tracking of targets, Identification of targets Small field of view beneficial Collaborative sensing Formulation: : pixel position and depth hypothesis : number of valid reprojection of the pixel in the bundle : photometric error between reference and current image Amaury Dame Active Vision Lab Oxford Robotics Research Group 4 Slide 4

5
**Depth map estimation Reprojection in image bundle**

Example reference image pixel Typical aims of surveillance: Detection of targets Wide field of view beneficial Tracking of targets, Identification of targets Small field of view beneficial Collaborative sensing Photo error Depth hypotheses Reprojection of depth hypotheses on one image of bundle Amaury Dame Active Vision Lab Oxford Robotics Research Group 5 Slide 5

6
**Depth map filtering approach**

Problem: Uniform regions in reference image do not give discriminative enough photometric error Typical aims of surveillance: Detection of targets Wide field of view beneficial Tracking of targets, Identification of targets Small field of view beneficial Collaborative sensing Idea: Assume that depth is smooth on uniform regions Use total variational approach where depth map is the functional to optimize: photometric error defines the data term the smoothness constraint defines the regularization. Amaury Dame Active Vision Lab Oxford Robotics Research Group 6 Slide 6

7
**Depth map filtering approach**

Formulation: First term : regularization constraint, g is defined so that it is 0 for image gradients and 1 for uniform regions. So that gradient on depth map is penalized for uniform regions Second term : data term defined by the photometric error. Huber norm: differentiable replacement to L1 norm that better preserve discontinuities compared to L2. Typical aims of surveillance: Detection of targets Wide field of view beneficial Tracking of targets, Identification of targets Small field of view beneficial Collaborative sensing Amaury Dame Active Vision Lab Oxford Robotics Research Group 7 Slide 7

8
**Regularisation effect**

Total variational optimisation L2 norm L1 norm QU(f1)=1 QU(f2)=0.1 QU(f3)=0.01 TV(f1)=1 TV(f2)=1 TV(f3)=1 Regularisation effect Typical aims of surveillance: Detection of targets Wide field of view beneficial Tracking of targets, Identification of targets Small field of view beneficial Collaborative sensing Image denoising [Pock08] Amaury Dame Active Vision Lab Oxford Robotics Research Group 8 Slide 8

9
**Depth map filtering approach**

Formulation : Problem : optimizing this equation directly requires linearising of cost volume. Expensive and cost volume has many local minima. Typical aims of surveillance: Detection of targets Wide field of view beneficial Tracking of targets, Identification of targets Small field of view beneficial Collaborative sensing Approximation : Introduce as an auxiliary variable, can be optimized with heuristic search Second terms brings original and auxiliary variable together Amaury Dame Active Vision Lab Oxford Robotics Research Group 9 Slide 9

10
**Total variational optimisation**

Classical approaches: Time Marching Scheme: steepest descent method Linearization of the Euler-Lagrange Equation Problem: optimization badly conditioned as (uniform regions) Euler lagrange equation - Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods - Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods Problem of previous solutions: degenerated as NablaU=0 Reformulation of regularization with primal dual method Dual variable p is introduced to compute the TV norm: Indeed: Amaury Dame Active Vision Lab Oxford Robotics Research Group 10 Slide 10

11
**Increasing solution accuracy ?**

Typical aims of surveillance: Detection of targets Wide field of view beneficial Tracking of targets, Identification of targets Small field of view beneficial Collaborative sensing Reminder: Before Approach: Q well modeled, perform Newton step on Q to update estimation a Equivalent to using Epsilon ? After one iteration Amaury Dame Active Vision Lab Oxford Robotics Research Group 11 Slide 11

12
**Dense tracking Inputs: Tracking as a registration problem**

3D texture model of the scene Pose at previous frame Tracking as a registration problem First inter-frame rotation estimation : the previous image is aligned on the current image to estimate a coarse inter-frame rotation Estimated pose is used to project the 3D model into 2.5D image The 2.5D image is registered with the current frame to find the current pose. Euler lagrange equation - Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods - Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods Problem of previous solutions: degenerated as NablaU=0 Two template matching problems Amaury Dame Active Vision Lab Oxford Robotics Research Group 12 Slide 12

13
**SSD optimisation Problem: Formulation: Hypothesis:**

Align template image T(x) with input image I(x). Formulation: find the transformation that best maps the pixels of the templates into the ones of the current image minimizing: are the displacement parameters to be optimized. Templates can be 2D, 2.5D, 3D as long as warp function defined to project model in 2D current image. Hypothesis: Know a coarse approximation of the template position (p0). Amaury Dame Active Vision Lab Oxford Robotics Research Group 13 Slide 13

14
**SSD optimisation Problem: Formulations: minimize**

The current estimation of p is iteratively updated to reach the minimum of the function. Formulations: Direct additional Direct compositional Inverse Euler lagrange equation - Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods - Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods Problem of previous solutions: degenerated as NablaU=0 Amaury Dame Active Vision Lab Oxford Robotics Research Group 14 Slide 14

15
**SSD optimisation Example: Direct additive method Minimize :**

First order Taylor expansion: Solution: with: Euler lagrange equation - Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods - Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods Problem of previous solutions: degenerated as NablaU=0 Amaury Dame Active Vision Lab Oxford Robotics Research Group 15 Slide 15

16
**SSD robustified Reminder: Method :**

Problem: In case of occlusion, the occluded pixels cause the optimum of the function to be changed. The occluded pixels have to be ignored from the optimization Euler lagrange equation - Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods - Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods Problem of previous solutions: degenerated as NablaU=0 Method : Only the pixels with a difference lower than a threshold are selected. Threshold is iteratively updated to get more selective as the optimization reaches the optimum. Amaury Dame Active Vision Lab Oxford Robotics Research Group 16 Slide 16

17
**Template matching Applications to DTAM: First rotation estimation:**

the template is the previous image that is matched with current image. Warp is defined on the space of all rotations. The initial estimate of p is identity. Full pose estimation template is 2.5D, warp is defined by full 3D motion estimation, that is The initial pose is given by the pose estimated at the previous frame and the inter frame rotation estimation. Euler lagrange equation - Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods - Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods Problem of previous solutions: degenerated as NablaU=0 Amaury Dame Active Vision Lab Oxford Robotics Research Group 17 Slide 17

18
**Conclusion First live full dense reconstruction system...**

Limitation from the smoothness assumption on depth... Euler lagrange equation - Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods - Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods Problem of previous solutions: degenerated as NablaU=0 Amaury Dame Active Vision Lab Oxford Robotics Research Group 18 Slide 18

19
Important references [Pock Thesis08] Fast total variation for Computer Vision [Baker IJCV04] Lucas-Kanade 20 years on: A unifying framework Typical aims of surveillance: Detection of targets Wide field of view beneficial Tracking of targets, Identification of targets Small field of view beneficial Collaborative sensing Amaury Dame Active Vision Lab Oxford Robotics Research Group 19 Slide 19

Similar presentations

OK

Joint work with Andre Lieutier Dassault Systemes Domain Theory and Differential Calculus Abbas Edalat Imperial College Oxford.

Joint work with Andre Lieutier Dassault Systemes Domain Theory and Differential Calculus Abbas Edalat Imperial College Oxford.

© 2018 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google