1. DTAM: Dense Tracking and Mapping in Real-Time
Newcombe, Lovegrove & Davison ICCV11
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group

2. Introduction Input : Objective : Single hand held RGB camera
Dense mapping
Dense tracking
Typical aims of surveillance:
Detection of targets  Wide field of view beneficial
Tracking of targets, Identification of targets
 Small field of view beneficial
Collaborative sensing
Input image
3D dense map
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 2
Slide 2

3. System overview
Plan :
Depth map estimation : Notations, photometric error energy, TV optimisation
Dense tracking
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 3
Slide 3

4. Depth map estimation Principle: Formulation:
S depth hypothesis are considered for each pixel of the reference image Ir
Each corresponding 3D point is projected onto a bundle of images Im
Keep the depth hypothesis that best respects the color consistency from the reference to the bundle of images
Typical aims of surveillance:
Detection of targets  Wide field of view beneficial
Tracking of targets, Identification of targets
 Small field of view beneficial
Collaborative sensing
Formulation:
: pixel position and depth hypothesis
: number of valid reprojection of the pixel in the bundle
: photometric error between reference and current image
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 4
Slide 4

5. Depth map estimation Reprojection in image bundle
Example reference image pixel
Typical aims of surveillance:
Detection of targets  Wide field of view beneficial
Tracking of targets, Identification of targets
 Small field of view beneficial
Collaborative sensing
Photo error
Depth hypotheses
Reprojection of depth hypotheses on one image of bundle
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 5
Slide 5

6. Depth map filtering approach
Problem:
Uniform regions in reference image do not give discriminative enough photometric error
Typical aims of surveillance:
Detection of targets  Wide field of view beneficial
Tracking of targets, Identification of targets
 Small field of view beneficial
Collaborative sensing
Idea:
Assume that depth is smooth on uniform regions
Use total variational approach where depth map is the functional to optimize:
photometric error defines the data term
the smoothness constraint defines the regularization.
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 6
Slide 6

7. Depth map filtering approach
Formulation:
First term : regularization constraint, g is defined so that it is 0 for image gradients and 1 for uniform regions. So that gradient on depth map is penalized for uniform regions
Second term : data term defined by the photometric error.
Huber norm: differentiable replacement to L1 norm that better preserve discontinuities compared to L2.
Typical aims of surveillance:
Detection of targets  Wide field of view beneficial
Tracking of targets, Identification of targets
 Small field of view beneficial
Collaborative sensing
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 7
Slide 7

8. Regularisation effect
Total variational optimisation
L2 norm
L1 norm
QU(f1)=1
QU(f2)=0.1
QU(f3)=0.01
TV(f1)=1
TV(f2)=1
TV(f3)=1
Regularisation effect
Typical aims of surveillance:
Detection of targets  Wide field of view beneficial
Tracking of targets, Identification of targets
 Small field of view beneficial
Collaborative sensing
Image denoising
[Pock08]
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 8
Slide 8

9. Depth map filtering approach
Formulation :
Problem : optimizing this equation directly requires linearising of cost volume. Expensive and cost volume has many local minima.
Typical aims of surveillance:
Detection of targets  Wide field of view beneficial
Tracking of targets, Identification of targets
 Small field of view beneficial
Collaborative sensing
Approximation :
Introduce as an auxiliary variable, can be optimized with heuristic search
Second terms brings original and auxiliary variable together
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 9
Slide 9

10. Total variational optimisation
Classical approaches:
Time Marching Scheme: steepest descent method
Linearization of the Euler-Lagrange Equation
Problem: optimization badly conditioned as (uniform regions)
Euler lagrange equation
- Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods
- Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods
Problem of previous solutions: degenerated as NablaU=0
Reformulation of regularization with primal dual method
Dual variable p is introduced to compute the TV norm:
Indeed:
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 10
Slide 10

11. Increasing solution accuracy ?
Typical aims of surveillance:
Detection of targets  Wide field of view beneficial
Tracking of targets, Identification of targets
 Small field of view beneficial
Collaborative sensing
Reminder:
Before
Approach:
Q well modeled, perform Newton step on Q to update estimation a
Equivalent to using Epsilon ?
After one iteration
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 11
Slide 11

12. Dense tracking Inputs: Tracking as a registration problem
3D texture model of the scene
Pose at previous frame
Tracking as a registration problem
First inter-frame rotation estimation : the previous image is aligned on the current image to estimate a coarse inter-frame rotation
Estimated pose is used to project the 3D model into 2.5D image
The 2.5D image is registered with the current frame to find the current pose.
Euler lagrange equation
- Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods
- Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods
Problem of previous solutions: degenerated as NablaU=0
Two template matching problems
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 12
Slide 12

13. SSD optimisation Problem: Formulation: Hypothesis:
Align template image T(x) with input image I(x).
Formulation:
find the transformation that best maps the pixels of the templates into the ones of the current image minimizing:
are the displacement parameters to be optimized.
Templates can be 2D, 2.5D, 3D as long as warp function defined to project model in 2D current image.
Hypothesis:
Know a coarse approximation of the template position (p0).
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 13
Slide 13

14. SSD optimisation Problem: Formulations: minimize
The current estimation of p is iteratively updated to reach the minimum of the function.
Formulations:
Direct additional
Direct compositional
Inverse
Euler lagrange equation
- Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods
- Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods
Problem of previous solutions: degenerated as NablaU=0
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 14
Slide 14

15. SSD optimisation Example: Direct additive method Minimize :
First order Taylor expansion:
Solution:
with:
Euler lagrange equation
- Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods
- Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods
Problem of previous solutions: degenerated as NablaU=0
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 15
Slide 15

16. SSD robustified Reminder: Method :
Problem: In case of occlusion, the occluded pixels cause the optimum of the function to be changed. The occluded pixels have to be ignored from the optimization
Euler lagrange equation
- Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods
- Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods
Problem of previous solutions: degenerated as NablaU=0
Method :
Only the pixels with a difference lower than a threshold are selected.
Threshold is iteratively updated to get more selective as the optimization reaches the optimum.
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 16
Slide 16

17. Template matching Applications to DTAM: First rotation estimation:
the template is the previous image that is matched with current image. Warp is defined on the space of all rotations. The initial estimate of p is identity.
Full pose estimation
template is 2.5D, warp is defined by full 3D motion estimation, that is
The initial pose is given by the pose estimated at the previous frame and the inter frame rotation estimation.
Euler lagrange equation
- Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods
- Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods
Problem of previous solutions: degenerated as NablaU=0
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 17
Slide 17

18. Conclusion First live full dense reconstruction system...
Limitation from the smoothness assumption on depth...
Euler lagrange equation
- Solved using Time marching scheme, similar to ;ost of approches used to solve level set methods
- Solved using linearization of Euler lagrange ie Taylor expansion, result in sparse system to resolve that is usually solved using Jacobi or Gauss-Seidel methods
Problem of previous solutions: degenerated as NablaU=0
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 18
Slide 18

19. Important references
[Pock Thesis08] Fast total variation for Computer Vision
[Baker IJCV04] Lucas-Kanade 20 years on: A unifying framework
Typical aims of surveillance:
Detection of targets  Wide field of view beneficial
Tracking of targets, Identification of targets
 Small field of view beneficial
Collaborative sensing
Amaury Dame
Active Vision Lab
Oxford Robotics Research Group 19
Slide 19