Motion from image and inertial measurements (additional slides) Dennis Strelow Carnegie Mellon University

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Outline Robust image feature tracking (in detail) Lucas-Kanade and real sequences The “smalls” tracker Motion from omnidirectional images

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: Lucas- Kanade and real sequences (1) Combining image and inertial measurements improves our situation, but… we still need accurate feature tracking tracking some sequences do not come with inertial measurements

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: Lucas- Kanade and real sequences (2) better feature tracking for improved 6 DOF motion estimation remaining results will be image-only

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: Lucas- Kanade and real sequences (3) Lucas-Kanade has been the go-to feature tracker for shape-from-motion minimizes a correlation-like matching error using general minimization evaluates the matching error at only a few locations subpixel resolution

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: Lucas- Kanade and real sequences (4) Additional heuristics used to apply Lucas- Kanade to shape-from-motion: task:heuristic: choose features to trackhigh image texture identify mistracked, occluded, no-longer-visible convergence, matching error handle large motionsimage pyramid

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: Lucas- Kanade and real sequences (5) But Lucas-Kanade performs poorly on many real sequences…

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: the “smalls” tracker (1) smalls is a new feature tracker targeted at 6 DOF motion estimation exploits the rigid scene assumption eliminates the heuristics normally used with Lucas-Kanade SIFT is an enabling technology here

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: the “smalls” tracker (2) First step: epipolar geometry estimation use SIFT to establish matches between the two images get the 6 DOF camera motion between the two images get the epipolar geometry relating the two images

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: the “smalls” tracker (3)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: the “smalls” tracker (4)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: the “smalls” tracker (5) Second step: track along epipolar lines use nearby SIFT matches to get initial position on epipolar line exploits the rigid scene assumption eliminates heuristic: pyramid

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: the “smalls” tracker (6) Third step: prune features geometrically inconsistent features are marked as mistracked and removed clumped features are pruned eliminates heuristic: detecting mistracked features based on convergence, error

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: the “smalls” tracker (7) Fourth step: extract new features spatial image coverage is the main criterion required texture is minimal when tracking is restricted to the epipolar lines eliminates heuristic: extracting only textured features

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: the “smalls” tracker (8)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: the “smalls” tracker (9) left: odometry onlyright: images only average error: 1.74 m maximum error: 5.14 m total distance: 230 m

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Robust image feature tracking: the “smalls” tracker (10) Recap: exploits the rigid scene and eliminates heuristics allows hands-free tracking for real sequences can still be defeated by textureless areas or repetitive texture

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Outline Robust image feature tracking (in detail) Motion from omnidirectional images

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Motion from omnidirectional images (1)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Motion from omnidirectional images (2)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Motion from omnidirectional images (3)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Motion from omnidirectional images (4)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Motion from omnidirectional images (5) left: non-rigid cameraright: rigid camera squares: ground truth points solid: image-only estimates dash-dotted: image-and-inertial estimates

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, Motion from omnidirectional images (6) In this experiment: omni images conventional images + inertial have roughly the same advantages But in general: inertial has some advantages that omni images alone can’t produce omni images can be harder to use