Miguel Lourenço, João P. Barreto, Abed Malti Institute for Systems and Robotics, Faculty of Science and Technology University of Coimbra, Portugal Feature Detection and Matching in Images with Radial Distortion

Presentation Outline SIFT Features – brief overview RD problems in keypoint detection and matching  Theoretical reasoning  Experimental validation Improvement to the SIFT algorithm to enhance it with RD Real experiments – a comparison study Motion estimation and 3D reconstruction in endoscopic images Name / Location / Date Slide 2

Motivation for keypoint detection and matching Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 3 Point correspondence across multiple views  Camera calibration  Sparse 3D reconstruction  Recover camera/robot motion  Visual Slam Representation of image content  Image retrieval applications  Recognition tasks (e.g. Voc-tree)  Image compression Partioning of the descriptor space

SIFT Features (Lowe, IJCV 2004 – 6725 citations on google scholar) Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 4 SIFT is probably the most broadly used algorithm for keypoint detection and matching How does SIFT work ?  Image salient points detected in a scale space framework Increase scale Gaussian pyramid DoG pyramid (x,y,sigma)

SIFT Features (Lowe, IJCV 2004 – 6725 citations on Scholar google) Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 5 SIFT is probably the most broadly used algorithm for keypoint detection and matching How does SIFT work ?  Image salient points detected in a scale space framework  SIFT descriptor is computed based on local image gradient on a scale and rotation normalized patch

Problem statement (1/2) Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 6 X Z O' Q What is radial distortion?  Bending of the light rays pulling image points towards the center along radial direction O Cameras with radial lens distortion are often used in computer and robotic vision applications Mini-lens Fish-eye lens Boroscope

Problem statement (2/2) Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 7 SIFT is invariant to rotation and scale but it is not invariant to RD Our Contribution: Modifications to the original SIFT for invariance to image RD Assumptions:  RD can be fairly described by the division model (Fitzgibbon, CVPR 2001)  RD is roughly known ( e.g. line stretching ) (Barreto, CVIU 2006) 336 correct matches421 correct matches

Tracking RD effects in SIFT How does RD affect the SIFT algorithm?  Study using images with artificially added distortion  Isolate the RD effect in SIFT detection and matching  Reliable ground truth Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 8 Improvement to the SIFT algorithm to handle RD issues Results on real imagery RD = 0%RD = 15% RD = 35% RD = 55%

How does RD affect keypoint detection? Repeatability of keypoint detection decreases with increasing distortion Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 9 Filtering bounds Regular DoG pyramid‘Distorted’ DoG pyramid  Small features (fine scale) tend to disappear during the blurring process  Coarse features tend to be detected at finer levels of scale  Flat regions (e.g. edges) start gain to strong gradient variations

Proposed Solution: Adaptive smoothing We can avoid the reconstruction artifacts by using an adaptive filter Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 10 Radial distortion must be removed before the Gaussian smoothing Rectification (~ 1.5 seconds in Matlab)

Standard vs Adaptive Gaussian smoothing Inherent properties of the standard Gaussian filter  Decouple the convolution mask in X and Y directions Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 11 Advantages of the Simplified Adaptive Filter  Shape only depends on the radius of the convolution window  Isotropic filter that can be decoupled for each image radius Simplification of the adaptive filter

Detection repeatability (synthetic adding of RD) Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 12 Better Repeatability results for keypoint detection Repeatability More robust to calibration errors Error in calibration Lower computational time than image rectification Computational time

How does RD affect matching? RD modifies the local structures in the image and by consequence the gradients are affected Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 13 Changes in local gradients of the image deteriorates SIFT descriptor performance Proposed solution: Compute gradients in the distorted image and perform implicit correction using the jacobian matrix of the distortion function

Matching evaluation (synthetic adding of RD) Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 14 Compressive effect adds new contributions to the descriptor that do not occur in undistorted views The matching performance can be improved by correcting image gradients before building the descriptor Implicit gradient correction outperforms explicit image rectification for distortion amounts up to 25%. Implicit gradient correctionSIFT in Rectified ImagesSIFT in RD Images

Experiments with Real Images Planar scenes for repeatability test and scenes with depth variation for motion estimation  Firewire camera with regular lens (~ 10% ) of distortion  Dragonfly camera with mini lens (~ 25% ) of distortion  Firewire camera with fish-eye lens (~ 45% ) of distortion Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 15

Planar images Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 16 10% distortion 25% distortion 45% distortion SIFT in RD ImagesSIFT in Rectified ImagesOur method 176 matches294 matches 364 matches 201 matches310 matches 401matches 112 matches253 matches 326 matches

Motion recovery / Sparse 3D reconstruction Scenes with depth variation where wrong matches are discarded using epipolar geometric constraints Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 17 10% distortion 25% distortion 45% distortion Main Scene

Experimental evaluation Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 18 Main SceneNumber of Inliers RMS rotation angle3D reconstruction Inliers Distribution

Conclusions / Future Work We proposed a set of modifications to the original SIFT algorithm (RD-SIFT) for achieving invariance to radial distortion. The additional computational overhead is minimum Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 19 RD – SIFT proved to be superior to explicit image correction  Better repeatability and retrieval performance  Less computational overhead  Increased robustness to calibration errors Future Work  Extend the approach to other keypoint detectors (e.g. MSER and SURF)  Real-time implementation using GPGPU (to make available to the community)  Get rid of calibration dependence

Name / Location / Date Slide 20

THANKS FOR COMING Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 21

Detection after explicit RD correction Correct the radial distortion via image rectification Miguel Lourenço – Anchorage, Alaska – ICRA 2010 Slide 22 Rectification (~ 1.5 seconds in Matlab) Drawbacks of this approach  Signal reconstruction introduces artifacts affecting SIFT performance Image1.5x Image (Bilinear)1.5x Image (Bicubic)