Features, Feature descriptors, Matching Jana Kosecka George Mason University

MSRI Workshop, January Computer Vision Visual Sensing Images I(x,y) – brightness patterns - image appearance depends on structure of the scene - material and reflectance properties of the objects - position and strength of light sources

MSRI Workshop, January photometric properties of the environment geometric properties of the environment What gives rise to images

MSRI Workshop, January Basic ingredients Radiance – amount of energy emitted along certain direction Iradiance – amount of energy received along certain direction BRDF – bidirectional reflectance distribution Lambertian surfaces – the appearance depends only on radiance, not on the viewing direction Image intensity for a Lambertian surface

MSRI Workshop, January Challenges

MSRI Workshop, January Image Primitives and Matching Given an image point in left image, what is the (corresponding) point in the right image, which is the projection of the same 3-D point

MSRI Workshop, January Image Primitives and Correspondence Difficulties – ambiguities, large changes of appearance, due to change of viewpoint, non-uniquess

MSRI Workshop, January Correspondence Lambertian assumption Rigid body motion Matching - Correspondence radiance

MSRI Workshop, January Translational model Affine model Transformation of the intensity values taking into account occlusions and noise Local Deformation Models

MSRI Workshop, January Matching and Correspondence Motivated by problems Reconstruction of 3D scene from multiple views Object recognition using (constellation of) features models Varieties Small base-line matching Wide base-line matching – large view point changes For now assuming Lambertian assumption – appearance of a local surface patch is independent of the viewpoint

MSRI Workshop, January Translational model RHS approximation by the first two terms of Taylor series Small baseline Brightness constancy constraint Feature Tracking and Optical Flow

MSRI Workshop, January Integrate around over image patch Solve Feature Tracking and Optical flow

MSRI Workshop, January rank(G) = 0 blank wall problem rank(G) = 1 aperture problem rank(G) = 2 enough texture – good feature candidates Conceptually: In reality: choice of threshold is involved Optical Flow, Feature Tracking

MSRI Workshop, January Affine feature tracking Intensity offset Contrast change

MSRI Workshop, January Qualitative properties of the motion fields Previous method - assumption locally constant flow Alternative regularization techniques (locally smooth flow fields, integration along contours) Optical Flow

MSRI Workshop, January Compute eigenvalues of G If smalest eigenvalue  of G is bigger than  - mark pixel as candidate feature point Alternatively feature quality function (Harris Corner Detector) Point Feature Extraction

MSRI Workshop, January Harris Corner Detector - Example

MSRI Workshop, January Feature Selection Compute Image Gradient Compute Feature Quality measure for each pixel Search for local maxima Feature Quality Function Local maxima of feature quality function

MSRI Workshop, January Feature Tracking Translational motion model Closed form solution 1. Build an image pyramid 2. Start from coarsest level 3. Estimate the displacement at the coarsest level 4. Iterate until finest level

MSRI Workshop, January Coarse to fine feature tracking 1. compute 2. warp the window in the second image by 3. update the displacement 4. go to finer level 5. At the finest level repeat for several iterations 0 2 1

MSRI Workshop, January Tracked Features

MSRI Workshop, January Wide baseline matching Point features detected by Harris Corner detector

MSRI Workshop, January Sum of squared differences Normalize cross-correlation Sum of absolute differences Region based Similarity Metric

MSRI Workshop, January NCC score for two widely separated views NCC score

MSRI Workshop, January Advanced matching techniques ( ) 1. Selected salient image locations - points, pieces of countours 2. Associate Local photometric descriptors 3. Invariance to image transformations + illumination changes NCC - is not invariant with respect to image transformation

MSRI Workshop, January Summary of the approach Very good results in the presence of occlusion and clutter local information discriminant greyvalue information robust estimation of the global relation between images for limited view point changes Solution for more general view point changes wide baseline matching (different viewpoint, scale and rotation) local invariant descriptors based on greyvalue information

MSRI Workshop, January Local descriptors Greyvalue derivatives Invariance to image rotation : differential invariants [Koenderink87]

MSRI Workshop, January Feature Detection and Matching Detection of interest points/regions Harris detector (extension to scale and affine invariance) Computation of descriptors for each point (e.g. diff. invariants, steerable filters, SIFT descriptor) Similarity of descriptors (Euclidean distance, Mahalanobis Distance)

MSRI Workshop, January Keypoint Detector and SIFT Descriptor Each image is characterized by a set of scale- invariant keypoints and their associated descriptors [D. Lowe,2000] Keypoints - extrema in DOG pyramid Descriptor – 8 bin orientation histograms computed over 4 x 4 grid overlayed over pixel neighbourhood and stacked together to form a 128 dim feature vector

MSRI Workshop, January SIFT Keypoints

MSRI Workshop, January Overview Scale invariance is not sufficient for large baseline changes State of the art on affine invariant points/regions Affine invariant interest points Application to recognition

MSRI Workshop, January Scale invariant interest points Invariant points + associated regions [Mikolajczyk & Schmid’01] multi-scale Harris points selection of points at the characteristic scale with Laplacian Courtesy of Schimd’01

MSRI Workshop, January Viewpoint changes Locally approximated by an affine transformation detected scale invariant regionprojected region Courtesy of Schimd’01

MSRI Workshop, January Affine invariant Harris points Localization & scale influence affine neighhorbood => affine invariant Harris points (Mikolajczyk & Schmid’02) Iterative estimation of these parameters 1. localization – local maximum of the Harris measure 2. scale – automatic scale selection with the Laplacian 3. affine neighborhood – normalization with second moment matrix Repeat estimation until convergence Initialization with multi-scale interest points

MSRI Workshop, January Alternative features/descriptors Affine invariant regions (Tuytelaars et al.’00) ellipses fitted to intensity maxima parallelogram formed by interest points and edges Maximally stable regions (Matas et al. BMVC’02) regions stable across large range of thresholds, connected components of thresholded image descriptors – rotationaly and affine invariant and color moments

MSRI Workshop, January Feature Matches 33 correct matches Courtesy of Schimd’01

MSRI Workshop, January Pieces of Countour/Line descriprors Select salient pieces using scale invariant detection techniques Characterize either the intensity profile along contour/or local neighbourhood with sideness information – form the descriptor Type of suitable salient regions depends of the class of objects Computational model of visual attention can guide the process of selecting salient regions

MSRI Workshop, January Additional changes of the appearance