1. Recognition and Matching based on local invariant features
Cordelia Schmid
INRIA, Grenoble
David Lowe
Univ. of British Columbia

2. ( ) Introduction Local invariant photometric descriptors
local descriptor
Local : robust to occlusion/clutter + no segmentation
Photometric : distinctive
Invariant : to image transformations + illumination changes

3. History - Matching Matching based on line segments
Not very discriminant
Solution : matching with interest points & correlation
[ A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry,
Z. Zhang, R. Deriche, O. Faugeras and Q. Luong,
Artificial Intelligence 1995 ]

4. Approach Extraction of interest points with the Harris detector
Comparison of points with cross-correlation
Verification with the fundamental matrix

5. Interest points extracted with Harris (~ 500 points)
Harris detector
Interest points extracted with Harris (~ 500 points)

6. Cross-correlation matching
Initial matches (188 pairs)

7. Robust estimation of the fundamental matrix
Global constraints
Robust estimation of the fundamental matrix
99 inliers
89 outliers

8. Summary of the approach
Very good results in the presence of occlusion and clutter
local information
discriminant greyvalue information
robust estimation of the global relation between images
for limited view point changes
Solution for more general view point changes
wide baseline matching (different viewpoint, scale and rotation)
local invariant descriptors based on greyvalue information

9. History - Recognition Color histogram [Swain 91]
Each pixel is described
by a color vector
Distribution of color vectors
is described by a histogram
=> not robust to occlusion, not invariant, not distinctive

10. . . . . History - Recognition Eigenimages [Turk 91]
Each face vector is represented in the eigenimage space
eigenvectors with the highest eigenvalues = eigenimages
The new image is projected into the eigenimage space
determine the closest face . . . .
not robust to occlusion, requires segmentation, not invariant,
discriminant

11. History - Recognition Geometric invariants [Rothwell 92]
Function with a value independent of the transformation
Invariant for image rotation : distance of two points
Invariant for planar homography : cross-ratio
where
=> local and invariant, not discriminant, requires sub-pixel extraction of primitives

12. History - Recognition
Problems : occlusion, clutter, image transformations, distinctiveness
Solution : recognition with local photometric invariants
[ Local greyvalue invariants for image retrieval,
C. Schmid and R. Mohr,
PAMI 1997 ]

13. Approach
( )
local descriptor
1) Extraction of interest points (characteristic locations)
2) Computation of local descriptors
3) Determining correspondences
4) Selection of similar images

14. Interest points Geometric features 2D characteristics of the signal
repeatable under transformations
2D characteristics of the signal
high informational content
Comparison of different detectors [Schmid98]
Harris detector

15. Harris detector Based on the idea of auto-correlation
Important difference in all directions => interest point

16. Harris detector Auto-correlation function for a point and a shift
Discret shifts can be avoided with the auto-correlation matrix

17. Auto-correlation matrix
Harris detector
Auto-correlation matrix

18. Harris detection Auto-correlation matrix Interest point detection
captures the structure of the local neighborhood
measure based on eigenvalues of this matrix
2 strong eigenvalues => interest point
1 strong eigenvalue => contour
0 eigenvalue => uniform region
Interest point detection
threshold on the eigenvalues
local maximum for localization

19. Local descriptors
( )
local descriptor
Descriptors characterize the local neighborhood of a point

20. Greyvalue derivatives
Local descriptors
Greyvalue derivatives

21. Local descriptors
Invariance to image rotation : differential invariants [Koen87]

22. Local descriptors Robustness to illumination changes
In case of an affine transformation

23. or normalization of the image patch with mean and variance
Local descriptors
Robustness to illumination changes
In case of an affine transformation
or normalization of the image patch with mean and variance

24. Determining correspondences
( ) ?
( ) =
Vector comparison using the Mahalanobis distance

25. Selection of similar images
In a large database
voting algorithm
additional constraints
Rapid acces with an indexing mechanism

26. Voting algorithm
local characteristics
vector of
( )

27. Voting algorithm 2 1 1 } }
I is the corresponding model image 1 2

28. Additional constraints
Semi-local constraints
neighboring points should match
angles, length ratios should be similar
Global constraints
robust estimation of the image transformation (homogaphy, epipolar geometry) 1 1 2 2 3 3

29. Results
database with ~1000 images

30. Results

31. Results

32. Summary of the approach
Very good results in the presence of occlusion and clutter
local information
discriminant greyvalue information
invariance to image rotation and illumination
Not invariance to scale and affine changes
Solution for more general view point changes
local invariant descriptors to scale and rotation
extraction of invariant points and regions

33. Approach for Matching and Recognition
Detection of interest points/regions
Harris detector (extension to scale and affine invariance)
Blob detector based on Laplacian
Computation of descriptors for each point
Similarity of descriptors
Semi-local constraints
Global verification

34. Approach for Matching and Recognition
Detection of interest points/regions
Computation of descriptors for each point
greyvalue patch, diff. invariants, steerable filter, SIFT descriptor
Similarity of descriptors
correlation, Mahalanobis distance, Euclidean distance
Semi-local constraints
Global verification

35. Approach for Matching and Recognition
Detection of interest points/regions
Computation of descriptors for each point
Similarity of descriptors
Semi-local constraints
geometrical or statistical relations between neighborhood points
Global verification
robust estimation of geometry between images

36. Overview 8:30-8:45 Scale invariant interest points
8:45-9:00 SIFT descriptors
9:00-9:25 Affine invariance of interest points + applications
9:25-9:45 Evaluation of interest points + descriptors
9:45-10:15 Break

37. Overview 10:15-11:15 Object recognition system, demo, applications
11:15-11:45 Recognition of textures and object classes
11:45-12:00 Future directions + discussion