1. Lecture 4 FEATURE DETECTION
Computer Vision
Lecture 4
FEATURE DETECTION

2. Today’s Topics
Feature Detection
Feature Description
Feature Matching

3. Questions of Interest How to find camera parameters?
Where is the camera, where is it directed at?
What is the movement of the camera?
Where are the objects located in 3D?
What are the dimensions of objects in 3D?
What is the 3D structure of a scene?
How to process stereo video?
How to detect and match image features?
How to stitch images?

4. Introduction to Feature Detection

5. Image Mozaics

6. Object Recognition

7. Image Matching

8. Image Matching using Feature Points
NASA Mars Rover images

9. Feature Points are Used for …
Image alignment (e.g., mosaics)
3D reconstruction
Motion tracking
Object recognition
Indexing and database retrieval
Robot navigation …

10. Advantages of Local Features
Locality
features are local, so robust to occlusion and clutter
Distinctiveness
can differentiate a large database of objects
Quantity
hundreds or thousands in a single image
Efficiency
real-time performance achievable
Generality
exploit different types of features in different situations

11. Point vs. Line Features

12. Processing Stages
Feature detection/extraction (look for stable/accurate features)
Feature description (aim for invariance to transformations)
Feature matching (for photo collection) or tracking (for video)

13. Feature Detection

14. Invariant Local Features
Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters
Features Descriptors

15. Choosing interest points
Where would you tell your friend to meet you?
Corners!

16. Choosing interest points
Where would you tell your friend to meet you?
Blobs!

17. Stable (Good) Features: High Accuracy in Localization

18. Stable (Good) Features
Aperture problem
Good feature
“flat” region: no change in all directions
“edge”:
no change along the edge direction
“corner”: significant change in all directions (uniqueness)

19. Autocorrelation: Indicator of “Good”ness

20. Autocorrelation Calculation
Autocorrelation can be approximated by sum-squared-difference (SSD):

21. Corner Detection: Mathematics
Change in appearance of window w(x,y)
for the shift [u,v]:
I(x, y)
E(u, v)
E(3,2)
w(x, y)

22. Corner Detection: Mathematics
Change in appearance of window w(x,y)
for the shift [u,v]:
I(x, y)
E(u, v)
E(0,0)
w(x, y)

23. Corner Detection: Mathematics
Change in appearance of window w(x,y)
for the shift [u,v]:
Intensity
Window function
Shifted intensity or
Window function w(x,y) =
Gaussian
1 in window, 0 outside
Source: R. Szeliski

24. Corner Detection: Mathematics
Change in appearance of window w(x,y)
for the shift [u,v]:
We want to find out how this function behaves for small shifts
E(u, v)

25. Corner Detection: Mathematics
Change in appearance of window w(x,y)
for the shift [u,v]:
We want to find out how this function behaves for small shifts
Local quadratic approximation of E(u,v) in the neighborhood of (0,0) is given by the second-order Taylor expansion:

26. Taylor Expansion

27. Corner Detection: Mathematics
Second-order Taylor expansion of E(u,v) about (0,0):

28. Corner Detection: Mathematics
Second-order Taylor expansion of E(u,v) about (0,0):

29. Corner Detection: Mathematics
Second-order Taylor expansion of E(u,v) about (0,0):

30. Corner Detection: Mathematics
The quadratic approximation simplifies to
where M is a second moment matrix computed from image derivatives: M

31. Corners as distinctive interest points
2 x 2 matrix of image derivatives (averaged in neighborhood of a point).
Notation:

32. Interpreting the second moment matrix
The surface E(u,v) is locally approximated by a quadratic form. Let’s try to understand its shape.

33. Interpreting the second moment matrix
First, consider the axis-aligned case (gradients are either horizontal or vertical)
If either λ is close to 0, then this is not a corner, so look for locations where both are large.

34. Interpreting the second moment matrix
Consider a horizontal “slice” of E(u, v):
This is the equation of an ellipse.

35. Interpreting the second moment matrix
Consider a horizontal “slice” of E(u, v):
This is the equation of an ellipse.
Diagonalization of M:
The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R
direction of the slowest change
direction of the fastest change
(max)-1/2
(min)-1/2

36. Interpreting the eigenvalues
Classification of image points using eigenvalues of M: 2
“Edge” 2 >> 1
“Corner” 1 and 2 are large, 1 ~ 2; E increases in all directions
1 and 2 are small; E is almost constant in all directions
“Edge” 1 >> 2
“Flat” region 1

37. Corner response function
α: constant (0.04 to 0.06)
“Edge” R < 0
“Corner” R > 0
|R| small
“Edge” R < 0
“Flat” region

38. Harris corner detector
Compute M matrix for each image window to get their cornerness scores.
Find points whose surrounding window gave large corner response (f> threshold)
Take the points of local maxima, i.e., perform non-maximum suppression
C.Harris and M.Stephens. “A Combined Corner and Edge Detector.” Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.  38

39. Harris Detector [Harris88]
1. Image derivatives Ix Iy
Second moment matrix
(optionally, blur first)
Ix2
Iy2
IxIy
2. Square of derivatives
g(IxIy)
3. Gaussian filter g(sI)
g(Ix2)
g(Iy2)
4. Cornerness function – both eigenvalues are strong
5. Non-maxima suppression
har

40. Use Sobel Operator for Gradient Computation
Gaussian
derivative of Gaussian -1 1 -2 2 1 2 -1 -2
Horizontal derivative
Vertical derivative

41. Harris Detector Example

42. f value (red = high, blue = low)

43. Threshold (f > value)

44. Find Local Maxima of f

45. Harris Features (in red)

46. Many Existing Detectors Available
Hessian & Harris [Beaudet ‘78], [Harris ‘88] Laplacian, DoG [Lindeberg ‘98], [Lowe 1999] Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01] Harris-/Hessian-Affine [Mikolajczyk & Schmid ‘04] EBR and IBR [Tuytelaars & Van Gool ‘04] MSER [Matas ‘02] Salient Regions [Kadir & Brady ‘01] Others…
K. Grauman, B. Leibe

47. Feature Description

48. Feature Descriptor
How can we extract feature descriptors that are invariant to variations (transformations) and yet still discriminative enough to establish correct correspondences? ?

49. Transformation Invariance
Find features that are invariant to transformations
geometric invariance: translation, rotation, scale
photometric invariance: brightness, exposure

50. Transformation Invariance
We would like to find the same features regardless of the transformation between images. This is called transformational invariance.
Most feature methods are designed to be invariant to
Translation, 2D rotation, scale
They can usually also handle
Limited affine transformations (some are fully affine invariant)
Limited 3D rotations (SIFT works up to about 60 degrees)
Limited illumination/contrast changes
n차원 공간이 1차식으로 나타내어지는 점대응(點對應) (x,y)→(x', y')을 말한다. 일반적으로 평면 위의 임의의 직선좌표계에 관하여         x'＝a₁x＋b₁y＋c₁         y'＝a₂x＋b₂y＋c₂ (a₁b₂－a₂b₁≠0) 라는 꼴의 1차식으로 나타내어진다. 의사변환(擬似變換)이라고도 한다. 일반적으로 n차원 공간에서 점대응 (x₁,x₂,…,xn) → (y₁,y₂…,yn) 이 1차식 yi = ai1x1 + ai2x2 + … + ainxn 인 꼴로 나타내어질 때, 이 대응을 아핀변환이라고 한다.
In geometry, an affine transformation , affine map or an affinity (from the Latin, affinis , "connected with") is a function between affine spaces which preserves points, straight lines and planes. Also, sets of parallel lines remain parallel after an affine transformation. An affine transformation does not necessarily preserve angles between lines...

51. Types of invariance
Illumination

52. Types of invariance
Illumination
Scale

53. Types of invariance
Illumination
Scale
Rotation

54. Types of invariance
Illumination
Scale
Rotation
Affine

55. Types of invariance Illumination Scale Rotation Affine
Full Perspective

56. How to Achieve Invariance
Make sure the detector is invariant
Harris is invariant to translation and rotation
Scale is trickier
A common approach is to detect features at many scales using a Gaussian pyramid (e.g., MOPS)
More sophisticated methods find “the best scale” to represent each feature (e.g., SIFT)
Design an invariant feature descriptor
A descriptor captures the information in a region around the detected feature point
The simplest descriptor: a square window of pixels
What’s this invariant to?
Let’s look at some better approaches…

57. Rotational Invariance for Feature Descriptors
Find dominant orientation of the image patch. This is given by e+, the eigenvector of M corresponding to +
Rotate the patch to horizontal direction
Intensity normalize the window by subtracting the mean, dividing by the standard deviation in the window
40 pixels
8 pixels
CSE 576: Computer Visio
Adapted from slide by Matthew Brown

58. Scale Invariance
Detecting features at the finest stable scale possible may not always be appropriate
e.g., when matching images with little high frequency detail (e.g., clouds), fine-scale features may not exist.
One solution to the problem is to extract features at a variety of scales
e.g., by performing the same operations at multiple resolutions in a pyramid and then matching features at the same level.
This kind of approach is suitable when the images being matched do not undergo large scale changes
e.g., when matching successive aerial images taken from an airplane or
stitching panoramas taken with a fixed-focal-length camera.

59. Scale Invariance for Feature Descriptors

60. Scale Invariant Detection
Key idea: find scale that gives local maximum of f
f is a local maximum in both position and scale

61. Scale Invariant Feature Transform (SIFT) Features
In order to detect the local maxima and minima of f, each sample point is compared to its eight neighbors in the current image and nine neighbors in the scale above and below
It is selected only if it is larger than all of these neighbors or smaller than all of them.
Instead of extracting features at many different scales and then matching all of them, it is more efficient to extract features that are stable in both location and scale.

62. Scale Invariant Feature Transform
Take 16x16 square window around detected feature
Compute edge orientation (angle of the gradient - 90) for each pixel
Throw out weak edges (threshold gradient magnitude)
Create histogram of surviving edge orientations 2
angle histogram
Adapted from slide by David Lowe

63. SIFT Descriptor
Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)
Compute an orientation histogram for each cell
16 cells * 8 orientations = 128 dimensional descriptor
Adapted from slide by David Lowe

64. Properties of SIFT Extraordinarily robust matching technique
Can handle changes in viewpoint
Up to about 60 degree out of plane rotation
Can handle significant changes in illumination
Sometimes even day vs. night (below)
Fast and efficient—can run in real time
Lots of code available

65. Feature Matching

66. Feature Matching
Given a feature in image I1, how to find the best match in image I2?
Define a distance function that compares two descriptors
Test all the features in I2, find the one with minimum distance

67. Feature Distance f1 f2 f2' I2 I1
SSD is commonly used to measure the difference between two features f1 and f2
Use ratio-of-distance = SSD(f1, f2) / SSD(f1, f2’) as a measure of reliability of the feature match:
f2 is best SSD match to f1 in I2
f2’ is 2nd best SSD match to f1 in I2
Ratio-of-distance is large for ambiguous matches f1 f2
f2' I2 I1

68. Evaluating Feature Matching Performance

69. Evaluating the Results
How can we measure the performance of a feature matcher? 50 75
200
feature distance

70. True/false positives The distance threshold affects performance 50 75
True positives = # of detected matches that are correct
Suppose we want to maximize these—how to choose threshold?
False positives = # of detected matches that are incorrect
Suppose we want to minimize these—how to choose threshold? 50 75
200
feature distance
false match
true match

71. Terminology True positives = # of detected matching features
False negatives= # of missed matching features
False positives = # of detected not matching features
True negatives = # of missed not matching features
Detected to have a match
Matching features
Detected to have a match

72. Recall, Precision, and F-Number
Fraction of detected true positives among all positives
fraction of relevant instances that are retrieved
Fraction of true positives among all detected as positives
fraction of retrieved instances that are relevant
Approaches to 1 only when both recall and precision approach to 1

73. True and False Positive Rates
Fraction of detected true positives among all positives
Fraction of detected false positives among all negatives

74. Evaluating the Performance of a Feature Matcher
true positive rate
0.7 1
0.1
false positive rate

75. Receiver Operating Characteristic (ROC) Curve
0.7 1
false positive rate
true positive rate
0.1
Generated by counting # current/incorrect matches, for different thresholds
Want to maximize area under the curve (AUC)
Useful for comparing different feature matching methods
For more info: