1. CS 558 C OMPUTER V ISION Lecture VII: Corner and Blob Detection Slides adapted from S. Lazebnik

2. O UTLINE Corner detection Why detecting features? Finding corners: basic idea and mathematics Steps of Harris corner detector Blob detection Scale selection Laplacian of Gaussian (LoG) detector Difference of Gaussian (DoG) detector Affine co-variant region

3. O UTLINE Corner detection Why detecting features? Finding corners: basic idea and mathematics Steps of Harris corner detector Blob detection Scale selection Laplacian of Gaussian (LoG) detector Difference of Gaussian (DoG) detector Affine co-variant region

4. F EATURE EXTRACTION : C ORNERS 9300 Harris Corners Pkwy, Charlotte, NC

5. O UTLINE Corner detection Why detecting features? Finding corners: basic idea and mathematics Steps of Harris corner detector Blob detection Scale selection Laplacian of Gaussian (LoG) detector Difference of Gaussian (DoG) detector Affine co-variant region

6. W HY EXTRACT FEATURES ? Motivation: panorama stitching We have two images – how do we combine them?

7. W HY EXTRACT FEATURES ? Motivation: panorama stitching We have two images – how do we combine them? Step 1: extract features Step 2: match features

8. W HY EXTRACT FEATURES ? Motivation: panorama stitching We have two images – how do we combine them? Step 1: extract features Step 2: match features Step 3: align images

9. C HARACTERISTICS OF GOOD FEATURES Repeatability The same feature can be found in several images despite geometric and photometric transformations Saliency Each feature is distinctive Compactness and efficiency Many fewer features than image pixels Locality A feature occupies a relatively small area of the image; robust to clutter and occlusion

10. A PPLICATIONS Feature points are used for: Image alignment 3D reconstruction Motion tracking Robot navigation Indexing and database retrieval Object recognition

11. O UTLINE Corner detection Why detecting features? Finding corners: basic idea and mathematics Steps of Harris corner detector Blob detection Scale selection Laplacian of Gaussian (LoG) detector Difference of Gaussian (DoG) detector Affine co-variant region

12. F INDING C ORNERS Key property: in the region around a corner, image gradient has two or more dominant directions Corners are repeatable and distinctive C.Harris and M.Stephens. "A Combined Corner and Edge Detector.“ Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988. "A Combined Corner and Edge Detector.“

13. C ORNER D ETECTION : B ASIC I DEA We should easily recognize the point by looking through a small window Shifting a window in any direction should give a large change in intensity “edge”: no change along the edge direction “corner”: significant change in all directions “flat” region: no change in all directions Source: A. Efros

14. C ORNER D ETECTION : M ATHEMATICS Change in appearance of window w(x,y) for the shift [u,v]: I(x, y) E(u, v) E(3,2) w(x, y)

15. C ORNER D ETECTION : M ATHEMATICS I(x, y) E(u, v) E(0,0) w(x, y) Change in appearance of window w(x,y) for the shift [u,v]:

16. C ORNER D ETECTION : M ATHEMATICS Intensity Shifted intensity Window function or Window function w(x,y) = Gaussian1 in window, 0 outside Source: R. Szeliski Change in appearance of window w(x,y) for the shift [u,v]:

17. C ORNER D ETECTION : M ATHEMATICS We want to find out how this function behaves for small shifts Change in appearance of window w(x,y) for the shift [u,v]: E(u, v)

18. C ORNER D ETECTION : M ATHEMATICS Local quadratic approximation of E(u,v) in the neighborhood of (0,0) is given by the second-order Taylor expansion: We want to find out how this function behaves for small shifts Change in appearance of window w(x,y) for the shift [u,v]:

19. C ORNER D ETECTION : M ATHEMATICS Second-order Taylor expansion of E(u,v) about (0,0):

20. C ORNER D ETECTION : M ATHEMATICS Second-order Taylor expansion of E(u,v) about (0,0):

21. C ORNER D ETECTION : M ATHEMATICS Second-order Taylor expansion of E(u,v) about (0,0):

22. C ORNER D ETECTION : M ATHEMATICS The quadratic approximation simplifies to where M is a second moment matrix computed from image derivatives: M

23. The surface E(u,v) is locally approximated by a quadratic form. Let’s try to understand its shape. I NTERPRETING THE SECOND MOMENT MATRIX

24. First, consider the axis-aligned case (gradients are either horizontal or vertical) If either λ is close to 0, then this is not a corner, so look for locations where both are large. I NTERPRETING THE SECOND MOMENT MATRIX

25. Consider a horizontal “slice” of E(u, v): I NTERPRETING THE SECOND MOMENT MATRIX This is the equation of an ellipse.

26. Consider a horizontal “slice” of E(u, v): I NTERPRETING THE SECOND MOMENT MATRIX This is the equation of an ellipse. The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R direction of the slowest change direction of the fastest change ( max ) -1/2 ( min ) -1/2 Diagonalization of M:

27. V ISUALIZATION OF SECOND MOMENT MATRICES

29. I NTERPRETING THE EIGENVALUES 1 2 “Corner” 1 and 2 are large, 1 ~ 2 ; E increases in all directions 1 and 2 are small; E is almost constant in all directions “Edge” 1 >> 2 “Edge” 2 >> 1 “Flat” region Classification of image points using eigenvalues of M:

30. C ORNER RESPONSE FUNCTION “Corner” R > 0 “Edge” R < 0 “Flat” region |R| small α : constant (0.04 to 0.06)

31. O UTLINE Corner detection Why detecting features? Finding corners: basic idea and mathematics Steps of Harris corner detector Blob detection Scale selection Laplacian of Gaussian (LoG) detector Difference of Gaussian (DoG) detector Affine co-variant region

32. H ARRIS DETECTOR : S TEPS 1. Compute Gaussian derivatives at each pixel 2. Compute second moment matrix M in a Gaussian window around each pixel 3. Compute corner response function R 4. Threshold R 5. Find local maxima of response function (nonmaximum suppression) C.Harris and M.Stephens. “A Combined Corner and Edge Detector.” Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988. “A Combined Corner and Edge Detector.”

33. H ARRIS D ETECTOR : S TEPS

34. Compute corner response R

35. H ARRIS D ETECTOR : S TEPS Find points with large corner response: R>threshold

36. H ARRIS D ETECTOR : S TEPS Take only the points of local maxima of R

37. H ARRIS D ETECTOR : S TEPS

38. I NVARIANCE AND COVARIANCE We want corner locations to be invariant to photometric transformations and covariant to geometric transformations Invariance: image is transformed and corner locations do not change Covariance: if we have two transformed versions of the same image, features should be detected in corresponding locations

39. A FFINE INTENSITY CHANGE Only derivatives are used => invariance to intensity shift I  I + b Intensity scaling: I  a I R x (image coordinate) threshold R x (image coordinate) Partially invariant to affine intensity change I  a I + b

40. I MAGE TRANSLATION Derivatives and window function are shift-invariant Corner location is covariant w.r.t. translation

41. I MAGE ROTATION Second moment ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner location is covariant w.r.t. rotation

42. S CALING All points will be classified as edges Corner Corner location is not covariant to scaling!

43. O UTLINE Corner detection Why detecting features? Finding corners: basic idea and mathematics Steps of Harris corner detector Blob detection Scale selection Laplacian of Gaussian (LoG) detector Difference of Gaussian (DoG) detector Affine co-variant region

44. B LOB DETECTION

45. O UTLINE Corner detection Why detecting features? Finding corners: basic idea and mathematics Steps of Harris corner detector Blob detection Scale selection Laplacian of Gaussian (LoG) detector Difference of Gaussian (DoG) detector Affine co-variant region

46. Goal: independently detect corresponding regions in scaled versions of the same image Need scale selection mechanism for finding characteristic region size that is covariant with the image transformation A CHIEVING SCALE COVARIANCE

47. R ECALL : E DGE DETECTION f Source: S. Seitz Edge Derivative of Gaussian Edge = maximum of derivative

48. E DGE DETECTION, T AKE 2 f Edge Second derivative of Gaussian (Laplacian) Edge = zero crossing of second derivative Source: S. Seitz

49. F ROM EDGES TO BLOBS Edge = ripple Blob = superposition of two ripples Spatial selection: the magnitude of the Laplacian response will achieve a maximum at the center of the blob, provided the scale of the Laplacian is “matched” to the scale of the blob maximum

50. We want to find the characteristic scale of the blob by convolving it with Laplacians at several scales and looking for the maximum response However, Laplacian response decays as scale increases: S CALE SELECTION Why does this happen? increasing σ original signal (radius=8)

51. S CALE NORMALIZATION The response of a derivative of Gaussian filter to a perfect step edge decreases as σ increases

52. S CALE NORMALIZATION The response of a derivative of Gaussian filter to a perfect step edge decreases as σ increases To keep response the same (scale-invariant), must multiply Gaussian derivative by σ Laplacian is the second Gaussian derivative, so it must be multiplied by σ 2

53. E FFECT OF SCALE NORMALIZATION Scale-normalized Laplacian response Unnormalized Laplacian response Original signal maximum

54. O UTLINE Corner detection Why detecting features? Finding corners: basic idea and mathematics Steps of Harris corner detector Blob detection Scale selection Laplacian of Gaussian (LoG) detector Difference of Gaussian (DoG) detector Affine co-variant region

55. B LOB DETECTION IN 2D Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D

56. B LOB DETECTION IN 2D Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D Scale-normalized:

57. At what scale does the Laplacian achieve a maximum response to a binary circle of radius r? S CALE SELECTION r imageLaplacian

58. At what scale does the Laplacian achieve a maximum response to a binary circle of radius r? To get maximum response, the zeros of the Laplacian have to be aligned with the circle The Laplacian is given by: Therefore, the maximum response occurs at S CALE SELECTION r image circle Laplacian 0

59. C HARACTERISTIC SCALE We define the characteristic scale of a blob as the scale that produces peak of Laplacian response in the blob center characteristic scale T. Lindeberg (1998). "Feature detection with automatic scale selection." International Journal of Computer Vision 30 (2): pp 77--116."Feature detection with automatic scale selection."

60. S CALE - SPACE BLOB DETECTOR 1. Convolve image with scale-normalized Laplacian at several scales

61. S CALE - SPACE BLOB DETECTOR : E XAMPLE

63. S CALE - SPACE BLOB DETECTOR 1. Convolve image with scale-normalized Laplacian at several scales 2. Find maxima of squared Laplacian response in scale-space

64. S CALE - SPACE BLOB DETECTOR : E XAMPLE

65. O UTLINE Corner detection Why detecting features? Finding corners: basic idea and mathematics Steps of Harris corner detector Blob detection Scale selection Laplacian of Gaussian (LoG) detector Difference of Gaussian (DoG) detector Affine co-variant region

66. Approximating the Laplacian with a difference of Gaussians: (Laplacian) (Difference of Gaussians) E FFICIENT IMPLEMENTATION

67. David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004."Distinctive image features from scale-invariant keypoints.”

68. I NVARIANCE AND COVARIANCE PROPERTIES Laplacian (blob) response is invariant w.r.t. rotation and scaling Blob location and scale is covariant w.r.t. rotation and scaling What about intensity change?

69. O UTLINE Corner detection Why detecting features? Finding corners: basic idea and mathematics Steps of Harris corner detector Blob detection Scale selection Laplacian of Gaussian (LoG) detector Difference of Gaussian (DoG) detector Affine co-variant region

70. A CHIEVING AFFINE COVARIANCE Affine transformation approximates viewpoint changes for roughly planar objects and roughly orthographic cameras

71. A CHIEVING AFFINE COVARIANCE direction of the slowest change direction of the fastest change ( max ) -1/2 ( min ) -1/2 Consider the second moment matrix of the window containing the blob: Recall: This ellipse visualizes the “characteristic shape” of the window

72. A FFINE ADAPTATION EXAMPLE Scale-invariant regions (blobs)

73. A FFINE ADAPTATION EXAMPLE Affine-adapted blobs

74. F ROM COVARIANT DETECTION TO INVARIANT DESCRIPTION Geometrically transformed versions of the same neighborhood will give rise to regions that are related by the same transformation What to do if we want to compare the appearance of these image regions? Normalization : transform these regions into same- size circles

75. Problem: There is no unique transformation from an ellipse to a unit circle We can rotate or flip a unit circle, and it still stays a unit circle A FFINE NORMALIZATION

76. To assign a unique orientation to circular image windows: Create histogram of local gradient directions in the patch Assign canonical orientation at peak of smoothed histogram E LIMINATING ROTATION AMBIGUITY 0 2 

77. F ROM COVARIANT REGIONS TO INVARIANT FEATURES Extract affine regionsNormalize regions Eliminate rotational ambiguity Compute appearance descriptors SIFT (Lowe ’04)

78. I NVARIANCE VS. COVARIANCE Invariance: features(transform(image)) = features(image) Covariance: features(transform(image)) = transform(features(image)) Covariant detection => invariant description