1. Feature Detection

2. Image features Global features –global properties of an image, including intensity histogram, frequency domain descriptors, covariance matrix and high order statistics, etc. Local features –local regions with special properties, including edges, corners, lines, curves, regions with special properties, etc Depending on applications, various features are useful. We will focus on edges and corners in this lecture

3. Edges Edge points are pixels at or around which the image values undergo a sharp variation – pixels with large gradient

18. Canny hysteresis thresholding Goal: find local maximums that are true edges Assumption –true edges should have large strength in general –pixels belong to true edges are connected to each other Problem: some edge pixels has lower values than false edge pixels Solution: hysteresis algorithm – a pixel belongs to a true edge if its edge strength is larger than a threshold t and is linked to some points with edge strength larger than t

20. Fitting Lines to Edges (Least Squares) Given: Many pairs Find: Parameters Minimize: Average square distance: Using: Note:

21. Problem with Parameterization Line that minimizes E!! Solution: Use a different parameterization (same as the one we used in computing Minimum Moment of Inertia) Note: Error E must be formulated carefully!

22. Computer Vision - A Modern Approach Set: Fitting Slides by D.A. Forsyth Line fitting can be max. likelihood - but choice of model is important

23. Curve Fitting Find Polynomial: that best fits the given points Minimize: Using: Note:is LINEAR in the parameters (a, b, c, d)

24. Line Grouping Problem Slide credit: David Jacobs

25. This is difficult because of: Extraneous data: clutter or multiple models –We do not know what is part of the model? –Can we pull out models with a few parts from much larger amounts of background clutter? Missing data: only some parts of model are present Noise Cost: –It is not feasible to check all combinations of features by fitting a model to each possible subset

26. Hough Transform Elegant method for direct object recognition Edges need not be connected Complete object need not be visible Key Idea: Edges VOTE for the possible model

27. Image and Parameter Spaces Equation of Line: Find: Consider point: Image Space Parameter Space Parameter space also called Hough Space

28. Line Detection by Hough Transform Parameter Space 11 11 11 2 11 11 11 Algorithm: Quantize Parameter Space Create Accumulator Array Set For each image edge increment: If lies on the line: Find local maxima in

29. Better Parameterization NOTE: Large Accumulator More memory and computations Improvement: Line equation: Here Given points find (Finite Accumulator Array Size) Image Space Hough Space ? Hough Space Sinusoid

30. Image space Votes Horizontal axis is θ, vertical is rho.

31. Image space votes

33. Mechanics of the Hough transform Difficulties –how big should the cells be? (too big, and we merge quite different lines; too small, and noise causes lines to be missed) How many lines? –Count the peaks in the Hough array –Treat adjacent peaks as a single peak Which points belong to each line? –Search for points close to the line –Solve again for line and iterate

34. Fewer votes land in a single bin when noise increases.

35. Adding more clutter increases number of bins with false peaks.

36. Real World Example Original Edge Detection Found Lines Parameter Space

37. Finding Circles by Hough Transform Equation of Circle: If radius is known: Accumulator Array (2D Hough Space)

38. Finding Circles by Hough Transform Equation of Circle: If radius is not known: 3D Hough Space! Use Accumulator array What is the surface in the hough space?

39. Using Gradient Information Gradient information can save lot of computation: Edge Location Edge Direction Need to increment only one point in Accumulator!! Assume radius is known:

40. Real World Circle Examples Crosshair indicates results of Hough transform, bounding box found via motion differencing.

41. Finding Coins Original Edges (note noise)

42. Finding Coins (Continued) Penn y Quarters

43. Finding Coins (Continued) Coin finding sample images from: Vivek Kwatra Note that because the quarters and penny are different sizes, a different Hough transform (with separate accumulators) was used for each circle size.

44. Generalized Hough Transform Model Shape NOT described by equation

45. Generalized Hough Transform Model Shape NOT described by equation

46. Generalized Hough Transform Find Object Center given edges Create Accumulator Array Initialize: For each edge point For each entry in table, compute: Increment Accumulator: Find Local Maxima in

48. Hough Transform: Comments Works on Disconnected Edges Relatively insensitive to occlusion Effective for simple shapes (lines, circles, etc) Trade-off between work in Image Space and Parameter Space Handling inaccurate edge locations: Increment Patch in Accumulator rather than a single point

49. Corner features Sources: intersection of image lines, corner patterns in the images, etc Stable across sequence of images

50. An introductory example: Harris corner detector C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988

51. The Basic Idea We should easily recognize the point by looking through a small window Shifting a window in any direction should give a large change in intensity

52. Harris Detector: Basic Idea “flat” region: no change in all directions “edge”: no change along the edge direction “corner”: significant change in all directions

53. Harris Detector: Mathematics Window-averaged change of intensity for the shift [u,v]: Intensity Shifted intensity Window function or Window function w(x,y) = Gaussian1 in window, 0 outside

54. Harris Detector: Mathematics Expanding E(u,v) in a 2 nd order Taylor series expansion, we have,for small shifts [u,v], a bilinear approximation: where M is a 2  2 matrix computed from image derivatives:

55. Harris Detector: Mathematics Intensity change in shifting window: eigenvalue analysis 1, 2 – eigenvalues of M direction of the slowest change direction of the fastest change ( max ) -1/2 ( min ) -1/2 Ellipse E(u,v) = const

56. Selecting Good Features 1 and 2 are large

57. Selecting Good Features large  1, small 2

58. Selecting Good Features small  1, small 2

59. Harris Detector: Mathematics 1 2 “Corner” 1 and 2 are large, 1 ~ 2 ; E increases in all directions 1 and 2 are small; E is almost constant in all directions “Edge” 1 >> 2 “Edge” 2 >> 1 “Flat” region Classification of image points using eigenvalues of M:

60. Harris Detector: Mathematics Measure of corner response: (k – empirical constant, k = 0.04-0.06)

61. Harris Detector: Mathematics 1 2 “Corner” “Edge” “Flat” R depends only on eigenvalues of M R is large for a corner R is negative with large magnitude for an edge |R| is small for a flat region R > 0 R < 0 |R| small

62. Harris Detector The Algorithm: –Find points with large corner response function R (R > threshold) –Take the points of local maxima of R

63. Harris Detector: Workflow

64. Compute corner response R

65. Harris Detector: Workflow Find points with large corner response: R>threshold

66. Harris Detector: Workflow Take only the points of local maxima of R

67. Harris Detector: Workflow

68. Harris Detector: Summary Average intensity change in direction [u,v] can be expressed as a bilinear form: Describe a point in terms of eigenvalues of M : measure of corner response A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive

69. Harris Detector: Some Properties Rotation invariance?

70. Harris Detector: Some Properties Rotation invariance Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response R is invariant to image rotation

71. Harris Detector: Some Properties Invariance to image intensity change?

72. Harris Detector: Some Properties Partial invariance to additive and multiplicative intensity changes Only derivatives are used => invariance to intensity shift I  I + b Intensity scale: I  a I R x (image coordinate) threshold R x (image coordinate)

73. Harris Detector: Some Properties Invariant to image scale?

74. Harris Detector: Some Properties Not invariant to image scale! All points will be classified as edges Corner !

75. We want to: detect the same interest points regardless of image changes

76. Models of Image Change Geometry –Rotation –Similarity (rotation + uniform scale) –Affine (scale dependent on direction) valid for: orthographic camera, locally planar object Photometry –Affine intensity change (I  a I + b)

77. Rotation Invariant Detection Harris Corner Detector C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000

78. Scale Invariant Detection Consider regions (e.g. circles) of different sizes around a point Regions of corresponding sizes will look the same in both images

79. Scale Invariant Detection The problem: how do we choose corresponding circles independently in each image?

80. Scale Invariant Detection Solution: –Design a function on the region (circle), which is “scale invariant” (the same for corresponding regions, even if they are at different scales) Example: average intensity. For corresponding regions (even of different sizes) it will be the same. scale = 1/2 –For a point in one image, we can consider it as a function of region size (circle radius) f region size Image 1 f region size Image 2

81. Scale Invariant Detection Common approach: scale = 1/2 f region size Image 1 f region size Image 2 Take a local maximum of this function Observation : region size, for which the maximum is achieved, should be invariant to image scale. s1s1 s2s2 Important: this scale invariant region size is found in each image independently!

82. Scale Invariant Detection A “good” function for scale detection: has one stable sharp peak f region size ba d f region size bad f region size Good ! For usual images: a good function would be a one which responds to contrast (sharp local intensity change)

83. Scale Invariant Detection Functions for determining scale Kernels: where Gaussian Note: both kernels are invariant to scale and rotation (Laplacian) (Difference of Gaussians)

89. Scale Invariant Detection: Summary Given: two images of the same scene with a large scale difference between them Goal: find the same interest points independently in each image Solution: search for maxima of suitable functions in scale and in space (over the image) Methods: 1.Harris-Laplacian [Mikolajczyk, Schmid]: maximize Laplacian over scale, Harris’ measure of corner response over the image 2.SIFT [Lowe]: maximize Difference of Gaussians over scale and space

90. Point Descriptors We know how to detect points Next question: How to match them? ? Point descriptor should be: 1.Invariant 2.Distinctive

91. SIFT – Scale Invariant Feature Transform 1 Empirically found 2 to show very good performance, invariant to image rotation, scale, intensity change, and to moderate affine transformations 1 D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV 2004 2 K.Mikolajczyk, C.Schmid. “A Performance Evaluation of Local Descriptors”. CVPR 2003 Scale = 2.5 Rotation = 45 0

92. CVPR 2003 Tutorial Recognition and Matching Based on Local Invariant Features David Lowe Computer Science Department University of British Columbia

93. Invariant Local Features Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters SIFT Features

94. Advantages of invariant local features Locality: features are local, so robust to occlusion and clutter (no prior segmentation) Distinctiveness: individual features can be matched to a large database of objects Quantity: many features can be generated for even small objects Efficiency: close to real-time performance Extensibility: can easily be extended to wide range of differing feature types, with each adding robustness

95. Scale invariance Requires a method to repeatably select points in location and scale: The only reasonable scale-space kernel is a Gaussian (Koenderink, 1984; Lindeberg, 1994) An efficient choice is to detect peaks in the difference of Gaussian pyramid (Burt & Adelson, 1983; Crowley & Parker, 1984 – but examining more scales) Difference-of-Gaussian with constant ratio of scales is a close approximation to Lindeberg’s scale- normalized Laplacian (can be shown from the heat diffusion equation)

96. Scale space processed one octave at a time

97. Key point localization Detect maxima and minima of difference-of-Gaussian in scale space Fit a quadratic to surrounding values for sub-pixel and sub- scale interpolation (Brown & Lowe, 2002) Taylor expansion around point: Offset of extremum (use finite differences for derivatives):

98. Select canonical orientation Create histogram of local gradient directions computed at selected scale Assign canonical orientation at peak of smoothed histogram Each key specifies stable 2D coordinates (x, y, scale, orientation)

99. Example of keypoint detection Threshold on value at DOG peak and on ratio of principle curvatures (Harris approach) (a) 233x189 image (b) 832 DOG extrema (c) 729 left after peak value threshold (d) 536 left after testing ratio of principle curvatures

100. SIFT vector formation Thresholded image gradients are sampled over 16x16 array of locations in scale space Create array of orientation histograms 8 orientations x 4x4 histogram array = 128 dimensions

101. Feature stability to noise Match features after random change in image scale & orientation, with differing levels of image noise Find nearest neighbor in database of 30,000 features

102. Feature stability to affine change Match features after random change in image scale & orientation, with 2% image noise, and affine distortion Find nearest neighbor in database of 30,000 features

103. Distinctiveness of features Vary size of database of features, with 30 degree affine change, 2% image noise Measure % correct for single nearest neighbor match

106. A good SIFT features tutorial http://www.cs.toronto.edu/~jepson/csc2503/tutSIFT04. pdf By Estrada, Jepson, and Fleet.

107. Other References Harris & Stephens (1988) http://www.bmva.org/bmvc/1988/avc-88-023.pdf http://www.bmva.org/bmvc/1988/avc-88-023.pdf Lowe (2004) http://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf http://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf Mikolajczyk & Schmid (2005) http://lear.inrialpes.fr/pubs/2005/MS05/mikolajcz yk_pami05.pdf http://lear.inrialpes.fr/pubs/2005/MS05/mikolajcz yk_pami05.pdf Brown & Lowe (2007) http://cvlab.epfl.ch/~brown/papers/ijcv2007.pdf http://cvlab.epfl.ch/~brown/papers/ijcv2007.pdf