1. Lecture 10: Harris Corner Detector CS4670/5670: Computer Vision Kavita Bala

2. Announcements HW 1 will be out early next week Quiz 1 graded

3. Finding Corners Key property: in the region around a corner, image gradient has two or more dominant directions Corners are repeatable and distinctive C.Harris and M.Stephens, 1988. "A Combined Corner and Edge Detector.“ Proceedings of the 4th Alvey Vision Conference: pages 147--151. "A Combined Corner and Edge Detector.“

4. Feature extraction: Corners 9300 Harris Corners Pkwy, Charlotte, NC Slides from Rick Szeliski, Svetlana Lazebnik, and Kristin Grauman

5. Local measures of uniqueness Suppose we only consider a small window of pixels – What defines whether a feature is a good or bad candidate? Credit: S. Seitz, D. Frolova, D. Simakov

6. Corner Detection: Basic Idea We should easily recognize the point by looking through a small window Shifting a window in any direction should give a large change in intensity “edge”: no change along the edge direction “corner”: significant change in all directions “flat” region: no change in all directions Source: A. Efros

7. Consider shifting the window W by (u,v) how do the pixels in W change? compare each pixel before and after by summing up the squared differences (SSD) this defines an SSD “error” E(u,v): Harris corner detection: the math W

8. Corner Detection: Mathematics We want to find out how this function behaves for small shifts Change in appearance of window for the shift [u,v]: E(u, v)

9. Taylor Series expansion of I: If the motion (u,v) is small, then first order approximation is good Plugging this into the formula on the previous slide… Small motion assumption

10. Corner detection: the math Consider shifting the window W by (u,v) define an SSD “error” E(u,v): W

11. Corner detection: the math Consider shifting the window W by (u,v) define an SSD “error” E(u,v): W Thus, E(u,v) is locally approximated as a quadratic error function

12. The surface E(u,v) is locally approximated by a quadratic form. Let’s try to understand its shape. Interpreting the second moment matrix

13. The surface E(u,v) is locally approximated by a quadratic form. The second moment matrix Let’s try to understand its shape.

14. Horizontal edge: u v E(u,v) M M M

15. Vertical edge: u v E(u,v) M M

16. Consider a horizontal “slice” of E(u, v): Interpreting the second moment matrix This is the equation of an ellipse.

17. Quick eigenvalue/eigenvector review The eigenvectors of a matrix A are the vectors x that satisfy: The scalar is the eigenvalue corresponding to x – The eigenvalues are found by solving: – Say, A = H is a 2x2 matrix, so we have – The solution: Once you know, you find x by solving

18. Consider a horizontal “slice” of E(u, v): Interpreting the second moment matrix This is the equation of an ellipse. The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R direction of the slowest change direction of the fastest change ( max ) -1/2 ( min ) -1/2 Diagonalization of M:

19. Corner detection: the math Eigenvalues and eigenvectors of M Define shift directions with the smallest and largest change in error x max = direction of largest increase in E max = amount of increase in direction x max x min = direction of smallest increase in E min = amount of increase in direction x min x min x max M M

20. Interpreting the eigenvalues 1 2 “Corner” 1 and 2 are large, 1 ~ 2 ; E increases in all directions 1 and 2 are small; E is almost constant in all directions “Edge” 1 >> 2 “Edge” 2 >> 1 “Flat” region Classification of image points using eigenvalues of M:

21. Corner detection: the math How are max, x max, min, and x min relevant for feature detection? Need a feature scoring function

22. Corner detection: the math How are max, x max, min, and x min relevant for feature detection? Need a feature scoring function Want E(u,v) to be large for small shifts in all directions the minimum of E(u,v) should be large, over all unit vectors [u v] this minimum is given by the smaller eigenvalue ( min ) of M

23. Corner detection summary Here’s what you do Compute the gradient at each point in the image Create the M matrix from the entries in the gradient Compute the eigenvalues Find points with large response ( min > threshold) Choose those points where min is a local maximum as features

24. Corner detection summary Here’s what you do Compute the gradient at each point in the image Create the H matrix from the entries in the gradient Compute the eigenvalues. Find points with large response ( min > threshold) Choose those points where min is a local maximum as features

25. The Harris operator min is a variant of the “Harris operator” for feature detection The trace is the sum of the diagonals, i.e., trace(H) = h 11 + h 22 Very similar to min but less expensive (no square root) Called the “Harris Corner Detector” or “Harris Operator” Actually the Noble variant of the Harris Corner Detector Lots of other detectors, this is one of the most popular

26. Corner response function “Corner” R > 0 “Edge” R < 0 “Flat” region |R| small α : constant (0.04 to 0.15)

27. The Harris operator Harris operator

28. Harris Detector [ Harris88 ] Second moment matrix 28 1. Image derivatives 2. Square of derivatives 3. Gaussian filter g(  I ) IxIx IyIy Ix2Ix2 Iy2Iy2 IxIyIxIy g(I x 2 )g(I y 2 ) g(I x I y ) 4. Cornerness function – both eigenvalues are strong har 5. Non-maxima suppression (optionally, blur first)

29. Weighting the derivatives In practice, using a simple window W doesn’t work too well Instead, we’ll weight each derivative value based on its distance from the center pixel

30. Harris detector example

31. f value (red high, blue low)

32. Threshold (f > value)

33. Find local maxima of f

34. Harris features (in red)