1. CS223b, Jana Kosecka Image Features Local, meaningful, detectable parts of the image. Line detection Corner detection Motivation Information content high Invariant to change of view point, illumination Reduces computational burden Uniqueness Can be tuned to a task at hand

2. CS223b, Jana Kosecka Canne Edge Detector Edge detection involves 3 steps: –Noise smoothing –Edge enhancement –Edge localization J. Canny formalized these steps to design an optimal edge detector How to go from derivatives to edges ? Horizontal edges Before:

3. CS223b, Jana Kosecka Compute image derivatives if gradient magnitude >  and the value is a local maximum along gradient direction – pixel is an edge candidate Canny edge detector gradient magnitude original image Edge Detection

4. CS223b, Jana Kosecka Algorithm Canny Edge detector The input is image I; G is a zero mean Gaussian filter (std =  ) 1.J = I * G (smoothing) 2.For each pixel (i,j): (edge enhancement) –Compute the image gradient »  J(i,j) = (J x (i,j),J y (i,j))’ –Estimate edge strength »e s (i,j) = (J x 2 (i,j)+ J y 2 (i,j)) 1/2 –Estimate edge orientation »e o (i,j) = arctan(J x (i,j)/J y (i,j)) The output are images E s - Edge Strength - Magnitude and Edge Orientation E o -

5. CS223b, Jana Kosecka E s has large values at edges: Find local maxima … but it also may have wide ridges around the local maxima (large values around the edges)Th

6. CS223b, Jana Kosecka NONMAX_SUPRESSION The inputs are E s & E o (outputs of CANNY_ENHANCER) Consider 4 directions D={ 0,45,90,135} wrt x For each pixel (i,j) do: 1.Find the direction d  D s.t. d  E o (i,j) (normal to the edge) 2.If {E s (i,j) is smaller than at least one of its neigh. along d } I N (i,j)=0 Otherwise, I N (i,j)= E s (i,j) The output is the thinned edge image I N Edge orientation

7. CS223b, Jana Kosecka Graphical Interpretation x x

8. CS223b, Jana Kosecka Thresholding Edges are found by thresholding the output of NONMAX_SUPRESSION If the threshold is too high: –Very few (none) edges High MISDETECTIONS, many gaps If the threshold is too low: –Too many (all pixels) edges High FALSE POSITIVES, many extra edges

9. CS223b, Jana Kosecka SOLUTION: Hysteresis Thresholding Es(i,j)> H Es(i,j)<H Es(i,j)>L Es(i,j)<L Es(i,j)>L

10. CS223b, Jana Kosecka Canny Edge Detection (Example) courtesy of G. Loy gap is gone Original image Strong edges only Strong + connected weak edges Weak edges

11. CS223b, Jana Kosecka Other Edge Detectors (2 nd order derivative filters)

12. CS223b, Jana Kosecka First-order derivative filters (1D) Sharp changes in gray level of the input image correspond to “peaks” of the first-derivative of the input signal.F(x) F ’(x) x

13. CS223b, Jana Kosecka Second-order derivative filters (1D) Peaks of the first-derivative of the input signal, correspond to “zero-crossings” of the second- derivative of the input signal.F(x) F ’(x) x F’’(x)

14. CS223b, Jana Kosecka NOTE: F’’(x)=0 is not enough! –F’(x) = c has F’’(x) = 0, but there is no edge The second-derivative must change sign, -- i.e. from (+) to (-) or from (-) to (+) The sign transition depends on the intensity change of the image – i.e. from dark to bright or vice versa.

15. CS223b, Jana Kosecka Edge Detection (2D) 1D 2D I(x)I(x,y) d 2 I(x) dx 2 = 0 xy |  I(x,y)| =(I x 2 (x,y) + I y 2 (x,y)) 1/2 > Th tan  = I x (x,y)/ I y (x,y) F(x) x dI(x) dx > Th  2 I(x,y) =I x x (x,y) + I yy (x,y)=0 Laplacian

16. CS223b, Jana Kosecka Notes about the Laplacian:  2 I(x,y) is a SCALAR –  Can be found using a SINGLE mask –  Orientation information is lost  2 I(x,y) is the sum of SECOND-order derivatives  2 I(x,y) is the sum of SECOND-order derivatives –But taking derivatives increases noise –Very noise sensitive! It is always combined with a smoothing operation:It is always combined with a smoothing operation:

17. CS223b, Jana Kosecka LOG Filter First smooth (Gaussian filter), Then, find zero-crossings (Laplacian filter):  2 (I(x,y) * G(x,y)) –O(x,y) =  2 (I(x,y) * G(x,y)) Using linearity: –  2 G(x,y) * I(x,y) – O(x,y) =  2 G(x,y) * I(x,y) –This filter is called: “Laplacian of the Gaussian” (LOG)

18. CS223b, Jana Kosecka 1D Gaussian

19. CS223b, Jana Kosecka First Derivative of a Gaussian Positive Negative As a mask, it is also computing a difference (derivative)

20. CS223b, Jana Kosecka Second Derivative of a Gaussian2D “Mexican Hat”

21. CS223b, Jana Kosecka An edge is not a line... How can we detect lines ?

22. CS223b, Jana Kosecka Finding lines in an image Option 1: –Search for the line at every possible position/orientation –What is the cost of this operation? Option 2: –Use a voting scheme: Hough transform

23. CS223b, Jana Kosecka Finding lines in an image Connection between image (x,y) and Hough (m,b) spaces –A line in the image corresponds to a point in Hough space –To go from image space to Hough space: given a set of points (x,y), find all (m,b) such that y = mx + b x y m b m0m0 b0b0 image spaceHough space

24. CS223b, Jana Kosecka Finding lines in an image Connection between image (x,y) and Hough (m,b) spaces –A line in the image corresponds to a point in Hough space –To go from image space to Hough space: given a set of points (x,y), find all (m,b) such that y = mx + b –What does a point (x 0, y 0 ) in the image space map to? x y m b image spaceHough space A: the solutions of b = -x 0 m + y 0 this is a line in Hough space x0x0 y0y0

25. CS223b, Jana Kosecka Hough transform algorithm Typically use a different parameterization –d is the perpendicular distance from the line to the origin –  is the angle this perpendicular makes with the x axis –Why? Idea – keep an accumulator array (Hough space) and let each edge pixel contribute to it Line candidates are the maxima in the accumulator array

26. CS223b, Jana Kosecka Typical Hough Transform Basic Hough transform algorithm 1. Initialize H[d, q]=0 2. For each edge point I[x,y] in the image 3. For q = 0 to 180 H[d, q] += 1 where point is now a sinusoid in Hough space Find the value(s) of (d, q) where H[d, q] is maximum The detected line in the image is given b What’s the running time (measured in # votes)?

27. CS223b, Jana Kosecka Radon transform

28. CS223b, Jana Kosecka Hough Transform for Curves The H.T. can be generalized to detect any curve that can be expressed in parametric form: –Y = f(x, a1,a2,…ap) –a1, a2, … ap are the parameters –The parameter space is p-dimensional –The accumulating array is LARGE!

29. CS223b, Jana Kosecka H.T. Summary H.T. is a “voting” scheme –points vote for a set of parameters describing a line or curve. The more votes for a particular set – the more evidence that the corresponding curve is present in the image. Can detect MULTIPLE curves in one shot. Computational cost increases with the number of parameters describing the curve.

30. CS223b, Jana Kosecka   x y Edge detection, non-maximum suppression (traditionally Hough Transform – issues of resolution, threshold selection and search for peaks in Hough space) Connected components on edge pixels with similar orientation - group pixels with common orientation Non-max suppressed gradient magnitude Line fitting

31. CS223b, Jana Kosecka Line fitting lines determined from eigenvalues and eigenvectors of A Candidate line segments - associated line quality second moment matrix associated with each connected component v 1 - eigenvector of A Line Fitting

32. CS223b, Jana Kosecka Corners contain more edges than lines. A point on a line is hard to match. Corner detection

33. CS223b, Jana Kosecka Finding Corners Intuition: Right at corner, gradient is ill defined. Near corner, gradient has two different values.

34. CS223b, Jana Kosecka Formula for Finding Corners We look at matrix: Sum over a small region, the hypothetical corner Gradient with respect to x, times gradient with respect to y Matrix is symmetric

35. CS223b, Jana Kosecka First, consider case where: This means all gradients in neighborhood are: (k,0) or (0, c) or (0, 0) (or off-diagonals cancel). What is region like if: 1.l1 = 0? 2.l2 = 0? 3.l1 = 0 and l2 = 0? 4.l1 > 0 and l2 > 0?

36. CS223b, Jana Kosecka General Case: From Linear Algebra, it follows that because C is symmetric: With R a rotation matrix. So every case is like one on last slide.

37. CS223b, Jana Kosecka So, to detect corners Filter image. Compute magnitude of the gradient everywhere. We construct C in a window. Use Linear Algebra to find  and  If they are both big, we have a corner.

38. CS223b, Jana Kosecka Compute eigenvalues of G If smalest eigenvalue  of G is bigger than  - mark pixel as candidate feature point Alternatively feature quality function (Harris Corner Detector) Point Feature Extraction

39. CS223b, Jana Kosecka % Harris Corner detector - by Kashif Shahzad sigma=2; thresh=0.1; sze=11; disp=0; % Derivative masks dy = [-1 0 1; -1 0 1; -1 0 1]; dx = dy'; %dx is the transpose matrix of dy % Ix and Iy are the horizontal and vertical edges of image Ix = conv2(bw, dx, 'same'); Iy = conv2(bw, dy, 'same'); % Calculating the gradient of the image Ix and Iy g = fspecial('gaussian',max(1,fix(6*sigma)), sigma); Ix2 = conv2(Ix.^2, g, 'same'); % Smoothed squared image derivatives Iy2 = conv2(Iy.^2, g, 'same'); Ixy = conv2(Ix.*Iy, g, 'same'); % My preferred measure according to research paper cornerness = (Ix2.*Iy2 - Ixy.^2)./(Ix2 + Iy2 + eps); % We should perform nonmaximal suppression and threshold mx = ordfilt2(cornerness,sze^2,ones(sze)); % Grey-scale dilate cornerness = (cornerness==mx)&(cornerness>thresh); % Find maxima [rws,cols] = find(cornerness); clf ; imshow(bw); hold on; p=[cols rws]; plot(p(:,1),p(:,2),'or'); title('\bf Harris Corners')

40. CS223b, Jana Kosecka Example (  =0.1)

41. CS223b, Jana Kosecka Example (  =0.01)

42. CS223b, Jana Kosecka Example (  =0.001)

43. CS223b, Jana Kosecka Harris Corner Detector - Example