CS440 / ECE 448 – Fall 2006 Lecture #2 CS 440 / ECE 448 Introduction to Artificial Intelligence Fall 2006 Instructor: Eyal Amir TAs: Deepak Ramachandran (head TA), Jaesik Choi

CS440 / ECE 448 – Fall 2006 Lecture #2 Edge detection Convert a 2D image into a set of curves –Extracts salient features of the scene –More compact than pixels

CS440 / ECE 448 – Fall 2006 Lecture #2 Origin of Edges Edges are caused by a variety of factors depth discontinuity surface color discontinuity illumination discontinuity surface normal discontinuity

CS440 / ECE 448 – Fall 2006 Lecture #2 Last Time: Detecting Local Features Convolution: Slide a window over the image, creating an image of dot products Detect –Line –Edge / Edgel –Corners –Direction of the above

CS440 / ECE 448 – Fall 2006 Lecture #2 Effect of  (Gaussian kernel size) Canny with original The choice of depends on desired behavior –large detects large scale edges –small detects fine features

CS440 / ECE 448 – Fall 2006 Lecture #2 Edge detection by subtraction original

CS440 / ECE 448 – Fall 2006 Lecture #2 Edge detection by subtraction smoothed (5x5 Gaussian)

CS440 / ECE 448 – Fall 2006 Lecture #2 Edge detection by subtraction smoothed – original (scaled by 4, offset +128) Why does this work? filter demo

CS440 / ECE 448 – Fall 2006 Lecture #2 Gaussian - image filter Laplacian of Gaussian Gaussian delta function

CS440 / ECE 448 – Fall 2006 Lecture #2 An edge is not a line... How can we detect lines ?

CS440 / ECE 448 – Fall 2006 Lecture #2 Finding lines in an image Option 1: –Search for the line at every possible position/orientation –What is the cost of this operation? Option 2: –Use a voting scheme: Hough transform

CS440 / ECE 448 – Fall 2006 Lecture #2 Finding lines in an image Connection between image (x,y) and Hough (m,b) spaces –A line in the image corresponds to a point in Hough space –To go from image space to Hough space: given a set of points (x,y), find all (m,b) such that y = mx + b x y m b m0m0 b0b0 image spaceHough space

CS440 / ECE 448 – Fall 2006 Lecture #2 Finding lines in an image Connection between image (x,y) and Hough (m,b) spaces –A line in the image corresponds to a point in Hough space –To go from image space to Hough space: given a set of points (x,y), find all (m,b) such that y = mx + b –What does a point (x 0, y 0 ) in the image space map to? x y m b image spaceHough space A: the solutions of b = -x 0 m + y 0 this is a line in Hough space x0x0 y0y0

CS440 / ECE 448 – Fall 2006 Lecture #2 Hough transform algorithm Typically use a different parameterization –d is the perpendicular distance from the line to the origin –  is the angle this perpendicular makes with the x axis –Why? Basic Hough transform algorithm 1.Initialize H[d,  ]=0 2.for each edge point I[x,y] in the image for  = 0 to 180 H[d,  ] += 1 3.Find the value(s) of (d,  ) where H[d,  ] is maximum 4.The detected line in the image is given by What’s the running time (measured in # votes)?

CS440 / ECE 448 – Fall 2006 Lecture #2 Extensions Extension 1: Use the image gradient 1.same 2.for each edge point I[x,y] in the image compute unique (d,  ) based on image gradient at (x,y) H[d,  ] += 1 3.same 4.same 1.What’s the running time measured in votes? 2.Extension 2 –give more votes for stronger edges Extension 3 –change the sampling of (d,  ) to give more/less resolution Extension 4 –The same procedure can be used with circles, squares, or any other shape

CS440 / ECE 448 – Fall 2006 Lecture #2 Extensions Extension 1: Use the image gradient 1.same 2.for each edge point I[x,y] in the image compute unique (d,  ) based on image gradient at (x,y) H[d,  ] += 1 3.same 4.same 1.What’s the running time measured in votes? 2.Extension 2 –give more votes for stronger edges Extension 3 –change the sampling of (d,  ) to give more/less resolution Extension 4 –The same procedure can be used with circles, squares, or any other shape

CS440 / ECE 448 – Fall 2006 Lecture #2 Hough demos Line : Circle :

CS440 / ECE 448 – Fall 2006 Lecture #2 Generalizing the H.T. The H.T. can be used even if the curve has not a simple analytic form! 1.Pick a reference point (x c,y c ) 2.For i = 1,…,n : 1.Draw segment to P i on the boundary. 2. Measure its length r i, and its orientation  i. 3.Write the coordinates of (x c,y c ) as a function of r i and  i 4.Record the gradient orientation  i at P i. 3.Build a table with the data, indexed by  i. (x c,y c ) iiii riririri PiPiPiPi iiii x c = x i + r i cos(  i ) y c = y i + r i sin(  i )

CS440 / ECE 448 – Fall 2006 Lecture #2 Generalizing the H.T. (x c,y c ) PiPiPiPi iiii riririri iiii x c = x i + r i cos(  i ) y c = y i + r i sin(  i ) Suppose, there were m different gradient orientations: (m <= n) 112 mm112 mm... (r 1 1,  1 1 ),(r 1 2,  1 2 ),…,(r 1 n1,  1 n1 ) (r 2 1,  2 1 ),(r 2 2,  1 2 ),…,(r 2 n2,  1 n2 )... (r m 1,  m 1 ),(r m 2,  m 2 ),…,(r m nm,  m nm ) jjjj rjrjrjrj jjjj H.T. table

CS440 / ECE 448 – Fall 2006 Lecture #2 Generalized H.T. Algorithm: x c = x i + r i cos(  i ) y c = y i + r i sin(  i ) Finds a rotated, scaled, and translated version of the curve: (x c,y c ) PiPiPiPi iiii Sr i iiii  PjPjPjPj jjjj Sr j jjjj  PkPkPkPk iiii Sr k kkkk  1.Form an A accumulator array of possible reference points (x c,y c ), scaling factor S and Rotation angle . 2.For each edge (x,y) in the image: 1.Compute  (x,y) 2.For each (r,  ) corresponding to  (x,y) do: 1.For each S and  : 1.x c = x i + r(  ) S cos[  (  ) +  ] 2.y c = y i + r(  ) S sin[  (  ) +  ] 3.A(x c,y c,S,  ) ++ 3.Find maxima of A.

CS440 / ECE 448 – Fall 2006 Lecture #2 H.T. Summary H.T. is a “voting” scheme –points vote for a set of parameters describing a line or curve. The more votes for a particular set – the more evidence that the corresponding curve is present in the image. Can detect MULTIPLE curves in one shot. Computational cost increases with the number of parameters describing the curve.

CS440 / ECE 448 – Fall 2006 Lecture #2

CS440 / ECE 448 – Fall 2006 Lecture #2 Edge detection What real-world position does a pixel represent?

CS440 / ECE 448 – Fall 2006 Lecture #2 Simple Segmentation: Subtraction Next: Positioning

CS440 / ECE 448 – Fall 2006 Lecture #2 Processing an Image Image: array of pixels Features: lines, shapes, invariants, colors, positions, histograms Transformation to real-world, 3D coordinates Segmentation, aggregation matching against prototype/class Localizing

CS440 / ECE 448 – Fall 2006 Lecture #2 Processing an Image Image: array of pixels Features: lines, shapes, invariants, colors, positions, histograms Transformation to real-world, 3D coordinates Localizing

CS440 / ECE 448 – Fall 2006 Lecture #2 GEOMETRIC MODELS OF IMAGE FORMATION Pinhole camera model CAMERA INTRINSIC PARAMETERS From metric to pixel coordinates 3-D EUCLIDEAN SPACE & RIGID-BODY MOTION Coordinates and coordinate frames Rigid-body motion and homogeneous coordinates Rest of Today: Vision & Localization

CS440 / ECE 448 – Fall 2006 Lecture #2 IMAGE FORMATION – Pinhole Camera Model Pinhole Frontal pinhole Z

CS440 / ECE 448 – Fall 2006 Lecture #2 3-D EUCLIDEAN SPACE - Cartesian Coordinate Frame Standard base vectors: Coordinates of a point in space:

CS440 / ECE 448 – Fall 2006 Lecture #2 3-D EUCLIDEAN SPACE - Vectors A “free” vector is defined by a pair of points : Coordinates of the vector :

CS440 / ECE 448 – Fall 2006 Lecture #2 3-D EUCLIDEAN SPACE – Inner Product and Cross Product Inner product between two vectors: Cross product between two vectors:

CS440 / ECE 448 – Fall 2006 Lecture #2 RIGID-BODY MOTION – Rotation Rotation matrix: Coordinates are related by: R is Orthogonal:

CS440 / ECE 448 – Fall 2006 Lecture #2 RIGID-BODY MOTION – Rotation and Translation Coordinates are related by: Velocities are related by:

CS440 / ECE 448 – Fall 2006 Lecture #2 RIGID-BODY MOTION – Homogeneous Coordinates 3-D coordinates are related by: Homogeneous coordinates: Homogeneous coordinates/velocities are related by:

CS440 / ECE 448 – Fall 2006 Lecture #2 IMAGE FORMATION – Pinhole Camera Model Pinhole Frontal pinhole Z

CS440 / ECE 448 – Fall 2006 Lecture #2 IMAGE FORMATION – Pinhole Camera Model 2-D coordinates Homogeneous coordinates

CS440 / ECE 448 – Fall 2006 Lecture #2 pixel coordinates Linear transformation CAMERA PARAMETERS – Pixel Coordinates metric coordinates

CS440 / ECE 448 – Fall 2006 Lecture #2 CAMERA PARAMETERS – Calibration Matrix and Camera Model Pinhole camera Pixel coordinates Calibration matrix (intrinsic parameters) Projection matrix Camera model

CS440 / ECE 448 – Fall 2006 Lecture #2 Homogeneous coordinates of a 3-D point Homogeneous coordinates of its 2-D image IMAGE FORMATION – Image of a Point Projection of a 3-D point to an image plane

CS440 / ECE 448 – Fall 2006 Lecture #2 Homogeneous representation of a 3-D line IMAGE FORMATION – Image of a Line Homogeneous representation of its 2-D image Projection of a 3-D line to an image plane

CS440 / ECE 448 – Fall 2006 Lecture # Images are all “incident” at the corresponding features in space; 2.Features in space have many types of incidence relationships; 3.Features in space have many types of metric relationships. Multiple Images

CS440 / ECE 448 – Fall 2006 Lecture #2 CAMERA PARAMETERS – Radial Distortion Nonlinear transformation along the radial direction Distortion correction: make lines straight

CS440 / ECE 448 – Fall 2006 Lecture #2 Next class: Probabilistic Knowledge

CS440 / ECE 448 – Fall 2006 Lecture #2

CS440 / ECE 448 – Fall 2006 Lecture #2 Brooks and Binford, 1981 Sullivan and Ponce, 1998 Murase and Nayar, 1992 Schmid and Mohr, 1996 Invariants (Weiss, 1988; Rothwell et al., 1992; etc.)

CS440 / ECE 448 – Fall 2006 Lecture #2 Face Camel Bug Human ??

CS440 / ECE 448 – Fall 2006 Lecture #2 EXTRA SLIDES (not included in class presentation)

CS440 / ECE 448 – Fall 2006 Lecture #2 Hough Transform for Curves The H.T. can be generalized to detect any curve that can be expressed in parametric form: –Y = f(x, a1,a2,…ap) –a1, a2, … ap are the parameters –The parameter space is p-dimensional –The accumulating array is LARGE!

CS440 / ECE 448 – Fall 2006 Lecture #2 Corners contain more edges than lines. A point on a line is hard to match. Corner detection

CS440 / ECE 448 – Fall 2006 Lecture #2 Corners contain more edges than lines. A corner is easier

CS440 / ECE 448 – Fall 2006 Lecture #2 Edge Detectors Tend to Fail at Corners

CS440 / ECE 448 – Fall 2006 Lecture #2 Finding Corners Intuition: Right at corner, gradient is ill defined. Near corner, gradient has two different values.

CS440 / ECE 448 – Fall 2006 Lecture #2 Formula for Finding Corners We look at matrix: Sum over a small region, the hypothetical corner Gradient with respect to x, times gradient with respect to y Matrix is symmetric WHY THIS?

CS440 / ECE 448 – Fall 2006 Lecture #2 First, consider case where: This means all gradients in neighborhood are: (k,0) or (0, c) or (0, 0) (or off-diagonals cancel). What is region like if:    and   and 

CS440 / ECE 448 – Fall 2006 Lecture #2 General Case: From Linear Algebra, it follows that because C is symmetric: With R a rotation matrix. So every case is like one on last slide.

CS440 / ECE 448 – Fall 2006 Lecture #2 So, to detect corners Filter image. Compute magnitude of the gradient everywhere. We construct C in a window. Use Linear Algebra to find  and  If they are both big, we have a corner.

CS440 / ECE 448 – Fall 2006 Lecture #2

CS440 / ECE 448 – Fall 2006 Lecture #2 Brooks and Binford, 1981 Sullivan and Ponce, 1998 Murase and Nayar, 1992 Schmid and Mohr, 1996 Invariants (Weiss, 1988; Rothwell et al., 1992; etc.)

CS440 / ECE 448 – Fall 2006 Lecture #2 Face Camel Bug Human ??

CS440 / ECE 448 – Fall 2006 Lecture #2 Question #2: What is an appropriate object representation for describing people, animals, chairs, boats, shoes, etc. ?? Do we really believe that local pixel signatures and their geometric/statistical relationships are sufficient? or

CS440 / ECE 448 – Fall 2006 Lecture #2 The Blum transform, 1967 Generalized cylinders Binford, 1971

CS440 / ECE 448 – Fall 2006 Lecture #2 Zhu and Yuille, 1996

CS440 / ECE 448 – Fall 2006 Lecture #2 Question #3: How do we construct object descriptions from images? = How do we segment images? = How do we compute our feature vectors?

CS440 / ECE 448 – Fall 2006 Lecture #2 Forsyth, 2000

CS440 / ECE 448 – Fall 2006 Lecture #2 Question #4: How can we formalize the object recognition process? What should the corresponding optimization process try to optimize? or

CS440 / ECE 448 – Fall 2006 Lecture #2

CS440 / ECE 448 – Fall 2006 Lecture #2

CS440 / ECE 448 – Fall 2006 Lecture #2 Origin of Edges Edges are caused by a variety of factors depth discontinuity surface color discontinuity illumination discontinuity surface normal discontinuity

CS440 / ECE 448 – Fall 2006 Lecture #2 Edge detection How can you tell that a pixel is on an edge?

CS440 / ECE 448 – Fall 2006 Lecture #2 Edge detection 1.Detection of short linear edge segments (edgels) 2.Aggregation of edgels into extended edges 3. (maybe parametric description)

CS440 / ECE 448 – Fall 2006 Lecture #2 Question #2: What is an appropriate object representation for describing people, animals, chairs, boats, shoes, etc. ?? Do we really believe that local pixel signatures and their geometric/statistical relationships are sufficient? or

CS440 / ECE 448 – Fall 2006 Lecture #2 The Blum transform, 1967 Generalized cylinders Binford, 1971

CS440 / ECE 448 – Fall 2006 Lecture #2 Zhu and Yuille, 1996

CS440 / ECE 448 – Fall 2006 Lecture #2 Question #3: How do we construct object descriptions from images? = How do we segment images? = How do we compute our feature vectors?

CS440 / ECE 448 – Fall 2006 Lecture #2 Forsyth, 2000

CS440 / ECE 448 – Fall 2006 Lecture #2 Question #4: How can we formalize the object recognition process? What should the corresponding optimization process try to optimize? or

CS440 / ECE 448 – Fall 2006 Lecture #2

CS440 / ECE 448 – Fall 2006 Lecture #2 IMAGE FORMATION – Perspective Imaging Image courtesy of C. Taylor “The Scholar of Athens,” Raphael, 1518