1 Chapter 21 Machine Vision

2 Chapter 21 Contents (1) l Human Vision l Image Processing l Edge Detection l Convolution and the Canny Edge Detector l Segmentation l Classifying Edges

3 Chapter 21 Contents (2) l Using Texture l Structural Texture Analysis l Determining Shape and Orientation from Texture l Interpreting Motion l Making Use of Vision l Face Recognition

4 Human Vision

5 Image Processing l Image Processing consists of the following components: nImage capture nEdge detection nSegmentation nThree dimensional segmentation nRecognition and analysis l Image capture is the process of converting a visual scene into data that can be processed.

6 Edge Detection (1) l Edge detection is the first phase in processing image data. l The following images show a photograph of a hand and the edges detected in this image.

7 Edge Detection (2) l Every edge represents some kind of discontinuity in the image. l Most edges are depth discontinuities. l Other discontinuities are: nSurface orientation discontinuities nSurface reflectance discontinuities nIllumination discontinuities (shadows)

8 Convolution and the Canny Edge Detector (1) l One method of edge detection is to differentiate the image: nDiscontinuities will have the highest differentials. l This does not work well with noisy images l Convolution is better for such images.

9 Convolution and the Canny Edge Detector (2) l The convolution of two discrete functions f(a, b) and g(a, b) is defined as follows: l The convolution of continuous functions f(a,b) and g(a,b) is defined as follows: l An image can be smoothed, to eliminate noise, by convolving it with the Gaussian function:

10 Convolution and the Canny Edge Detector (3) l The image, after smoothing, can be differentiated to detect the edges. l The peaks in the differential correspond to the edges in the original image. l In fact, the same result can be obtained by convolving the image with the differential of G:

11 Convolution and the Canny Edge Detector (4) l This method only works with one-dimensional edges. To detect two dimensional egdes we convolve with two filters, and square and add the results: l where I(x, y) is the value of the pixel at location (x, y) in the image. Filter 1 is G’ σ (x) G σ (y) Filter 2 is G’ σ (y) G σ (x) l This is the Canny edge detector.

12 Segmentation l Once the edges have been detected, this can be used to segment the image. l Segmentation involves dividing the image into areas which do not contain edges. l These areas will not have sharp changes in colour or shading. l In fact, edge detection will not always entirely segment an image. l Another method is thresholding. l Thresholding involves joining pixels together that have similar colors.

13 Classifying Edges (1) l After extracting edges, it is useful to classify the edges. nA convex edge is an edge between two faces that are at an angle of more than 180° from each other. nA concave edge is an edge between two faces that are at an angle of less than 180° from each other. nAn occluding edge is a depth discontinuity.

14 Classifying Edges (2) l The following diagram shows a line drawing that has had all its edges classified as convex (+), concave (-) or occluding (arrow):

15 Classifying Edges (3) l Most vertices represent a meeting of three faces. l There are only sixteen possible ways these trihedral vertices can be labeled:

16 Classifying Edges (4) l The Waltz algorithm uses this constraint. l This works as follows: nThe first edge that is visited is marked with all possible labels. nThen the algorithm moves onto an adjacent edge, and attempts to label it. nIf an edge cannot be labeled, the algorithm backtracks. nThus, depth-first search is applied to attempt to find a consistent labeling for the whole image.

17 Using Texture (1) l Textures, such as these, tell us a great deal about images, including: nOrientation nShape l We can also determine what the pictures on the right are showing, simply by their textures.

18 Using Texture (2) l A statistical method of determining texture is to use co-occurrence matrices. l D(m, n) is the number of pairs of pixels in our picture, P, for which: P(i, j) = m P(i + δi, j + δj) = n i and j are pixels in P, and δi and δj are small increments. l D defines how likely it is that any two pixels a particular distance apart (δi and δj) will have a particular pair of values. l The co-occurrence matrix is defined as: C = D + D T where D T is the transposition of D.

19 Structural Texture Analysis l The structural approach treats textures as being made up of individual units called texels. l In this image, each tile is a texel. l Texel analysis involves searching for repeated patterns and shapes within an image.

20 Determining Shape and Orientation from Texture (1) l These are good examples of pictures where texture helps to determine the shape and orientation. l Note that the second image, although it is a flat, two dimensional shape, looks like a sphere. l This is because this is the only sensible way for our brains to explain the texture.

21 Determining Shape and Orientation from Texture (2) l One way to determine orientation is to assume that each texel is flat. l Thus the extent of distortion of the shape of the texel will tell us what angle it is being viewed at. l Orientation involves determining slant (σ) and tilt (τ), as shown here:

22 Interpreting Motion l Detecting motion is vital in mammalian vision. l Similarly, agents that interact with the real world need to be able to interpret motion. l We are interested in two types of motion: nActual motion of other objects nApparent motion caused by the motion of the agent.

23 Interpreting Motion (1) l Detecting motion is vital in mammalian vision. l Similarly, agents that interact with the real world need to be able to interpret motion. l We are interested in two types of motion: nActual motion of other objects nApparent motion caused by the motion of the agent. l This apparent motion is known as optical flow, and the vectors that define the apparent motion are the motion field.

24 Interpreting Motion (2) l The arrows on this photo show the motion field.

25 Making Use of Vision l What purpose does machine vision really serve? l It can be used to control mobile agents or unmanned vehicles such as those sent to other planets. l Another purpose is to identify objects in the agent’s environment. nIf the agent is to interact with these objects (pick them up, sit on them, talk to them) it must be able to recognize that they are there.

26 Face Recognition l An example of a problem that humans are extremely good at solving, but computers are very bad at. l Faces must be recognized in varying lighting conditions, from different angles and distances, and with other variable elements such as facial hair, glasses, hats and natural aging. l Methods used in face recognition vary, but many involve principle component analysis: nIdentifying those features that most differentiate one face from another, and treating those as a vector which is to be compared.