1. Robotics Chapter 6 – Machine Vision Dr. Amit Goradia

2. Topics Introduction – 2 hrs Coordinate transformations – 6 hrs Forward Kinematics - 6 hrs Inverse Kinematics -6 hrs Velocity Kinematics - 2 hrs Trajectory Planning - 6 hrs Robot Dynamics (Introduction) - 2 hrs Force Control (Introduction) -1 hrs Task Planning - 6 hrs Machine Vision - 6 hrs

3. Machine Vision Objectives To recover useful information about a scene from its 2-D projections. To take images as inputs and produce other types of outputs (object shape, object contour, etc.) Geometry + Measurement + Interpretation To create a model of the real world from images.

4. Related Fields Image processing –Transformation of images into other images –Image compression, image enhancement –Useful in early stages of a machine vision system Computer graphics Pattern recognition Artificial intelligence

5. Vision System Hardware

6. Image Representation Images are represented by a 2D intensity function f (x,y) where: –x,y represent spatial coordinates –Value of f is proportional to the brightness (grey level) of the image. A digital image is represented by a matrix u(m,n) whole rows and columns represent a point in space and the element value represents the grey level of the image. Each point is referred to as a pixel.

7. Image Representation Typical grey levels stored in powers of 2 (computer representation) The indices [i, j] of pixels : integer values that specify the rows and columns in pixel values – 8-bit image can represent 256 shades of grey.

8. Sampling and Quantization Sampling –The real image is sampled at a finite number of points. –Sampling rate : image resolution how many pixels the digital image will have e.g.) 640 x 480, 320 x 240, etc. Pixel –Each image sample –At the sample point, an integer value of the image intensity Quantization –Each sample is represented with the finite word size of the computer. –How many intensity levels can be used to represent the intensity value at each sample point. –e.g.) 2 8 = 256, 2 5 = 32, etc

9. Euclidean V/s Projective Geometry Euclidean geometry describes shapes as they are –Properties of objects are unchanged by rigid motions –Preserves: length, angles, parallelism Projective Geomerty describes objects as they appear –Lengths, angles, parallelism become distorted –Provides a mathematical model for how objects appear in 3D.

10. Projective Geometry Center of projection (COP) – center of the camera lenses origin of the camera frame Direction of Projection (DOP) – viewing from infinity Planar geometric projectionsNon-planar projections

11. Classical Views

12. Orthographic Projections Multi-view orthographic projection Preserves angles 3 orthogonal views

13. Perspective Projections Vanishing Points –One point perspective –Two point perspective –Three point perspective

14. Projective Transforms in a Plane Projectivity –Mapping of points in a plane to points in a plane –3 aligned points mapped to 3 aligned points Also called –Homography –Collineation Represented by a 3x3 matrix multiplication

15. Projective Transformations Definition: A projectivity (Homography) is an invertible mapping h from P2 to itself such that three points x1,x2,x3 lie on the same line if and only if h(x1),h(x2),h(x3) do.

16. Special Projective Transformations

17. Perspective Projection Similar triangles approach to camera modeling

18. Perspective Projection Equations 3d world mapped to 2d projection in image plane Scene point Image coordinates

19. Perspective Projection Matrix Projection is a matrix multiplication using homogeneous coordinates

20. Intrinsic Camera Parameters fZ X O

21. Intrinsic Parameters: –Focal Length f –Pixel size s x, s y –Image center o x, o y –(Nonlinear radial distortion coefficients k 1, k 2 …) Calibration = Determine the intrinsic parameters of a camera

22. Importance of Intrinsic Parameters

23. Radial Distortion Models Types of distortion –Pincushion distortion –Barrel distortion –Tangential distortion Distortion models distance from center

24. Extrinsic Parameters Location of the camera origin with respect to the world frame.

25. Camera Calibration Find the intrinsic and extrinsic parameters Using calibration objects of known size and geometry Can be solved by optimization techniques.

26. Image Segmentation The purpose of image segmentation is to partition an image into meaningful regions with respect to a particular application The segmentation is based on measurements taken from the image and might be greylevel, colour, texture, depth or motion Segmentation Techniques –Thresholding –Clustering – Region based – Edge based – Model based – Watershed approach

27. Greylevel Histogram-Based Segmentation We will look at two very simple image segmentation techniques that are based on the greylevel histogram of an image –Thresholding –Clustering We will use a very simple object- background test image –We will consider a zero, low and high noise image 27

28. 28 Noise freeLow noiseHigh noise Greylevel Histogram-Based Segmentation

29. How do we characterise low noise and high noise? We can consider the histograms of our images –For the noise free image, its simply two spikes at i=100, i=150 –For the low noise image, there are two clear peaks centred on i=100, i=150 –For the high noise image, there is a single peak – two greylevel populations corresponding to object and background have merged 29 Greylevel Histogram-Based Segmentation

30. 30 Greylevel Histogram-Based Segmentation

31. Greylevel Thresholding Clear ‘valley’ present between to two peaks Picking threshold is the hard part: –Human operator decided the threshold – Use mean gray level of the image – A fixed proportion of pixels are detected ( set to 1) by the thresholding operation –Analyzing the histogram of an image

32. Greylevel Thresholding Simple algorithm –If the greylevel of pixel p <=T then pixel p is an object pixel –else Pixel p is a background pixel

33. Object and Shape Recognition Typically, recognition is applied after objects have been detected. Detection: where is the object? Recognition: what is the object? Note that, to detect an object, we already need to know what type of object it is. Recognition further refines the type.

34. Examples Hands: –Where is the hand? –What is the shape and orientation of the hand?

35. Examples Letters and numbers. –Detection: where is the number? –Recognition: what number is it?

36. Recognition Steps Training phase: –Build models of each class. Test phase: –Find the model that best matches the image. Model: a single structure (but as complicated as needed) that describes how patterns of a class are expected to be. Types of models: –Geometric moments –PCA (principle component analysis) based models –Many more…

37. Image Moments Computed on binary images. –Pixels are assumed to have values 0 or 1. Raw moments: Interpretation: –M 00 is what?

38. Image Moments Computed on binary images. –Pixels are assumed to have values 0 or 1. Raw moments: Interpretation: –M 00 is the area of the shape.

39. Image Moments Computed on binary images. –Pixels are assumed to have values 0 or 1. Raw moments: Interpretation: –M 00 is the area of the shape. –How can we express the center of the shape in terms of moments?

40. Image Moments Computed on binary images. –Pixels are assumed to have values 0 or 1. Raw moments: Interpretation: –M 00 is the area of the shape. –Defining the center of the shape using moments: [yc, xc] = [M 01 /M 00, M 10 /M 00 ].

41. Central Moments Like raw moments, but computed with respect to the centroid of the shape. What is mu 00 ?

42. Central Moments Like raw moments, but computed with respect to the centroid of the shape. What is mu 00 ? –mu 00 = the area of the shape. mu 10 = ?

43. Central Moments Like raw moments, but computed with respect to the centroid of the shape. What is mu 00 ? –mu 00 = the area of the shape. What is mu 00 ? mu 10 is the x coordinate of the centroid of the shape, in a coordinate system whose origin is that centroid. So, mu 10 = mu 01 = 0.

44. Central Moments Image 1 Image 2 Will the raw moments be equal? Will the central moments be equal?

45. Central Moments Image 1 Image 2 Will the raw moments be equal? No. Will the central moments be equal? Yes.

46. Central Moments Image 1 Image 2 Central moments are translation invariant.

47. Central Moments Image 1 Image 2 How can we make these moments translation and scale invariant?

48. Normalized Central Moments Image 1 Image 2 Normalized central moments are translation and scale invariant. For i+j >= 2: For i = 1, j = 0, or i = 0, j = 1: Just divide by shape area.

49. Hu Moments Image 1 Image 2 Translation, scale, and rotation invariant.

50. Using Moments Good for recognizing shapes that have been segmented very well. Each image can be represented as a vector of moments. E.g., a vector of 7 Hu moments. Can easily be used as part of: –Model-based recognition methods. Boosting, or PCA, can be applied on top of these vectors. –Nearest neighbor classification methods. Using Hu moments, recognition is translation, scale, and orientation-invariant.