1. Machinen Vision and Dig. Image Analysis 1 Prof. Heikki Kälviäinen CT50A6100 Lectures 8&9: Image Segmentation Professor Heikki Kälviäinen Machine Vision and Pattern Recognition Laboratory Department of Information Technology Faculty of Technology Management Lappeenranta University of Technology (LUT) Heikki.Kalviainen@lut.fi http://www.lut.fi/~kalviai http://www.it.lut.fi/ip/research/mvpr/

2. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 2 Content Motivation. Detection of discontinuities. Edge linking and boundary detection. –Local processing. –Global processing: Hough Transform. Thresholding. Region-oriented segmentation. –Region growing. –Region splitting and merging. Motion-based segmentation.

3. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 3 Segmentation: Motivation To extract important details in an image for building a feature vector = feature extraction. Feature selection => feature extraction => feature vector. Challenging question: What details?

4. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 4 Detection of discontinuities Point detection. Line detection. Edge detection. –Basic formulation. –Gradient operators. –Laplace operators. Corner detection. Combined detection.

5. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 5 Point detection Detection of isolated points: a) Apply the mask -1 -1 -1 -1 8 -1 -1 -1 -1 b) Threshold by using abs(R) > T where R is ∑w_i z_i (w_i mask, z_i point at location i)

6. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 6 Line detection Directed filters (lines of one pixel thick): Horizontal +45 ° -1 -1 -1 -1 -1 2 2 2 2-1 2 -1 -1 -1 -1 2 -1 -1 Vertical -45 ° -1 2 -1 2 -1 -1 -1 2 -1-1 2 -1 -1 2 -1-1 -1 2

7. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 7 Edge detection Edge = The boundary between two regions with relatively distinct gray-level properties. First-order derivates (Gradient operators). Second-order derivates (Laplace operators). Approaches with several techniques (e.g., Canny edge detector).

8. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 8 Edge detection: different kind of images

9. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 9 Edge detection: different kind of images (cont.) Images where almost each pixel belongs to edges.

10. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 10 Edge detection: Gradient operators Gradient of f(x,y): G_x ∂f/∂x df == G_y ∂f/∂y Magnitude: mag(df)= ((G_x)^2 + (G_y)^2)^(-1/2) approximation df = abs(G_x) + abs (G_y) Direction: α(x,y) = tan^{-1} (G_y/G_x)

11. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 11 Gradient operators Roberts:1 00 1 0 -11 0 Prewitt: -1 -1 -1-1 0 1 0 0 0-1 0 1 1 1 1-1 0 1 Sobel: -1 -2 -1-1 0 1 0 0 0-2 0 2 1 2 1-1 0 1

12. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 12 Edge detection: 3x3 Prewitt operator Horizontal mask (left top). Vertical mask (left bottom). Sum of the two masks (right middle).

13. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 13 Edge detection: Sobel and Prewitt operators Sobel operatorPrewitt operator

14. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 14 Laplace operators Laplacian of a 2-D function (second derivate operator): d^2 f = ∂^2f/∂x^2 + ∂f^2/∂y^2 3 x 3 mask: d^2 f = 4 z_5 – (z_2 + z_4 + z_6 + z_8) 0 -1 -0 -1 4 -1 0 -1 0

15. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 15 Laplace operators (cont.) Sensitive to noise. Finding the location of edges using its zero-crossing property. Convolve an image with the Laplacian of 2-D Gaussian function of the form h(x,y) = exp(-(x^2+y^2)/2σ^2) where σ is the standard deviation. Let r^2 = x^2 + y^2: d^2 h = (r^2-σ^2)/σ^4 exp(-(r^2)/2σ^2) zero-crossings at r = ±σ

16. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 16 Laplacian of Gaussian: Marr edge detection Convolution with a filter h(r) of ←  =1.5  =5.0 → zero-crossings

17. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 17 Edge detection: comparisons ← Sobel Prewitt → ← Roberts Laplacian of Gaussian →

18. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 18 Canny edge detector Smoothing the image and eliminating any noise. –Gaussian mask => a more blurred image. Finding the edge strength. –Gradient operators. Defining the edge direction. –Gradient operators. Thinning by nonmaximum suppression. –Suppress the pixels by a mask => thinner lines. Hysteresis thresholding. –Two thresholds for eliminating breaking of an edge contour => less streaking.

19. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 19 Harris corner detector Corners are usually very suitable interest points. As the intersection of two edges or as the region of two different strong edge orientations. Based on the local auto-correlation function of a signal C(x,y) which measures the local changes of the signal with patches shifted by a small amount in different directions. Corner response H(x) defined as a function of C(x,y). Interest points can be found as local maxima in the corner response.

20. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 20 Edge linking and boundary detection From intensity discontinuities to more general segmentation. –For example, from edge pixels to line segments. Local processing. –Analysis of a small neighborhood: Strength and direction of the gradient of edge pixels. Global processing: –Analysis of the whole image: Global relationships between pixels. –Hough Transform.

21. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 21 Thresholding: Foundation For example, to separate an object (objects) and background. One threshold: –Object point: f(x,y) > T. Two thresholds: –Object class1: T_1 < f(x,y) < T_2. –Object class2: f(x,y) > T_2. –Background: f(x,y) ≤ T_1.

22. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 22 Thresholding: Foundation (cont.) Operation. T = T[x, y, p(x,y), f(x,y)] where f(x,y) is the gray-level point of (x,y) p(x,y) denotes some local property of this point (e.g., the average gray-level of a neighborhood centered on (x,y)) A thresholded image is g(x,y) is defined as g(x,y) = 1 if f(x,y) > T 0 if f(x,y) ≤ T.

23. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 23 Thresholding: Role of illumination An image f(x,y) is a product of a reflectance component r(x,y) and an illumination component i(x,y). Original image => suitable histogram to thresholding Original image + illumination changes => unsuitable histogram to thresholding Ready for illumination changes?

24. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 24 Thresholding: Global and optimal How: –Simple global thresholding: just select T manually. –Optimal thresholding: optimally by probability densities (Gaussian) p(z) = exp(-(z-μ)^2 / (2σ^2)) / (√(2πσ)) where μ is the mean value and σ is the standard deviation Mixture probability density of two classes: p(z) = P_1 p_1(z) + P_2 p_2(z) where P_1 and P_2 are the a priori probabilities P_1 + P_2 = 1.

25. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 25 Thresholding: Global and optimal (cont.) The threshold value for which error is minimal P_1 p_1(T) = P_2 p_2(T) If σ = σ_1 = σ_2 by solving the equation, T is obtained as T = (μ_1 + μ_2)/2 + (σ^2/(μ_1 - μ_2)) ln(P_2/P_1) If P_1=P_2 the optimal threshold is the average of the means.

26. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 26 Thresholding: Global and optimal (cont.) Generally P_1 p_1(T) = P_2 p_2(T) can be formulated as AT^2 + BT + C = 0 where A = σ_1^2 - σ_2^2 B = 2 (μ_1 σ_1^2 - μ_2 σ_2^2) C = σ_1^2 μ_2^2 - σ_2^2 μ_1^2 + 2 σ_1^2 σ_2^2 ln(σ_2 P_1/ σ_1 P_2) Thus T is solved as T = -B ± √(B^2 - 4 AC)/(2A)

27. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 27 Thresholding: Categories of methods Mehmet Sezgin, Bulent Sankur, Survey over image thresholding techniques and quantitative performance evaluation, Journal of Electronic Imaging, Vol. 13, No. 1, 2004, pp. 146-165: Histogram shape-based methods. Clustering-based methods. Entropy-based methods. Object attribute-based methods. Spatial methods. Local methods.

28. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 28 Region-oriented segmentation Region growing. –Bottom-up. –Pixel by pixel. –Pixel aggregation. Region splitting and merging. –Top-down. –Subdivisions of the image. –Split and merge.

29. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 29 Region growing Pixel aggregation: start with a set of seed points and from these grow regions by appending to each seed point those neighboring pixels that have similar properties (such as gray level, texture, color). Which seed points? Similar properties? Should the criterion be constant or change as a function of current pixels.

30. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 30 Region splitting and merging Subdivide an image initially into a set of arbitrary, disjointed regions and then merge and/or split the regions in an attempt to satisfy given conditions. Method using a logical predicate P(R) over the points in set R (similar values or not): 1.Split into four disjointed quadrants any region R_i where P(R_i) = FALSE. 2.Merge any adjacent regions R_j and R_k for which P(R_j U R_K) = TRUE. 3.Stop when no further merging or splitting is possible.

31. Machinen Vision and Dig. Image Analysis Prof. Heikki Kälviäinen CT50A6100 31 Summary Image acquisition => digital image ↓ Preprocessing => better image ↓ Segmentation => features for classification/clustering.