Segmentation (Section 10.2) CS474/674 – Prof. Bebis

Segmentation The goal is to partition an image into regions (e.g., separate objects from background. Segmentation results could determine the eventual success or failure of image analysis. In general, this is a very difficult problem!

Segmentation (cont’d) Improve segmentation results by Introducing knowledge about the application domain Controlling the environment (e.g., industrial applications) Selecting special sensors to enhance the objects of interest (e.g., use thermal-infrared imaging for target recognition)

Segmentation Approaches Edge-based approaches Use the boundaries of regions to segment the image. Detect abrupt changes in intensity (discontinuities). Region-based approaches Use similarity among pixels to find different regions.

Definition of Edges Edges are significant local changes of intensity in an image. Edges typically occur on the boundary between two different regions in an image.

Goal of Edge Detection Produce a line “drawing” of a scene from an image of that scene. Important features can be extracted from the edges of an image (e.g., corners, lines, curves). These features are used by higher-level computer vision algorithms (e.g., recognition).

What Causes Intensity Changes? Geometric events Discontinuities in boundary or surface orientation and/or surface color and texture. Non-geometric events Illumination changes Specularities Shadows Inter-reflections

Edge Descriptors Edge normal: unit vector in the direction of maximum intensity change. Edge direction: unit vector to perpendicular to the edge normal. Edge position: the image position at which the edge is located. Edge strength: related to the local image contrast along the normal.

Modeling Intensity Changes Step edge: the image intensity abruptly changes from one value on one side of the discontinuity to a different value on the opposite side.

Modeling Intensity Changes (cont’d) Ramp edge: a step edge where the intensity change is not instantaneous but occur over a finite distance.

Modeling Intensity Changes (cont’d) Ridge edge: the image intensity abruptly changes value but then returns to the starting value within some short distance (i.e., usually generated by lines).

Modeling Intensity Changes (cont’d) Roof edge: a ridge edge where the intensity change is not instantaneous but occur over a finite distance (i.e., usually generated by the intersection of two surfaces).

Main Steps in Edge Detection (1) Smoothing: suppress as much noise as possible, without destroying true edges. (2) Enhancement: apply a filter to enhance the quality of the edges in the image i.e., sharpening

Main Steps in Edge Detection (cont’d) (3) Detection: determine which edge pixels should be discarded as noise and which should be retained i.e., threshold edge strength (4) Localization: determine the exact location of an edge i.e., sub-pixel resolution might be required for some applications, that is, estimate the location of an edge to better than the spacing between pixels.

Edge Detection Using Derivatives Points that lie on an edge can be detected by: (1) detecting local maxima or minima of the first derivative. (2) detecting the zero-crossings of the second derivative.

Edge Detection Using 1st Derivative – 1D (centered at x) upward step edge ramp edge downward step edge roof edge

Edge Detection Using 2nd Derivative – 1D Replace x+1 with x (i.e., centered at x):

Edge Detection Using 2nd Derivative – 1D (cont’d)

Edge Detection Using 2nd Derivative -1D (cont’d) upward step edge downward step edge ramp edge roof edge

Edge Detection Using 1st Derivative – 2D Gradient Gradient is defined as follows: Approximate gradient using finite differences:

Edge Detection Using 1st Derivative – 2D Gradient (cont’d) In pixel-coordinate notation, j corresponds to the x direction and i to the negative y direction:

Edge Detection Using 1st Derivative – 2D Roberts Edge Detector

Edge Detection Using 1st Derivative – 2D Another approximation Consider the arrangement of pixels about the pixel (i, j): The partial derivatives can be computed by: The constant c implies the emphasis given to pixels closer to the center of the mask.

Edge Detection Using 1st Derivative – 2D Prewitt Edge Detector Setting c = 1, we get the Prewitt operator:

Edge Detection Using 1st Derivative – 2D Sobel Edge Detector Setting c = 2, we get the Sobel operator:

Main Steps in Edge Detection (i.e., sqrt is expensive!)

Isotropic property of gradient magnitude The magnitude detects edges in all possible directions magnitude

Example (using Prewitt) (divisions by 2 are for normalization only)

Some Practical Issues Noise suppression-localization tradeoff Larger filters reduce noise, but worsen localization (i.e., it adds uncertainty to the location of the edge) and vice versa. smaller filter larger filter

Some Practical Issues (cont’d) How should we choose the threshold? gradient magnitude low threshold high threshold

Some Practical Issues (cont’d) Edge thinning and linking are usually required.

Criteria for Optimal Edge Detection (1) Good detection: the optimal detector must minimize the probability of false positives (detecting spurious edges caused by noise), as well as that of false negatives (missing real edges). (2) Good localization: the edges detected must be as close as possible to the true edges. (3) Single response constraint: the detector must return one point only for each true edge point; that is, minimize the number of local maxima around the true edge (created by noise).

The Canny edge detector Canny has shown that the first derivative of the Gaussian closely approximates the operator that optimizes the product of signal-to-noise ratio and localization. Canny’s analysis is based on "step-edges" corrupted by "additive Gaussian noise".

The Canny edge detector (cont’d)

The Canny edge detector (cont’d) Non-maxima suppression Preserve local maxima of the gradient magnitude. Suppress non-maxima along the direction of the gradient. Thin edges!

The Canny edge detector (cont’d) Non-maxima suppression (i,j)

The Canny edge detector (cont’d) Hysteresis thresholding/Edge Linking The output of non-maxima suppression still contains local maxima created by noise. Can we get rid of them just by using a single threshold? If we set a low threshold, some noisy maxima will be accepted too. If we set a high threshold, true maxima might be missed. The value of true maxima will fluctuate above and below the threshold, fragmenting the edge.

The Canny edge detector (cont’d) Hysteresis thresholding/Edge Linking gradient magnitude low threshold high threshold

The Canny edge detector (cont’d) Hysteresis thresholding/Edge Linking A more effective scheme is to use two thresholds: a low threshold tl a high threshold th (usually, th = 2tl) Note: large gaps are still difficult to bridge ...more powerful algorithms are needed!

Example – what is the effect of σ?

Example – what is the effect of σ?

Edge Detection Using 2nd Derivative – 2D Laplacian mask:

Edge Detection Using 2nd Derivative – 2D Laplacian Example detect zero-crossings

Edge Detection Using 2nd Derivative – 2D Properties of Laplacian It is an isotropic operator. It is cheaper to implement (i.e., one mask only). It does not provide information about edge direction It is more sensitive to noise (i.e., differentiates twice).

Edge Detection Using 2nd Derivative – 2D Laplacian of Gaussian (LoG) To reduce the noise effect, the image is first smoothed with a low-pass filter. In the case of the LoG, the low-pass filter is chosen to be a Gaussian. It can be shown that: σ controls smoothing

Edge Detection Using 2nd Derivative – 2D Laplacian of Gaussian (LoG) Example convolution result zero-crossings

Gradient vs LoG Comparison Gradient works well when the image contains sharp intensity transitions and low noise. Zero-crossings of LOG offer better localization, especially when the edges are not very sharp. step edge ramp edge edge can be better localized

Multiscale processing (scale space) A serious practical problem with any edge detector is the matter of choosing the scale of smoothing (e.g.,the value of σ using a Gaussian). For many applications, it is desirable to be able to process an image at multiple scales. We determine which edges are most significant in terms of the range of scales over which they are observed to occur.

Multiscale processing (cont’d)

Multiscale processing (cont’d)