1. Edge Detection Contour Segmentation Hough Transform
Edges
Edge Detection
Contour Segmentation
Hough Transform
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

2. Edges An approach to segmentation.
The analysis of the discontinuities in an image.
No correct answer?
An alternative to region based processing.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

3. Edge Detection – Topics
1st derivative edge detection
2nd derivative edge detection
Multispectral edge detection
Image sharpening
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

4. Edge Detection – What is an edge?
Where brightness changes abruptly
Edges have
Magnitude (Gradient)
Direction (Orientation)
Edge Profiles
Step
Real
Noisy
Edges can be located in color channels just as easily as in brightness channels.
Edges are pixels where brightness changes abruptly.
Second diagram shows the first two derivative of a function in 1 dimension. It should be clear that the ‘edge’ is the maximum of the first derivative and the zero crossing of the second derivative. This is clearer when the function changes more rapidly (e.g. brightness in an image). In two dimensions we can introduce the notion of edge direction as well.
Some edge detector are invariant to rotation (e.g. Laplacian)
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

5. Edge Detection – 1st derivative definitions
Calculus...
Rate of change in two directions
Vector variable:
Gradient Magnitude
Orientation (0o is East)
arg(x,y) is the angle in radians from the x acis to the point (x, y) (depends on how you look at the axes as to where 0 degrees is).
The various edge profiles are shown in order. The step edge is in many ways the most normal although in reality it would not be so sharp a step. The roof edge is typical of thin objects!
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

6. Edge detection – Digital images
Derivatives work on continuous functions
Map every point in the input image to the output
Discrete domain
Differences
Orthogonal
Need to define the partial derivates so that they:
Cross at a single middle point
Preferably cross at the centre of a pixel
Evaluate points which are not too close together
Deal with some degree of image noise
© Brooks/Cole Publishing Company
Derivates are a concept that works with continuous functions. Our digitial images are discrete so what do we do?
We use image differences.. In the case of a simple first derivative operator we look at partial derivates such as those shown in two orthogonal directions (I and J). We take the difference between two points which are n pixels apart (n is often 1). The difference between the two formulations shown is that the second is symmetric around the point being processed and given a measure of the gradient at the exact point rather than between that and neighbouring points. It is also important to note that the two differences cross in the center of the point, whereas in the first derivative they don’t overlap.
The categories of gradient operator are a little contrived.
Using image differences – This covers both first derivative and second derivative operators!
Based on zero-crossings – This just covers 2nd derivative operators
Matching to a parametric model – a bit different – an explicit edge model.
I would categorise by 1st, derivative, 2nd derivative and others.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

7. Edge detection – 1st derivative – Roberts
Convolution Masks
Sensitivity to Noise
Binary Images
The Roberts operator is one of the oldest. It is two simple simple 2x2 masks which giving partial derivatives in orthogonal directions where the derivatives intersect between the 4 pixels which are used. Hence the resulting edge image is shifted somewhat with relation to the original image.
The gradient can be computed as shown or (4.42) or in the more general case can be done as a root mean square (RMS) or a sum of absolutes (same as 4.42) of any two orthongal partial derivatives (x and y).
The orientation is computed simply as an inverse tangent of the two partial derivatives. Note that where the directions are not along the I and J axes the result of the inverse tangent would need to be adjusted if 0 degrees is to be East.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

8. Edge detection – 1st derivative – Compass
Compass edge detectors:
Partial derivatives defined for a number of orientations (typically 8)
Prewitt:
Use h3(i,j) and h1(i,j) to derive gradient and orientation
We should demonstrate the problems associated with doing the edge detection in two stages within TIPS… in both demos.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

9. Edge detection – 1st derivative – Compass
Sobel
We should demonstrate the problems associated with doing the edge detection in two stages within TIPS… in both demos.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

10. Edge detection – 1st derivative – Compass
Mat horizontal_derivative, vertical_derivative;
Sobel( gray_image, horizontal_derivative, CV_32F,1,0 );
Sobel( gray_image, vertical_derivative, CV_32F,0,1 );
Mat abs_gradient, l2norm_gradient, orientation;
abs_gradient = abs(horizontal_derivative) +
abs(vertical_derivative);
cartToPolar(horizontal_derivative,vertical_derivative,
l2norm_gradient,orientation);
We should demonstrate the problems associated with doing the edge detection in two stages within TIPS… in both demos.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

11. Edge detection – 1st derivative – Thresholding
Simple thresholding
Too many points
Too few points
Determining the correct threshold so that the resultant binary edge image has continuous thin edges is basically impossible.
The example shows An edge image (a) together with three thresholded versions (b), (c) & (d) thresholded at 40, 60 & 80 respectively
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

12. Edge detection – 1st derivative – Non maxima suppression
Use orientation information
Algorithm:
Quantise edge orientations
For all points (i,j)
Look at the 2 points orthogonal to edge
if gradient(i,j) < gradient(either of these 2 points)
output(i,j) = 0
else output(i,j) = gradient(i,j)
Now thresholding can be used… or hysteresis thresholding (detailed later…)
Determining the correct threshold so that the resultant binary edge image has continuous thin edges is basically impossible.
For non-maxima suppresion edge orientation/direction information is required.
The algorithm stated here uses a separate output to make it easy to understand. This also means it can be done in a single pass.
Hysteresis thresholding was used in Canny edge detection (uses 2 thresholds – one for definite edges and the other for edges which will be allowed if they are connected to a definite threshold).
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

13. Edge detection – 1st derivative – NMS
nms_result = gradients.clone();
for (int row=1; row < gradients.rows-1; row++)
for (int column=1; column < gradients.cols-1; column++) {
float curr_gradient = gradients.at<float>(row,column);
float curr_orientation = orientations.at<float>(row,column);
// Determine which neighbours to check
int direction = (((int) (16.0*(curr_orientation)/(2.0*PI))+15)%8)/2;
float gradient1 = 0.0, gradient2 = 0.0;
switch(direction)
case 0:
gradient1 = gradients.at<float>(row-1,column-1);
gradient2 = gradients.at<float>(row+1,column+1);
break;
We should demonstrate the problems associated with doing the edge detection in two stages within TIPS… in both demos.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

14. Edge detection – 1st derivative – NMS
case 1:
gradient1 = gradients.at<float>(row-1,column);
gradient2 = gradients.at<float>(row+1,column);
break;
case 2:
gradient1 = gradients.at<float>(row-1,column+1);
gradient2 = gradients.at<float>(row+1,column-1);
case 3:
gradient1 = gradients.at<float>(row,column+1);
gradient2 = gradients.at<float>(row,column-1); }
if ((gradient1 > curr_gradient) || (gradient2 > curr_gradient))
nms_result.at<float>(row,column) = 0.0;
We should demonstrate the problems associated with doing the edge detection in two stages within TIPS… in both demos.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

15. Edge detection – 2nd derivative – Laplace operator
Orientation independent
Determines gradient magnitude
Linear ramps
Noise
Zero-crossings
There are a number of possible masks for implementing the Laplace operator. It approximates the 2nd derivative although it is still implements using image differences. It can be looks at as the combination of two partial 2nd derivative (and when combined with the Gaussian this can make for a more efficient implmentation).
The operator is independent of orientation
It determines the edge gradient directly
It gives no response to linear ramps
It responses quite badly to noise
To find edges the resultant image needs to be searched for ‘zero-crossings’.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

16. Edge detection – 2nd derivative – Marr-Hildreth
Marr-Hildreth edge detector
2nd derivative zero crossings
How to compute robustly?
Smooth the image first
The smoothing filter must be
Smooth and roughly band-limited
Spatial localisation
Optimal solution
Gaussian filter
The first 2nd derivative edge detector was that of Marr-Hildreth, and this detector is still one of the most widely used. It is based on locating zero-crossings in the 2nd derivative
Second derivative edge detectors are extremely noisy so to detect edges robustly we need to reduce the level of noise by smoothing before applying the edge detector
The smoothing filter has to satisfy two criteria
that it is smooth and band limited in the frequency domain – it must limit the frequencies of edges in the image
Spatial localisation – the smoothing filter must not move the edges or their spatial relationships.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

17. Edge detection – 2nd derivative – Laplacian of Gaussian
Combine Gaussian smoothing with the Laplacian operator
Change ordering
Convolution Mask
Mexican hat filter
Pros and Cons
Larger area taken into account
Too much smoothing
Closed loops of edges
Ganglion Cells
Simply – apply Gaussian smoothing followed by the Laplacian.
We can change the order of the differentiation and convolution due to the linearity of the operators – so we get a single operator (convolution) to perform both steps.
We change determine a formula for computing a convolution mask based on (x,y) position in the mask and the standard deviation of the Gaussian. Note that c is just a normalizing multiplicative coefficient which normalises the sum of all mask elements to 0. When applying masks to images we usually have to normalise the result so that it fits within some range (e.g. in the 5x5 mask shown in the text the range of possible values are from (-255*16 to 255*16), and in this case we might divide the answer by 16 for storage).
Due to the shape of the filter it is often called the Mexican hat filter. Shown is the filter in 1-D.
The advantage of this filter is that it can taken a much larger area into account (when compared to normal edge operators) when computing edges. This is not always a good thing though – say if a number of edges are present within that area…
The operator often does too much smoothing – losing fine details such as corners.
The operator gives guaranteed closed loops of edges – which was presented as an important benefit but causes problems for some applications (or more to the point certain types of postprocessing).
It has been shown using neurophysiological experiments that the human eye retina (in the form of ganglion cells) perform operations which are very similar to Laplacian of Gaussian…
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

18. Edge detection – 2nd derivative – Laplacian of Gaussian
Mat laplacian;
Mat blurred_image1_gray;
GaussianBlur( gray_image, blurred_image, Size(5,5), 1.5 );
Laplacian( blurred_image, laplacian, CV_32F, 3 );
Simply – apply Gaussian smoothing followed by the Laplacian.
We can change the order of the differentiation and convolution due to the linearity of the operators – so we get a single operator (convolution) to perform both steps.
We change determine a formula for computing a convolution mask based on (x,y) position in the mask and the standard deviation of the Gaussian. Note that c is just a normalizing multiplicative coefficient which normalises the sum of all mask elements to 0. When applying masks to images we usually have to normalise the result so that it fits within some range (e.g. in the 5x5 mask shown in the text the range of possible values are from (-255*16 to 255*16), and in this case we might divide the answer by 16 for storage).
Due to the shape of the filter it is often called the Mexican hat filter. Shown is the filter in 1-D.
The advantage of this filter is that it can taken a much larger area into account (when compared to normal edge operators) when computing edges. This is not always a good thing though – say if a number of edges are present within that area…
The operator often does too much smoothing – losing fine details such as corners.
The operator gives guaranteed closed loops of edges – which was presented as an important benefit but causes problems for some applications (or more to the point certain types of postprocessing).
It has been shown using neurophysiological experiments that the human eye retina (in the form of ganglion cells) perform operations which are very similar to Laplacian of Gaussian…
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

19. Edge detection – 2nd derivative – Laplacian of Gaussian
Separable decomposition
Difference of Gaussians
Finding zero-crossings
Look for zeros?
Band thresholding?
Search using a 2x2 window
Limit search to significant 1st derivatives
Need to come up with efficient ways of applying this filter as the filter mask is typically very large (depending on sigma).
2-D Laplacian of Gaussian can be separated into a number of 1-D convolutions.
Difference of Gaussian is another way in which the Laplacian of Gaussian can be approximated by the difference of two gaussian averaging masks with substantiallly different values of sigma…
To find zero crossings is not simple
Can’t just look for zeros (as the domain is not continuous)
Could try to threshold a range of values around zero
Look in a 2x2 window and mark the top left point as a zero-crossing if a zero crossing exists in the window (i.e. any point >= 0 and some other point < 0). Note we are throwing position potentially to subpixel accuracy (of the z-c away!)
To avoid computing insignificant z-cs we could just search in the region of the image where the first derivative is over some threshold.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

20. Edge detection – 2nd derivative – Scale
Image processing techniques work locally
Size of neighbourhood to use?
Problem of scale
Chicken before the egg…
Using different resolutions
Create a model under each resolution
Study changes in model to obtain meta-knowledge
Many IP techniques work locally using individual pixels or more frequently pixels from a local area.
The problem is knowing what size of neighbourhood to use and the ‘right’ size depends on the size of the objects under invesigation.
This is effectively a problem of scale within the image (Consider
Knowing what the objects are assumes that it is clear how the image should be interpreted which is OK for industrial applications but for general vision seems intractable (a chicken before the egg type problem)
The solution to this is to study the image under different resolutions (scales). We create models at multiple resolutions and then study the way in which the model changes at different resolutions to obtain meta-knowledge which is unavailable at one scale.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

21. Edge detection – 2nd derivative – Scale
Example: Marr’s edge detection:
Smooth image using different Gaussians
Look for correspondences across scale to identify significant discontinuities
Many IP techniques work locally using individual pixels or more frequently pixels from a local area.
The problem is knowing what size of neighbourhood to use and the ‘right’ size depends on the size of the objects under invesigation.
This is effectively a problem of scale within the image (Consider
Knowing what the objects are assumes that it is clear how the image should be interpreted which is OK for industrial applications but for general vision seems intractable (a chicken before the egg type problem)
The solution to this is to study the image under different resolutions (scales). We create models at multiple resolutions and then study the way in which the model changes at different resolutions to obtain meta-knowledge which is unavailable at one scale.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

22. Edge detection – 2nd derivative – Canny
Optimality criteria:
Detection
Localisation
One response
Detection – edges should not be missed
Localisation – Distance between actual and located should be minimised
One response – Minimises multiple responses to a single edge
ALGORITHM: (Note that the Laplacian of Guassian does not compute orientation. Canny computes orientation, location and gradient in 3 separate steps).
Convolve with Gaussian with some st. dev. Sigma
Estimate the edge normal direction (i.e. orientation) n using the first derivative of the Guassian convolved image (g in the equations here – f in the book)
Find the edges, suppressing non-maxima values (non-maxima suppression). This is done by searching for zero-crossing in the directional second derivative of the Gaussian convolved image where that derivative is taken wrt to edge normals.
The edge gradient/magnitude/strength is computed based on the first derivative of the Guassian convolved image
Thresholding edges with hysteresis. The idea here is that edge points are connected. Two thresholds are used – a high gradient threshold over which all edge points are definitely edges, and a low threshold over which points which are connected to definitive edge points will be considered edge points. This attempts to get over the problem of edge contours which break up into unconnected segments.
Similarly to Marr’s detector we can do this processing for multiple scales (using different Guassians) and then ‘feature synthesis’ is used to combine edges from different scales (see book for more details – similar to Marr).
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

23. Edge detection – 2nd derivative – Canny algorithm
Algorithm (Compute orientation, location, & gradient)
Convolve image with Gaussian
Estimate edge orientation using first derivative
Locate the edges
Non-maxima suppression using zero-crossing
Compute edge magnitude using first derivative
Threshold edges with hysteresis
Repeat for mulitple scales
Feature synthesis
Detection – edges should not be missed
Localisation – Distance between actual and located should be minimised
One response – Minimises multiple responses to a single edge
ALGORITHM: (Note that the Laplacian of Guassian does not compute orientation. Canny computes orientation, location and gradient in 3 separate steps).
Convolve with Gaussian with some st. dev. Sigma
Estimate the edge normal direction (i.e. orientation) n using the first derivative of the Guassian convolved image (g in the equations here – f in the book)
Find the edges, suppressing non-maxima values (non-maxima suppression). This is done by searching for zero-crossing in the directional second derivative of the Gaussian convolved image where that derivative is taken wrt to edge normals.
The edge gradient/magnitude/strength is computed based on the first derivative of the Guassian convolved image
Thresholding edges with hysteresis. The idea here is that edge points are connected. Two thresholds are used – a high gradient threshold over which all edge points are definitely edges, and a low threshold over which points which are connected to definitive edge points will be considered edge points. This attempts to get over the problem of edge contours which break up into unconnected segments.
Similarly to Marr’s detector we can do this processing for multiple scales (using different Guassians) and then ‘feature synthesis’ is used to combine edges from different scales (see book for more details – similar to Marr).
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

24. Edge detection – 2nd derivative – Canny
Detection – edges should not be missed
Localisation – Distance between actual and located should be minimised
One response – Minimises multiple responses to a single edge
ALGORITHM: (Note that the Laplacian of Guassian does not compute orientation. Canny computes orientation, location and gradient in 3 separate steps).
Convolve with Gaussian with some st. dev. Sigma
Estimate the edge normal direction (i.e. orientation) n using the first derivative of the Guassian convolved image (g in the equations here – f in the book)
Find the edges, suppressing non-maxima values (non-maxima suppression). This is done by searching for zero-crossing in the directional second derivative of the Gaussian convolved image where that derivative is taken wrt to edge normals.
The edge gradient/magnitude/strength is computed based on the first derivative of the Guassian convolved image
Thresholding edges with hysteresis. The idea here is that edge points are connected. Two thresholds are used – a high gradient threshold over which all edge points are definitely edges, and a low threshold over which points which are connected to definitive edge points will be considered edge points. This attempts to get over the problem of edge contours which break up into unconnected segments.
Similarly to Marr’s detector we can do this processing for multiple scales (using different Guassians) and then ‘feature synthesis’ is used to combine edges from different scales (see book for more details – similar to Marr).
Canny( gray_image, binary_edges, 100, 200 );
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

25. Multispectral edge detection
Detect edges separately in each spectral band
Use maximal value OR some linear combination
Multi-spectral edge detector
Use a different colour model )
e.g. HLS space with linear combination of H & L
We’re skipping over (Parametric edge detectors) as it adds little and is not mainstream.
Section gives a 3 options for dealing with multispectral images
Detect in each band and combine somehow
Use the difference between values in two bands. The text says this is very informative. In our test case it was anything but.
Use a purpose designed multi-spectral edge detector. The one shown is based on the a Roberts like detector.
I give another option thatn the text ignores, which is to propose the image is some other colour space (where the colour space no longer represents separate spectral bands. An example is shown using HLS and combining the Hue and Lightness channels. There are particularly interesting differences particularly around the clock.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

26. Image sharpening Making edges steeper. Demo (Sharpening.twk)
This is thrown into the text at this point and seems a little off the point (but is interesting).
C is a positive coefficient giving the strength of sharpening
S(i,j) is a measure of image function sheerness (i.e. the rate of change; effectively the gradient). Can be computed using any measure of the Gradiant but Laplacian is good as it gives the gradient directly.
Diagram shows what is being done to a 1-D edge, and the example nicely illustrates it (better than text).
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

27. Image sharpening
Subtract a multiple (e.g. 0.3) of the Laplacian from the image.
image.convertTo( image_32bit, CV_32F );
Laplacian( image, laplacian, CV_32F, 3 );
Mat sharp_image_32bit = image_32bit - 0.3*laplacian;
sharp_image_32bit.convertTo( sharp_image, CV_8U );
This is thrown into the text at this point and seems a little off the point (but is interesting).
C is a positive coefficient giving the strength of sharpening
S(i,j) is a measure of image function sheerness (i.e. the rate of change; effectively the gradient). Can be computed using any measure of the Gradiant but Laplacian is good as it gives the gradient directly.
Diagram shows what is being done to a 1-D edge, and the example nicely illustrates it (better than text).
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

28. Contour Segmentation – Topics
Edge data representations
Border Detection
Line segment extraction
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

29. Contour Segmentation – Boundary Chain Codes
Each chain contains
Start point
A list of orientations
Features!
Orientation & Scale dependent
Slightly position dependent
Can be smoothed to reduce boundary noise
Difficulties obtaining consistent representation from image.
THIS IS THE START OF “Graph searching and BCCs”
These are one of the simplest represenatations of boundary points and were originally developed to be particularly efficient in terms of storage requirements. This is as a precursor to border tracing as the results of boundary tracing have to be stored in some sort of representation.
The edge orientations can specified using either 4 or 8 connectivity.
If considering this as a shape representation then we have to consider how useful it will be for further processing.
The representation changes hugely if the orientation of the object/region and also will change somewhat depending on object scale.
It is only position dependent to the extent of the start point, although it should be noted that the start point is somewhat arbitray for a close contour.
The boundary can be smoothed to reduce noise although it would need to be done sensibly (so that the contour doesn’t deform).
Obtaining a consistent representation from the image is made hard by contours which may not be closed (LoG guarantees this) and by contours with T junctions or cross roads (what do you do?)
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

30. Contour Segmentation – Boundary Chain Codes
THIS IS THE START OF “Graph searching and BCCs”
These are one of the simplest represenatations of boundary points and were originally developed to be particularly efficient in terms of storage requirements. This is as a precursor to border tracing as the results of boundary tracing have to be stored in some sort of representation.
The edge orientations can specified using either 4 or 8 connectivity.
If considering this as a shape representation then we have to consider how useful it will be for further processing.
The representation changes hugely if the orientation of the object/region and also will change somewhat depending on object scale.
It is only position dependent to the extent of the start point, although it should be noted that the start point is somewhat arbitray for a close contour.
The boundary can be smoothed to reduce noise although it would need to be done sensibly (so that the contour doesn’t deform).
Obtaining a consistent representation from the image is made hard by contours which may not be closed (LoG guarantees this) and by contours with T junctions or cross roads (what do you do?)
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

31. Contour Segmentation – Boundary Chain Codes
vector<vector<Point>> contours;
vector<Vec4i> hierarchy;
findContours( binary_edge_image, contours, hierarchy,
CV_RETR_CCOMP, CV_CHAIN_APPROX_NONE );
for (int contour_number=0;
(contour_number<contours.size()); contour_number++) {
Scalar colour( rand()&0xFF, rand()&0xFF, rand()&0xFF );
drawContours( display_image, contours, contour_number,
colour, 1, 8, hierarchy ); }
THIS IS THE START OF “Graph searching and BCCs”
These are one of the simplest represenatations of boundary points and were originally developed to be particularly efficient in terms of storage requirements. This is as a precursor to border tracing as the results of boundary tracing have to be stored in some sort of representation.
The edge orientations can specified using either 4 or 8 connectivity.
If considering this as a shape representation then we have to consider how useful it will be for further processing.
The representation changes hugely if the orientation of the object/region and also will change somewhat depending on object scale.
It is only position dependent to the extent of the start point, although it should be noted that the start point is somewhat arbitray for a close contour.
The boundary can be smoothed to reduce noise although it would need to be done sensibly (so that the contour doesn’t deform).
Obtaining a consistent representation from the image is made hard by contours which may not be closed (LoG guarantees this) and by contours with T junctions or cross roads (what do you do?)
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

32. Contour Segmentation – Border detection as graph searching
Nodes ni
Arcs (ni, nj) with associated cost and orientation
Border detection
Search for optimal path
Cost related to Gradient = s(xi)
Orientation = Edge direction = (xi)
ni, nj corresponding to 8-connected pixels xi, xj are connected by an arc if
(xi) and (xj) match the local border
We are going to consider the border searching problem in terms of graph searching. A graph is a general structure consisting of nodes and arcs. We consider oriented arcs which are numerically weighted (costs).
Border detection is a search for the optimal path in the weighted graph to find the best path which connects two nodes.
The nodes n correspond directly to pixel x (why we need both is beyond me).
An edge orientation image (a) with the orientation vector for each edge point shown. Possible source and destination edge points are shown in green and red respectively. The corresponding directed graph is also shown (b) with the nodes (which correspond to the edge pixels) shown as red dots and the oriented arcs (the allowed connections between nodes) shown as arrows between nodes.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

33. Contour Segmentation – A-algorithm graph search
Determine successors for all nodes:
For any given node ni based on (xi)
s(xi) > T, s(xj) > T
xj must be one of 3 neighbours in the direction [(xi)-/4, (xi)+/4]
| (xi) - (xj) | < /2
Expand(nA): Put all its successors into an OPEN list and evaluate accumulated cost f(nk) for these successors (Keep path from nA)
While OPEN   and nB  OPEN
Choose the node in the OPEN list with the lowest f(nk) and Expand
If nB  OPEN
Optimal path found
Otherwise No path
Problems:
Infinite loops
This is an slightlly alternative presentation to that given in the book for the A-algorithm graph search.
Note that the path that is kept for each node in the OPEN list could easily be represented as a BCC.
There are problems with the approach such as the possibility of an infinite loop (example shown in the figure). These can be prevented by for example not allowing backward search but this would seriously affect the shapes which could be identified. Note that the book gives an example of a geometric warping to straighten the feature before the search (the scenario is one in which there is some a priori information about the boundary.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

34. Contour Segmentation – Gradient field transform
Need to incorporate cost to the destination
ĥ(ni) = ĉ(ni , nB)
h(ni) = c(ni , nB)
Use the actual cost h(ni)?
How can we estimate the cost?
Better estimates…
Overcoming the problems of the previous approach we include the estimated cost of a path to the destination. In this case the node to expand will be that with the lowest cost (so far) added together with the estimated remaining cost to the destination.
ĥ(ni) = ĉ(ni , nB) = estimate of the cost of the path ni  nB
h(ni) = c(ni , nB) = true cost of the path ni  nB
In the first case (ĥ(ni) = 0) cost is not considered so the algorithm would be the same as the A-algorithm.
In the second case (ĥ(ni) = h(ni)) the estimate is the same as the true cost and hence only one path need be expanded.
In the third case we require the estimate always to be less than the actual cost and we also place an ordering constraint on the estimates. Unfortunately this is virtually impossible to guarantee.
The last case where the estimate is larger than the true cost will not guarantee to find the minimum cost path but will be faster.
IN PRACTICAL TERMS, the better the estimate is the smaller the number of nodes that would need to be expanded.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

35. Contour Segmentation – Estimating the cost
Cost so far
Strength of edges
(maximage s(xk)) - s(xi)
Almost always gives good results
Border curvature
diff[ (xi) - (xj) ]
Remaining cost…
Proximity to an approximate border
dist(xi, approximate_boundary)
Distance to the goal
dist(xi, xB)
The heuristic ‘the stronger the edges that form the border, the higer the probability of the border’ is very natural and almost always gives good results. This doesn’t give an estimate to a destination though – only allows for the cost so far to be evaluated.
Generally low border curvature is a trait to be sought… The diff function gives the absolute angle between the orientation. This could be combined with the previous measure to evaluate cost so far.
The proximity to an approximate border is based on have an estimate of what to expect and therefore favouring borders which go close to this estimate (e.g. skull detection application).
The distance to the goal is really the only one of these measures which really provides an estimate of the cost between the current node and the goal.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

36. Contour Segmentation – More efficient search
Pruning the solution tree
High average cost per unit length
Maximum cost
Max. path cost.
Depth first search
Reduce candidate edges
Multi-resolution processing
Determine approximate boundary first
Incorporation of higher-level knowledge
A priori knowledge
Pruning: Delete paths which have high cost per unit length
Least maximum: Cost of a path given as the cost of the most expensive link
Branch and bound: Delete any paths exceeding a max path cost. This can be computed beforehand or updated during the graph search. (Doesn’t seem very useful though – as only the minimum cost paths are expanded – unless the paths are all evenly expanded).
Lower bound: Reduce the likelihood that poor edges will be expanded by reducing the cost of all expanded nodes by the cost of the best successor (whose cost will be 0 – hence guaranteeing that this node will be expanded again immediately).
Multi-resolution: Assumes the boundary can be detected in the lower resolution image (which due to the scaling effects discussed in an earlier section) may not be the case.
Higher-level knowledge: e.g. expected boundary shape
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

37. Contour Segmentation – Heuristic search for image borders
Find all edges in an image
Algorithm:
Search for the strongest (unused) edge in the image. If none go to step 5.
Expand all the edges in front of the specified edge
Expand all the edges behind of the specified edge
If the edge chain consists of > 3 pixels store it. Go to step 1.
Modify edge chains to fill small breaks and to remove duplicate contours.
Repeat step 5 until stable.
THIS IS THE END OF “Graph searching and BCCs”
The purpose of this algorithm is to solve the general problem of identifying ‘all’ boundaries in an edge image.
Select an edge poinjt only if it is of sufficient magnitude
Expand the edge chain using the normal graph searching.
Same again
As part of this step all pixels on the edge chin should be marked used.
The 3 rules are (1) to get thin edges, (2) remove changes in lighting! And (3) to fill in small gaps (more complicated systems are sometimes used to fill in the gaps)
Iterate (not much) until the few changes are being made.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

38. Contour Segmentation – Segment sequences
Boundary
Sequence of segments
Segment
Start and end points: x1 and x2
Type: straight line, some type of curve, etc.
Other parameters
Accurate polygonal representation
Must define acceptable tolerances
Vertices – where?
THIS IS THE START OF “Border refining and segment sequences”
A boundary may be represented as a sequence of segments.
A segment then must have a start and end point (vertices) and if two segments are connected in a boundary x1 from one segment should match x2 from the other segment (or vice versa).
A polygonal represenation would use only straight line segments (this is the most common representation). If a curve is used other parameters must be provided (such as curvature).
To obtain an polynomial reprsentation we need some tolerances to specify what is acceptable as a straight line. Possibly our biggest problem though is determining where the vertices should be. We would like them to be stable (so if we see the same shape again, the vertices will be in the same places on the object!).
Bottom picture shows A thresholded edge image (a), the boundary chain codes extracted (b) using heuristic search, and the straight line segments extracted (c) & (d) with a maximum distance of 2 and 5 respectively.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

39. Contour Segmentation – Straight line extraction example
These two techniques are similar in what they are doing (and interestingly are from different chapters!).
The first technique is using the data to drive where the splits happen and this results in segments of very uneven lengths.
The second technique splits in a standard way and the edge segments are of more consistent lengths (note that some sections have to be divided more that others).
Note that section also talks snakes briefly mentioning them as another way so solve this problem.
Note also that when the algorihtms above are finished it would probably be worthwhile to try and merge segments (e.g. in the middle of the Divide and Conquer example) – perhaps even do something iterative to try and improve stabillity..
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

40. Contour Segmentation – Obtaining polygonal segments
Recursive boundary splitting
Split at furthest point
Keep going until with tolerance
Divide and conquer
Split in the middle
These two techniques are similar in what they are doing (and interestingly are from different chapters!).
The first technique is using the data to drive where the splits happen and this results in segments of very uneven lengths.
The second technique splits in a standard way and the edge segments are of more consistent lengths (note that some sections have to be divided more that others).
Note that section also talks snakes briefly mentioning them as another way so solve this problem.
Note also that when the algorihtms above are finished it would probably be worthwhile to try and merge segments (e.g. in the middle of the Divide and Conquer example) – perhaps even do something iterative to try and improve stabillity..
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

41. Contour Segmentation – Obtaining polygonal segments
vector<vector<Point>> approx_contours(contours.size());
for (int contour_number=0;
(contour_number<contours.size()); contour_number++)
approxPolyDP( Mat(contours[contour_number]),
approx_contours[contour_number], 3, true );
These two techniques are similar in what they are doing (and interestingly are from different chapters!).
The first technique is using the data to drive where the splits happen and this results in segments of very uneven lengths.
The second technique splits in a standard way and the edge segments are of more consistent lengths (note that some sections have to be divided more that others).
Note that section also talks snakes briefly mentioning them as another way so solve this problem.
Note also that when the algorihtms above are finished it would probably be worthwhile to try and merge segments (e.g. in the middle of the Divide and Conquer example) – perhaps even do something iterative to try and improve stabillity..
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

42. Contour Segmentation – Obtaining polygonal segments
vector<Vec4i> line_segments;
for (int contour_number=0;
(contour_number<contours.size()); contour_number++)
for (int segment_num=0; (segment_num <
approx_contours[contour_number].size()-1);
segment_num++)
line_segments.push_back( Vec4i(
approx_contours[contour_number][segment_num].x,
approx_contours[contour_number][segment_num].y,
approx_contours[contour_number][ segment_num+1].x,
approx_contours[contour_number][segment_num+1].y));
These two techniques are similar in what they are doing (and interestingly are from different chapters!).
The first technique is using the data to drive where the splits happen and this results in segments of very uneven lengths.
The second technique splits in a standard way and the edge segments are of more consistent lengths (note that some sections have to be divided more that others).
Note that section also talks snakes briefly mentioning them as another way so solve this problem.
Note also that when the algorihtms above are finished it would probably be worthwhile to try and merge segments (e.g. in the middle of the Divide and Conquer example) – perhaps even do something iterative to try and improve stabillity..
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

43. Contour Segmentation – Curved segments
What order of curves should we use?
Curves of Constant curvature
Second order polynomials (circles, ellipses, parabolas)
Where does one curve end and the next start?
Use straight lines instead…
Determining curvature
Cannot rely on orientation values
Cannot use nearest neighbour pixels
Look at well separated points…
When using curved segments we have to decide what type of curves we are going to use and then the problem of extracting them in a stable manner from the image becomes harder and harder (e.g. vertices for polygonal representation hopefully wil go to the corners. For curved segments where does one curve end and the next begin? The possibilities for multiple interpretations (all equally valid) is significantly increased.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

44. Hough Transform – Topics
Hough for lines
Hough for circles
Generalised Hough
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

45. Hough transform Direct transformation from image space to
probability of the existence of some feature…
Lines
Circles
Generalised shapes
THIS IS THE START OF “Hough transform”.
The Hough will work well even with noisy or occluded shapes.
Example Apps: Lane detection, coin size evaluation, eye tracking, …
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

46. Hough Transform – Circle detection
Equation of a circle
Assume constant radius r
Transform
From Image space (x, y)
To Hough space (a, b)
Algorithm
Initialise accumulator to 0
For every edge point
Increment cells in accumulator corresponding to all possible circle centers
Search for Maximums
The “Hough space” in this case is the circle likelihood… with coefficients a and b which are the circle centre (i.e. each cell represents the likelihood (accumulated evidence – hence “accumulator”) of there being a circle with centre (a,b))
We haven’t considered what size the Hough space should be nor what resolution it should be – we’ll look at that for the line, but it applied equally for the circle.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

47. Hough Transform – Circle detection
The “Hough space” in this case is the circle likelihood… with coefficients a and b which are the circle centre (i.e. each cell represents the likelihood (accumulated evidence – hence “accumulator”) of there being a circle with centre (a,b))
We haven’t considered what size the Hough space should be nor what resolution it should be – we’ll look at that for the line, but it applied equally for the circle.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

48. Hough Transform – Line detection
Line equation:
j = m.i + c
Lines in Hough space
What about i = c
r = i.cos θ + j.sin θ
Sinusoidal curves
The standard line equation (y = mx+c) – don’t ask why he uses k and q – does not provide a representation for x = c. Hence we will use a different line representation. However this standard line representation is simpler to think about. For example in the top diagram points A and B map to lines in (k, q) Hough space and these lines intersect in just one point which represents the line that goes through both of these points.
In the second case the equation is not as well know but s and theta correspond to the distance from the origin and the angle of the line wrt the X axis respectively. In this case a point (x,y) maps to a sinusoidal curve in Hough space (although the text seems to gloss over this – showing how a point in Hough space corresponds to a line in x,y space). The Java demo explains the whole thing quite well.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

49. Hough Transform – Line detection
The “Hough space” in this case is the circle likelihood… with coefficients a and b which are the circle centre (i.e. each cell represents the likelihood (accumulated evidence – hence “accumulator”) of there being a circle with centre (a,b))
We haven’t considered what size the Hough space should be nor what resolution it should be – we’ll look at that for the line, but it applied equally for the circle.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

50. Hough transform – in OpenCV
Hough for lines:
Probabilistic Hough for line segments:
Hough for circles:
vector<Vec2f> hough_lines;
HoughLines( binary_edge_image, hough_lines, 1, PI/200.0, 60);
vector<Vec4i> line_segments;
HoughLinesP( binary_edge_image, line_segments, 1.0,
PI/200.0, 20, 20, 5);
THIS IS THE START OF “Hough transform”.
The Hough will work well even with noisy or occluded shapes.
Example Apps: Lane detection, coin size evaluation, eye tracking, …
vector<Vec3f> circles;
HoughCircles( gray_image, circles, CV_HOUGH_GRADIENT,
2,20,300,20,5,15);
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

51. Hough Transform – Resolution of Hough Space
Circle
(a, b) could be anywhere up to r outside the image space
Can detect the circle centre to higher precision than the image points
Lines
–dist(0,0, M,N)  s  + dist(0,0, M,N)
-  θ  
Duplicated space!
Precision of s and θ application dependent
Partial circles may appear in the image whose centres would be outside the image. Often these are ignored.
Hough space for lines if taken in its full extent actually represents each line twice. This can be handy though as it allows discrimination of (black to white) vs. (white to black) lines. The precision of s and theta are application dependent but the higher the precision the slower the algorithm
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

52. Hough Transform – Efficient Implementation
Use edge orientations
Restrict the mapping into space
Use half the accumulator
For lines only
Multi-resolution
Process at a small resolution
Higher resolution to find accurate data
Problem
What if the size of the circle is unknown
3-D Accumulator??
If reliable edge orientation information is available and the edges are smooth (usually should be so), then edge orientation can be used to restict the Hough space which is mapped into by any edge point…
For lines we can consider only half the Hough space if we don’t care about the white-black, black-white thing. We have to be a bit clever when looking for maximum though as the Hough space then should be wrapped back on itself in an odd fashion.
For efficiency it is best to start processing at a small resolution and then after finding the maxima, generate new small higher resolution Hough spaces to obtain the maxima (and hence the equations) to more accuracy.
If the size of the circle isn’t known in advance we are faced with needing a 3-D accumulator (3rd dimension for the radius) which will cause lots of efficiency problems…
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

53. Hough Transform – Generalized
Consider an arbitrary shape
Define in terms of distance and angles from some reference xR
Distance r
Orientation 
Orientation  of line from xR through edge point
This will work for any number of (unconnected or connected) edge points.
Define a reference point ANYWHERE (inside or outside the shape)
For each edge point
determine the distance from the reference point
The orientation of the edge point
The orientation of the line from the reference point through the edge point
Note that multiple points can have the same orientation (as shown in the diagram)
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

54. Hough Transform – Generalized – Training
Training phase
Build up an R-table
For every  store (r, ) pairs
Have to quantise the orientation space into a discrete number of possible orientations. For each orientation a linked list of (r, ) pairs is maintained.
Two shapes and their associated R-tables. The edge orientations have been quantized quite
coarsely into 16 possibilities. In the case of the circle which has 32 boundary edge points each
orientation has two (r, a) pairs due to two of the points being quantized to each orientation. In the case
of the square again we are assuming 32 edge points and hence there are 8(r, a) pairs in four orthogonal
locations and no (r, a) pairs in the other orientations.
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014

55. Hough Transform – Generalized – Recognition
Recognition phase
Create an accumulator for xR
For every edge point
Determine its orientation 
Select a list of (r, ) pairs from the R-table
Determine the position of xR for each (, r, ) and increment the accumulator appropriately
Search for maximums
In our recognition phase we assume that the size and orientation are fixed. Hence we get a simple 2-D accumulator for the Hough space. In the book a 4-D accumulator is used allowing for both the size and the orientation of the shape to change.
Maybe draw some diagrams for standard shapes such as a square and a circle to help people understand…
Edges
Based on A Practical Introduction to Computer Vision with OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014