Exercise Class 11: Robust Tecuniques RANSAC, Hough Transform

Continuing Ex. Class 10: Feature matching ?

Feature matching Given a large amount of feature descriptors, there are many approaches to find correspondence between them: Exhaustive search for each feature in one image, look at all the other features in the other image(s) Hashing compute a short descriptor from each feature vector, or hash longer descriptors (randomly) Nearest neighbor techniques k-trees and their variants

What about outliers? Use a Robust Method This pair of images contains only small overlapping area. Any matching of point that is not contained in this area with other point is wrong. Moreover, not all of the interest points in the overlapping area were detected in both images. Also, there may be a wrong matching due to reasons like inappropriate measure, approximate NN etc.

What are robust methods? The goal of many algorithms is parametric fitting: extract parametric information from an image Extracting edges Extracting motion model from correspondence points In real images: inaccurate data Occlusions Noise Wrong feature extraction Data is likely to include multiple models / structures Goal of robust method: tolerate outliers points outliers Pseudo- outliers An important goal of many computer vision algorithms is to extract geometric information from an image, or from image sequences. Parametric models play a vital role in this and other activities in computer vision research. When engaged in parametric fitting in a computer vision context, it is important to recognize that data obtained from the image or image sequences may be inaccurate. It is almost unavoidable that data are contaminated (due to faulty feature extraction, sensor noise, segmentation errors, etc) and it is also likely that the data will include multiple structures.

Example: Robust Computation of 2D Homography from Hartley & Zisserman 500 Interest points per image 151 inliers 262 inliers 43 iterations… 117 outliers 268 assumed correspondences (Best match,SSD<20)

Simple Model Fit Problem Estimate a straight line fit to a set of 2D points Find the line that minimize the sum of squared perpendicular distances, subject to the condition that no valid point deviates from this line in more than t units. Actually we have 2 problems : Fit the line to the data Classify the data into inliers and outliers

Simple Model Fit Problem Estimate a straight line fit to a set of 2D points Classical techniques for parameter estimations : optimize (according to specific objective function) the fit of the model to all of the presented data. In such technique there is no internal mechanism for detection and rejection of gross errors Least Square (LS) error is not robust : one outlier point may cause a very big mistake

A good LS fit to a set of points. One outlier point leads to disaster. A least squares fit. The problem is the single point on the right; the error for that point is so large that it drags the line away from the other points (the total error is lower if these points are some way from the line, because then the point on the line is not so dramatically far away)

Another outlier, another disaster. Close-up of the poor fit to the “true” data.

Robust Norm We need to set s Parameters : unknown scale cost distance First way of handling the problem : define a more robust error function The issue here is that one wants to replace (distance)^2 with something that looks like distance^2 for small distances, and is about constant (bounded) for large distances. We use d^2/(d^2+s^2) for some parameter s, for different values of which the curve is plotted cost

Robust Norm cost distance The issue here is that one wants to replace (distance)^2 with something that looks like distance^2 for small distances, and is about constant for large distances. We use d^2/(d^2+s^2) for some parameter s, for different values of which the curve is plotted distance

The impact of tuning s Just right Too small Too large Here the parameter is too small, and the error for every point looks like distance^2 Here the parameter is too large, meaning that the error value is about constant for every point, and there is very little relationship between the line and the points Too small Too large

Simple Model Fit Problem Estimate a straight line fit to a set of 2D points How can we do better ? Assumption: Given the correct model, we can validate this model

RANSAC (RANdom SAmple Consensus) The idea: Sample each time 2 points randomly, and check the support of the resulted model Choose model with max support Support of the line: the number of points that lie within a distance threshold Select 2 points randomly. Those points define a line. The support of this line is measured by the number of points that lie within a distance threshold Repeat the selection process number of time and choose the line with the largest support. The idea is that if one of the points is outlier that line will not gain much support. In addition choosing the line with the larger number of support favorable better fit – for example it may be that 2 inliers (like point A,D) will also yield a possible line but not an optimal one. The last stage is to re-estimate the line from the set of inliers to improve the initial estimation of line In some other methods we would have tried to make model from many points and than eliminate outliers. Here we do it in the other way round : choose a few points, define a model and enlarge the data set with consistent points when possible

Iterate N times: At the end: RANSAC – The Algorithm Randomly sample S points from the data Solve the problem using the S points Check the solution using the rest of the points (the support) At the end: Use all the points which “agree” to compute the exact model.

Back to homography estimation RANSAC loop: Select four feature pairs (at random) Compute homography H (exact) Compute inliers where SSD(pi’, Hpi) < ε Keep largest set of inliers Re-compute least-squares H estimate on all of the inliers

Sufficient Number of Iterations (N) How Many Iterations are needed ? We want to ensure that in probability p at least one of the samples of S points is free from outliers (for example p=99%) w – inliers probability What is N ?

Sufficient Number of Iterations (N) p=0.99 w : inliers probability, : outlier probability Sample Size proportion of outliers e s 5% 10% 20% 25% 30% 40% 50% 2 3 5 6 7 11 17 4 9 19 35 13 34 72 12 26 57 146 16 24 37 97 293 8 20 33 54 163 588 44 78 272 1177 It is seen here that 2 points are the best choice as the required number of iterations is getting much larger as more samples are used in each step

Adaptively determining the number of Iterations e is often unknown a priori, so pick worst case, e.g. 50%, and adapt if more inliers are found, e.g. 80% would yield e=0.2 N=∞, sample_count =0 While N >sample_count repeat Choose a sample and count the number of inliers Set e=1-((number of inliers)/(total number of points)) Recompute N from e Increment the sample_count by 1 Terminate In many cases we do not know in advance the fraction of data outliers

RANSAC discussion Advantages: – General method suited for a wide range of model fitting problems – Easy to implement and easy to calculate its failure rate • Disadvantages: – Only handles a moderate percentage of outliers without cost blowing up – Many real problems have high rate of outliers (but sometimes selective choice of random subsets can help) In the next method presented we will be able to deal with larger percentage of outliers

Hough Transform

Hough Transform Finding Shapes • Straight Lines • Rectangles • Circles (proposed by Hough in ~1959) Finding Shapes • Straight Lines • Rectangles • Circles • Others More generally: A parametric voting

Applications Find signs in the images (rectangles, circles). 3D reconstruction of buildings (lines). Find the eye in a face image. Tracking More… The challenge: The image contains a lot of irrelevant information. There are partial occlusions. The image might be noisy

We will take the example of edge detection as main example for Hough transform application. The initial result of edge detection may be in many times noisy and hard to analyze. We are looking for ‘global’ operation that will recognize the line, not based on neighborhood information

Edge Detection Review INPUT IMAGE Vertical [-1 0 1] 2) Edge Enhancement 1) Noise Smoothing Horizontal [-1 0 1]T 3)Threshold EDGE IMAGE “GRADIENT” IMAGE

How do we detect lines from Edge Images? Raw Image Edge Image Segmented Lanes

How do we detect lines from Edge Images? Given an edge image, we would like to group sets of edge pixels to identify dominant lines in an image. One possibility is template matching in the edge image Generate hypothetical line “kernel” and convolve with the image Requires a large set of masks to accurately localize lines, which is computationally expensive We will instead perform an image transformation where edge pixels vote for lines Eventually reduces the problem to vote thresholding Each 2 points define a line Check for all other points: are they on the same line? Problem : n^2 lines, n^3 tests

The Parameters Space The line equation : y=ax+b (slope-intercept representation) Each line can be represented with 2 parameters: (a,b) If we define a 2D parameters space, each line is a point in this space. For each point on this line (x,y), there are infinite number of lines that goes through it in the parameters space: b=-xa+y

The Duality Property b y a x b y a x A point is mapped to a line, and a line is mapped to a point. Hough Image b y a x Hough Image Take all edge features. Build a table for the parameters space. Each edge point will cast a line in parameters space – points in the parameters space with higher scores are lines in the image How can We use this duality? b y a x

Hough Transform: use a voting scheme Apply an edge detector on the image (gradient + threshold, or using Canny’s method) Prepare a table h(a,b) for a,b in a certain range. Loop over the image: for each pixel (x,y) on an edge, vote for the cells (a,b) which satisfy the equation y = ax + b (meaning, increase the cell accumulator- h(a,b)++) Choose the cell with the maximal value (or the cells which pass a given threshold)

Example: Slope-Intercept Representation Original image Edge Map Take two strongest lines Hough (before threshold)

Issues to think of… Representation: Vertical lines are not covered by the representation y=ax+b. Range: Selecting the range of values for the table (a and b are not bounded!) Quantization: choosing a grid. Thresholding: Similar lines will get similar scores.

A better representation Use a polar representation instead: - The line normal d - The distance from (0,0) Hough Image d y x 

In the polar representation… d Note the two intersection points in the parameters space: these are (d, theta) and (-d, pie + theta) that represent the same line with normal (d, theta).

Polar Representation Example 1. Original image + edges 3. Image + detected lines 2. Hough (before threshold)

Range and Quantization In the polar representation it is easier to determine the range: The values of d and of  are now bounded ! The resolution can be application dependent. We have natural units (pixels for d, and degrees for ) Quantization: Option 1: Vote for the “nearest” cell (f.g – for each  take the nearest d). Option 2: Perform a weighted vote.

Smoothing b y a x b y a x Hough Image Without smoothing Hough Image With smoothing b y a x

Making the Decision Q. How can we find the two most dominant lines in the image ? The problem: if (r,) is the cell with the maximal value, then (r+,  +) will get a high score too.

Making the Decision(cont’) Solution 1: Search only for local maxima. Solution 2: “Back Mapping”- Perform a second vote, where each point votes only for the best line passing through it.

b y a x b y a x Using more image features. Example – using the local gradient Hough Image not using gradient b y a x Hough Image using gradient b y a x

b y a x b y a x Using more image features. Example – using the local gradient Hough Image not using gradient b y a x Hough Image using gradient b y a x

Detecting Complex Shapes – Circles Naïve solution: Construct a table h(a,b,r). For each pixel (x,y) which is edge in the image, vote for all the cells satisfying: Problem: The size of the table is O(N3). The voting for each pixels takes: O(N2). Solution: Use the gradient information - (Solve for the center (a,b) and then for r)

Circle Example With no orientation, each token (point) votes for all possible circles. With orientation, each token can vote for a smaller number of circles.

Finding Coins Original Edges (note noise)

Finding Coins (Continued) Penny Quarters

Finding Coins (Continued) Note that because the quarters and penny are different sizes, a different Hough transform (with separate accumulators) was used for each circle size. Coin finding sample images from: Vivik Kwatra