Computer Vision - A Modern Approach Linear Filters General process: Form new image whose pixels are a weighted sum of original pixel values, using the same set of weights at each point. Properties Output is a linear function of the input Output is a shift-invariant function of the input (i.e. shift the input image two pixels to the left, the output is shifted two pixels to the left) Example: smoothing by averaging form the average of pixels in a neighbourhood Example: smoothing with a Gaussian form a weighted average of pixels in a neighbourhood Example: finding a derivative Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Convolution Represent these weights as an image, H H is usually called the kernel Operation is called convolution it’s associative Result is: Notice weird order of indices all examples can be put in this form it’s a result of the derivation expressing any shift-invariant linear operator as a convolution. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Example: Smoothing by Averaging Here is the point to introduce some visual “notation”. I’ve given the kernel as an image on the top. Usually, these images are white for the largest value, and black for the smallest (which is zero in this case). The point of this pair of images is that if you convolve the one on the left with the kernel shown above you get the one on the right, which is nothing like a smoothed version of the one on the left (there’s some fairly aggressive ringing). Explain the ringing qualitatively in terms of the boundaries of the kernel, and point out that the effect would disappear if there weren’t sharp edges in the kernel. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Smoothing with a Gaussian Smoothing with an average actually doesn’t compare at all well with a defocussed lens Most obvious difference is that a single point of light viewed in a defocussed lens looks like a fuzzy blob; but the averaging process would give a little square. I always walk through the argument on the left rather carefully; it gives some insight into the significance of impulse responses or point spread functions. A Gaussian gives a good model of a fuzzy blob Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach An Isotropic Gaussian The picture shows a smoothing kernel proportional to (which is a reasonable model of a circularly symmetric fuzzy blob) Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Smoothing with a Gaussian You want to point out the absence of ringing effects here. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Differentiation and convolution Recall Now this is linear and shift invariant, so must be the result of a convolution. We could approximate this as (which is obviously a convolution; it’s not a very good way to do things, as we shall see) I tend not to prove “Now this is linear and shift invariant, so must be the result of a convolution” but leave it for people to look up in the chapter. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Finite differences Now the figure on the right is signed; darker is negative, lighter is positive, and mid grey is zero. I always ask 1) which derivative (y or x) is this? 2) Have I got the sign of the kernel right? (i.e. is it d/dx or -d/dx). Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Noise Simplest noise model independent stationary additive Gaussian noise the noise value at each pixel is given by an independent draw from the same normal probability distribution Issues this model allows noise values that could be greater than maximum camera output or less than zero for small standard deviations, this isn’t too much of a problem - it’s a fairly good model independence may not be justified (e.g. damage to lens) may not be stationary (e.g. thermal gradients in the ccd) Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach sigma=1 Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach sigma=16 Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Finite differences and noise Finite difference filters respond strongly to noise obvious reason: image noise results in pixels that look very different from their neighbours Generally, the larger the noise the stronger the response What is to be done? intuitively, most pixels in images look quite a lot like their neighbours this is true even at an edge; along the edge they’re similar, across the edge they’re not suggests that smoothing the image should help, by forcing pixels different to their neighbours (=noise pixels?) to look more like neighbours Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Finite differences responding to noise Increasing noise -> (this is zero mean additive gaussian noise) Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

The response of a linear filter to noise Do only stationary independent additive Gaussian noise with zero mean (non-zero mean is easily dealt with) Mean: output is a weighted sum of inputs so we want mean of a weighted sum of zero mean normal random variables must be zero Variance: recall variance of a sum of random variables is sum of their variances variance of constant times random variable is constant^2 times variance then if s is noise variance and kernel is K, variance of response is Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Filter responses are correlated over scales similar to the scale of the filter Filtered noise is sometimes useful looks like some natural textures, can be used to simulate fire, etc. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Smoothing reduces noise Generally expect pixels to “be like” their neighbours surfaces turn slowly relatively few reflectance changes Generally expect noise processes to be independent from pixel to pixel Implies that smoothing suppresses noise, for appropriate noise models Scale the parameter in the symmetric Gaussian as this parameter goes up, more pixels are involved in the average and the image gets more blurred and noise is more effectively suppressed Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach The effects of smoothing Each row shows smoothing with gaussians of different width; each column shows different realisations of an image of gaussian noise. There are two notions of gaussian here, related only by a vague analogy. I did this slide like this to put them clearly in apposition, because this point always confused me. There’s no particular reason that gaussian noise should attract smoothing with a gaussian kernel, just an irritating coincidence. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Gradients and edges Points of sharp change in an image are interesting: change in reflectance change in object change in illumination noise Sometimes called edge points General strategy determine image gradient now mark points where gradient magnitude is particularly large wrt neighbours (ideally, curves of such points). Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach There are three major issues: 1) The gradient magnitude at different scales is different; which should we choose? 2) The gradient magnitude is large along thick trail; how do we identify the significant points? 3) How do we link the relevant points up into curves? Figures show gradient magnitude of zebra at two different scales Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Smoothing and Differentiation Issue: noise smooth before differentiation two convolutions to smooth, then differentiate? actually, no - we can use a derivative of Gaussian filter because differentiation is convolution, and convolution is associative Important to point out that the derivative of gaussian kernels I’ve illustrated “look like” the effects that they’re trying to identify. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach 1 pixel 3 pixels 7 pixels The scale of the smoothing filter affects derivative estimates, and also the semantics of the edges recovered. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach We wish to mark points along the curve where the magnitude is biggest. We can do this by looking for a maximum along a slice normal to the curve (non-maximum suppression). These points should form a curve. There are then two algorithmic issues: at which point is the maximum, and where is the next one? Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Non-maximum suppression At q, we have a maximum if the value is larger than those at both p and at r. Interpolate to get these values. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Predicting the next edge point Assume the marked point is an edge point. Then we construct the tangent to the edge curve (which is normal to the gradient at that point) and use this to predict the next points (here either r or s). Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Remaining issues Check that maximum value of gradient value is sufficiently large drop-outs? use hysteresis use a high threshold to start edge curves and a low threshold to continue them. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Notice Something nasty is happening at corners Scale affects contrast Edges aren’t bounding contours Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach fine scale high threshold Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach coarse scale, high threshold Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach coarse scale low threshold Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Filters are templates Applying a filter at some point can be seen as taking a dot-product between the image and some vector Filtering the image is a set of dot products Insight filters look like the effects they are intended to find filters find effects they look like Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Normalized correlation Think of filters of a dot product now measure the angle i.e normalised correlation output is filter output, divided by root sum of squares of values over which filter lies Tricks: ensure that filter has a zero response to a constant region (helps reduce response to irrelevant background) subtract image average when computing the normalizing constant (i.e. subtract the image mean in the neighbourhood) absolute value deals with contrast reversal Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Positive responses Zero mean image, -1:1 scale Zero mean image, -max:max scale The filter is the little block in the top left hand corner. Notice this is a fair spot-detector. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Positive responses Zero mean image, -1:1 scale Zero mean image, -max:max scale The filter is the little block in the top left hand corner. Notice this is a fair bar-detector. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach This is a figure from the book, figure 7.16. Read the caption for the story! Figure from “Computer Vision for Interactive Computer Graphics,” W.Freeman et al, IEEE Computer Graphics and Applications, 1998 copyright 1998, IEEE Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

The Laplacian of Gaussian Another way to detect an extremal first derivative is to look for a zero second derivative Appropriate 2D analogy is rotation invariant the Laplacian Bad idea to apply a Laplacian without smoothing smooth with Gaussian, apply Laplacian this is the same as filtering with a Laplacian of Gaussian filter Now mark the zero points where there is a sufficiently large derivative, and enough contrast Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach sigma=4 contrast=1 LOG zero crossings contrast=4 sigma=2 Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach We still have unfortunate behaviour at corners This is the usual story; if you find edges as a level crossing of a function, then trihedral vertices are going to cause trouble. Also, notice the bulges at the corners. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Orientation representations The gradient magnitude is affected by illumination changes but it’s direction isn’t We can describe image patches by the swing of the gradient orientation Important types: constant window small gradient mags edge window few large gradient mags in one direction flow window many large gradient mags in one direction corner window large gradient mags that swing Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Representing Windows Types constant small eigenvalues Edge one medium, one small Flow one large, one small corner two large eigenvalues Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach On the left, I’ve plotted the gradients on top of the image. It’s hard to see what’s going on, which is why there’s more detail in the next slide. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Here we see gradient arrows at each pixel of a detailed view. In some regions, the arrows swing, which should be detectable using our analysis of H above Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach Here H was averaged over a small window, and I’m plotting ellipses x^t H^(-1) x=eps (so if H has one big and one small eigenvalue, I’ll see a heavily oriented ellipse; if H has two small eigenvalues, I’ll see a tiny circle, etc.) Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth

Computer Vision - A Modern Approach The average is now over a bigger window. Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth