1. Computer Vision - A Modern Approach
Linear Filters
General process:
Form new image whose pixels are a weighted sum of original pixel values, using the same set of weights at each point.
Properties
Output is a linear function of the input
Output is a shift-invariant function of the input (i.e. shift the input image two pixels to the left, the output is shifted two pixels to the left)
Example: smoothing by averaging
form the average of pixels in a neighbourhood
Example: smoothing with a Gaussian
form a weighted average of pixels in a neighbourhood
Example: finding a derivative
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

2. Computer Vision - A Modern Approach
Convolution
Represent these weights as an image, H
H is usually called the kernel
Operation is called convolution
it’s associative
Result is:
Notice weird order of indices
all examples can be put in this form
it’s a result of the derivation expressing any shift-invariant linear operator as a convolution.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

3. Example: Smoothing by Averaging
Here is the point to introduce some visual “notation”. I’ve given the kernel as an image on the top.
Usually, these images are white for the largest value, and black for the smallest (which is zero in
this case). The point of this pair of images is that if you convolve the one on the left with the kernel
shown above you get the one on the right, which is nothing like a smoothed version of the one on the left
(there’s some fairly aggressive ringing). Explain the ringing qualitatively in terms of the boundaries of
the kernel, and point out that the effect would disappear if there weren’t sharp edges in the kernel.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

4. Smoothing with a Gaussian
Smoothing with an average actually doesn’t compare at all well with a defocussed lens
Most obvious difference is that a single point of light viewed in a defocussed lens looks like a fuzzy blob; but the averaging process would give a little square.
I always walk through the argument on the left rather carefully; it gives some insight into the
significance of impulse responses or point spread functions.
A Gaussian gives a good model of a fuzzy blob
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

5. Computer Vision - A Modern Approach
An Isotropic Gaussian
The picture shows a smoothing kernel proportional to
(which is a reasonable model of a circularly symmetric fuzzy blob)
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

6. Smoothing with a Gaussian
You want to point out the absence of ringing effects here.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

7. Differentiation and convolution
Recall
Now this is linear and shift invariant, so must be the result of a convolution.
We could approximate this as
(which is obviously a convolution; it’s not a very good way to do things, as we shall see)
I tend not to prove “Now this is linear and shift invariant, so must be the result of a convolution” but
leave it for people to look up in the chapter.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

8. Computer Vision - A Modern Approach
Finite differences
Now the figure on the right is signed; darker is negative, lighter is positive, and mid grey is zero.
I always ask 1) which derivative (y or x) is this? 2) Have I got the sign of the kernel right? (i.e.
is it d/dx or -d/dx).
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

9. Computer Vision - A Modern Approach
Noise
Simplest noise model
independent stationary additive Gaussian noise
the noise value at each pixel is given by an independent draw from the same normal probability distribution
Issues
this model allows noise values that could be greater than maximum camera output or less than zero
for small standard deviations, this isn’t too much of a problem - it’s a fairly good model
independence may not be justified (e.g. damage to lens)
may not be stationary (e.g. thermal gradients in the ccd)
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

10. Computer Vision - A Modern Approach
sigma=1
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

11. Computer Vision - A Modern Approach
sigma=16
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

12. Finite differences and noise
Finite difference filters respond strongly to noise
obvious reason: image noise results in pixels that look very different from their neighbours
Generally, the larger the noise the stronger the response
What is to be done?
intuitively, most pixels in images look quite a lot like their neighbours
this is true even at an edge; along the edge they’re similar, across the edge they’re not
suggests that smoothing the image should help, by forcing pixels different to their neighbours (=noise pixels?) to look more like neighbours
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

13. Finite differences responding to noise
Increasing noise ->
(this is zero mean additive gaussian noise)
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

14. The response of a linear filter to noise
Do only stationary independent additive Gaussian noise with zero mean (non-zero mean is easily dealt with)
Mean:
output is a weighted sum of inputs
so we want mean of a weighted sum of zero mean normal random variables
must be zero
Variance:
recall
variance of a sum of random variables is sum of their variances
variance of constant times random variable is constant^2 times variance
then if s is noise variance and kernel is K, variance of response is
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

15. Filter responses are correlated
over scales similar to the scale of the filter
Filtered noise is sometimes useful
looks like some natural textures, can be used to simulate fire, etc.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

16. Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

17. Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

18. Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

19. Smoothing reduces noise
Generally expect pixels to “be like” their neighbours
surfaces turn slowly
relatively few reflectance changes
Generally expect noise processes to be independent from pixel to pixel
Implies that smoothing suppresses noise, for appropriate noise models
Scale
the parameter in the symmetric Gaussian
as this parameter goes up, more pixels are involved in the average
and the image gets more blurred
and noise is more effectively suppressed
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

20. Computer Vision - A Modern Approach
The effects of smoothing
Each row shows smoothing
with gaussians of different
width; each column shows
different realisations of
an image of gaussian noise.
There are two notions of gaussian here, related only by a vague analogy. I did this slide like this
to put them clearly in apposition, because this point always confused me. There’s no particular
reason that gaussian noise should attract smoothing with a gaussian kernel, just an irritating coincidence.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

21. Computer Vision - A Modern Approach
Gradients and edges
Points of sharp change in an image are interesting:
change in reflectance
change in object
change in illumination
noise
Sometimes called edge points
General strategy
determine image gradient
now mark points where gradient magnitude is particularly large wrt neighbours (ideally, curves of such points).
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

22. Computer Vision - A Modern Approach
There are three major issues:
1) The gradient magnitude at different scales is different; which should
we choose?
2) The gradient magnitude is large along thick trail; how
do we identify the significant points?
3) How do we link the relevant points up into curves?
Figures show gradient magnitude of zebra at two different scales
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

23. Smoothing and Differentiation
Issue: noise
smooth before differentiation
two convolutions to smooth, then differentiate?
actually, no - we can use a derivative of Gaussian filter
because differentiation is convolution, and convolution is associative
Important to point out that the derivative of gaussian kernels I’ve illustrated “look like” the effects
that they’re trying to identify.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

24. Computer Vision - A Modern Approach
1 pixel
3 pixels
7 pixels
The scale of the smoothing filter affects derivative estimates, and also
the semantics of the edges recovered.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

25. Computer Vision - A Modern Approach
We wish to mark points along the curve where the magnitude is biggest.
We can do this by looking for a maximum along a slice normal to the curve
(non-maximum suppression). These points should form a curve. There are
then two algorithmic issues: at which point is the maximum, and where is the
next one?
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

26. Computer Vision - A Modern Approach
Non-maximum
suppression
At q, we have a maximum if the value is larger than those at both p and at r. Interpolate to get these values.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

27. Computer Vision - A Modern Approach
Predicting
the next
edge point
Assume the marked point is an edge point. Then we construct the tangent to the edge curve (which is normal to the gradient at that point) and use this to predict the next points (here either r or s).
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

28. Computer Vision - A Modern Approach
Remaining issues
Check that maximum value of gradient value is sufficiently large
drop-outs? use hysteresis
use a high threshold to start edge curves and a low threshold to continue them.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

29. Computer Vision - A Modern Approach
Notice
Something nasty is happening at corners
Scale affects contrast
Edges aren’t bounding contours
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

30. Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

31. Computer Vision - A Modern Approach
fine scale
high
threshold
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

32. Computer Vision - A Modern Approach
coarse
scale,
high
threshold
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

33. Computer Vision - A Modern Approach
coarse
scale
low
threshold
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

34. Computer Vision - A Modern Approach
Filters are templates
Applying a filter at some point can be seen as taking a dot-product between the image and some vector
Filtering the image is a set of dot products
Insight
filters look like the effects they are intended to find
filters find effects they look like
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

35. Normalized correlation
Think of filters of a dot product
now measure the angle
i.e normalised correlation output is filter output, divided by root sum of squares of values over which filter lies
Tricks:
ensure that filter has a zero response to a constant region (helps reduce response to irrelevant background)
subtract image average when computing the normalizing constant (i.e. subtract the image mean in the neighbourhood)
absolute value deals with contrast reversal
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

36. Computer Vision - A Modern Approach
Positive responses
Zero mean image, -1:1 scale
Zero mean image, -max:max scale
The filter is the little block in the top left hand corner. Notice this is a fair spot-detector.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

37. Computer Vision - A Modern Approach
Positive responses
Zero mean image, -1:1 scale
Zero mean image, -max:max scale
The filter is the little block in the top left hand corner. Notice this is a fair bar-detector.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

38. Computer Vision - A Modern Approach
This is a figure from the book, figure Read the caption for the story!
Figure from “Computer Vision for Interactive Computer Graphics,” W.Freeman et al, IEEE Computer Graphics and Applications, 1998 copyright 1998, IEEE
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

39. The Laplacian of Gaussian
Another way to detect an extremal first derivative is to look for a zero second derivative
Appropriate 2D analogy is rotation invariant
the Laplacian
Bad idea to apply a Laplacian without smoothing
smooth with Gaussian, apply Laplacian
this is the same as filtering with a Laplacian of Gaussian filter
Now mark the zero points where there is a sufficiently large derivative, and enough contrast
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

40. Computer Vision - A Modern Approach
sigma=4
contrast=1
LOG zero crossings
contrast=4
sigma=2
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

41. Computer Vision - A Modern Approach
We still have unfortunate behaviour
at corners
This is the usual story; if you find edges as a level crossing of a function, then trihedral vertices
are going to cause trouble. Also, notice the bulges at the corners.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

42. Orientation representations
The gradient magnitude is affected by illumination changes
but it’s direction isn’t
We can describe image patches by the swing of the gradient orientation
Important types:
constant window
small gradient mags
edge window
few large gradient mags in one direction
flow window
many large gradient mags in one direction
corner window
large gradient mags that swing
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

43. Computer Vision - A Modern Approach
Representing Windows
Types
constant
small eigenvalues
Edge
one medium, one small
Flow
one large, one small
corner
two large eigenvalues
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

44. Computer Vision - A Modern Approach
On the left, I’ve plotted the gradients on top of the image. It’s hard to see what’s going on,
which is why there’s more detail in the next slide.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

45. Computer Vision - A Modern Approach
Here we see gradient arrows at each pixel of a detailed view. In some regions, the arrows swing,
which should be detectable using our analysis of H above
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

46. Computer Vision - A Modern Approach
Here H was averaged over a small window, and I’m plotting ellipses x^t H^(-1) x=eps (so if
H has one big and one small eigenvalue, I’ll see a heavily oriented ellipse; if H has two small
eigenvalues, I’ll see a tiny circle, etc.)
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth

47. Computer Vision - A Modern Approach
The average is now over a bigger window.
Computer Vision - A Modern Approach
Set: Linear Filters
Slides by D.A. Forsyth