1. Linear Filtering
CS 678
Spring 2018

2. Outline Linear filtering Box filter vs. Gaussian filter Median filter
Gaussian and Laplacian pyramids
Sharpening images
Some slides from Lazebnik

3. Motivation: Noise reduction
Given a camera and a still scene, how can you reduce noise?
Answer: take lots of images, average them
Take lots of images and average them!
What’s the next best thing?
Source: S. Seitz

4. Moving average
Let’s replace each pixel with a weighted average of its neighborhood
The weights are called the filter kernel
What are the weights for a 3x3 moving average? 1
“box filter”
Source: D. Lowe

5. Defining convolution f
Let f be the image and g be the kernel. The output of convolving f with g is denoted f * g. f
Convention: kernel is “flipped”
MATLAB: conv2 vs. filter2 (also imfilter)
Source: F. Durand

6. Key properties Linearity: filter(f1 + f2 ) = filter(f1) + filter(f2)
Shift invariance: same behavior regardless of pixel location: filter(shift(f)) = shift(filter(f))
Theoretical result: any linear shift-invariant operator can be represented as a convolution

7. Properties in more detail
Commutative: a * b = b * a
Conceptually no difference between filter and signal
Associative: a * (b * c) = (a * b) * c
Often apply several filters one after another: (((a * b1) * b2) * b3)
This is equivalent to applying one filter: a * (b1 * b2 * b3)
Distributes over addition: a * (b + c) = (a * b) + (a * c)
Scalars factor out: ka * b = a * kb = k (a * b)
Identity: unit impulse e = […, 0, 0, 1, 0, 0, …], a * e = a

8. Annoying details What is the size of the output?
MATLAB: filter2(g, f, shape)
shape = ‘full’: output size is sum of sizes of f and g
shape = ‘same’: output size is same as f
shape = ‘valid’: output size is difference of sizes of f and g
full
same
valid g g g g f f f g g g g g g g g

9. Annoying details What about near the edge?
the filter window falls off the edge of the image
need to extrapolate
methods:
clip filter (black)
wrap around
copy edge
reflect across edge
Source: S. Marschner

10. Annoying details What about near the edge?
the filter window falls off the edge of the image
need to extrapolate
methods (MATLAB):
clip filter (black): imfilter(f, g, 0)
wrap around: imfilter(f, g, ‘circular’)
copy edge: imfilter(f, g, ‘replicate’)
reflect across edge: imfilter(f, g, ‘symmetric’)
Source: S. Marschner

11. Practice with linear filters1 ?
Original
Source: D. Lowe

12. Practice with linear filters1
Original
Filtered
(no change)
Source: D. Lowe

13. Practice with linear filters1 ?
Original
Source: D. Lowe

14. Practice with linear filters1
Original
Shifted left
By 1 pixel
Source: D. Lowe

15. Practice with linear filters? 1
Original
Source: D. Lowe

16. Practice with linear filters1
Original
Blur (with a
box filter)
Source: D. Lowe

17. Practice with linear filters- 2 1 ?
(Note that filter sums to 1)
Original
Source: D. Lowe

18. Practice with linear filters- 2 1
Original
Sharpening filter
Accentuates differences with local average
Source: D. Lowe

19. Sharpening
Source: D. Lowe

20. Smoothing with box filter revisited
Smoothing with an average actually doesn’t compare at all well with a defocused lens
Most obvious difference is that a single point of light viewed in a defocused lens looks like a fuzzy blob; but the averaging process would give a little square
I always walk through the argument on the left rather carefully; it gives some insight into the
significance of impulse responses or point spread functions.
Source: D. Forsyth

21. Smoothing with box filter revisited
Smoothing with an average actually doesn’t compare at all well with a defocused lens
Most obvious difference is that a single point of light viewed in a defocused lens looks like a fuzzy blob; but the averaging process would give a little square
Better idea: to eliminate edge effects, weight contribution of neighborhood pixels according to their closeness to the center, like so:
I always walk through the argument on the left rather carefully; it gives some insight into the
significance of impulse responses or point spread functions.
“fuzzy blob”

22. Gaussian Kernel
5 x 5,  = 1
Constant factor at front makes volume sum to 1 (can be ignored, as we should re-normalize weights to sum to 1 in any case)
Source: C. Rasmussen

23. Choosing kernel width
Gaussian filters have infinite support, but discrete filters use finite kernels
Source: K. Grauman

24. Choosing kernel width
Rule of thumb: set filter half-width to about 3 σ

25. Example: Smoothing with a Gaussian

26. Mean vs. Gaussian filtering

27. Gaussian filters
Remove “high-frequency” components from the image (low-pass filter)
Convolution with self is another Gaussian
So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have
Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ√2
Separable kernel
Factors into product of two 1D Gaussians
Linear vs. quadratic in mask size
Source: K. Grauman

28. Separability of the Gaussian filter
Source: D. Lowe

29. Separability example * 2D convolution (center location only)
The filter factors into a product of 1D filters: * =
Perform convolution along rows:
Followed by convolution along the remaining column:
Source: K. Grauman

30. Separability
Why is separability useful in practice?

31. Noise
Salt and pepper noise: contains random occurrences of black and white pixels
Impulse noise: contains random occurrences of white pixels
Gaussian noise: variations in intensity drawn from a Gaussian normal distribution
Salt and pepper and impulse noise can be due to transmission errors (e.g., from deep space probe), dead CCD pixels, specks on lens
Source: S. Seitz

32. Gaussian noise Mathematical model: sum of many independent factors
Good for small standard deviations
Assumption: independent, zero-mean noise
Source: M. Hebert

33. Reducing Gaussian noise
Smoothing with larger standard deviations suppresses noise, but also blurs the image

34. Reducing salt-and-pepper noise
3x3
5x5
7x7
What’s wrong with the results?

35. Alternative idea: Median filtering
A median filter operates over a window by selecting the median intensity in the window
Better at salt’n’pepper noise
Not convolution: try a region with 1’s and a 2, and then 1’s and a 3
Is median filtering linear?
Source: K. Grauman

36. Median filter
What advantage does median filtering have over Gaussian filtering?
Robustness to outliers
Source: K. Grauman

37. Salt-and-pepper noise
Median filter
Salt-and-pepper noise
Median filtered
MATLAB: medfilt2(image, [h w])
Source: M. Hebert

38. Median vs. Gaussian filtering
3x3
5x5
7x7
Gaussian
Median

39. Image Pyramids Multi-resolution of images Gaussian pyramid
Laplacian pyramid

40. Scaled representations
Big bars (resp. spots, hands, etc.) and little bars are both interesting
Stripes and hairs, say
Inefficient to detect big bars with big filters
And there is superfluous detail in the filter kernel
Alternative:
Apply filters of fixed size to images of different sizes
Typically, a collection of images whose edge length changes by a factor of 2 (or root 2)
This is a pyramid (or Gaussian pyramid) by visual analogy

41. Gaussian Pyramids Very useful for representing images
Image Pyramid is built by using multiple copies of image at different scales.
Each level in the pyramid is ¼ of the size of previous level
The highest level is of the lowest resolution
The lowest level is of the highest resolution

42. Gaussian Pyramids

43. A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose

44. Aliasing Can’t shrink an image by taking every second pixel
If we do, characteristic errors appear
Common phenomenon
Wagon wheels rolling the wrong way in movies
Checkerboards misrepresented in ray tracing
Striped shirts look funny on colour television

45. Resample the checkerboard by taking one sample at each circle
Resample the checkerboard by taking one sample at each circle. In the case of the top left board, new representation is reasonable.
Top right also yields a reasonable representation. Bottom left is all black (dubious) and bottom right has checks that are too big.

46. The Gaussian pyramid Smooth with gaussians, because Synthesis
a gaussian*gaussian=another gaussian
Synthesis
smooth and sample
gaussians are low pass filters

47. Reduce (1D)

48. Reduce (1D)

49. Convolution Kernel
Symmetric
Sum of mask should be 1

50. Convolution Kernel
All nodes at a given level must contribute the same total weight to the nodes at the next higher level c c b b a

51. Convolution Kernel
a=0.4 GAUSSIAN, a=0.5 TRIANGULAR

52. What about 2D? Separability of Gaussian
Requires n2 k2 multiplications for n by n image and
k by k kernel.
Requires 2kn2 multiplications for n by n image and
k by k kernel.

53. Algorithm Apply 1D mask to alternate pixels along each row of image.
Apply 1D mask to alternate pixels along each column of resultant image from previous step.
Down sample

54. The Laplacian Pyramid Similar to edge detected images
Most pixels are zero
Can be used in image compression

55. Constructing Laplacian Pyramid
Compute Gaussian pyramid
Compute Laplacian pyramid as follows:

56. Reconstructing Image

57. The Laplacian Pyramid

58. This is a Gaussian pyramid

59. And this a Laplacian pyramid
And this a Laplacian pyramid. Notice that the lowest layer is the same in each case,
and that if I upsample the lowest layer and add to the next, I get the Gauss layer.
Notice also that each layer shows detail at a particular scale --- these are, basically, bandpass filtered versions of the image.

60. The Laplacian Pyramid Analysis Synthesis
preserve difference between upsampled Gaussian pyramid level and Gaussian pyramid level
band pass filter - each level represents spatial frequencies (largely) unrepresented at other levels
Synthesis
reconstruct Gaussian pyramid, take the finest layer

61. Sharpening revisited What does blurring take away? Let’s add it back:
original
smoothed (5x5)
detail = –
Let’s add it back:
original
detail
+ α
sharpened =

62. unit impulse (identity)
Unsharp mask filter
Gaussian
unit impulse
Laplacian of Gaussian
image
blurred image
unit impulse (identity)
f + a(f - f * g) = (1+a)f-af*g = f*((1+a)e-g)

63. Application: Hybrid Images
A. Oliva, A. Torralba, P.G. Schyns, “Hybrid Images,” SIGGRAPH 2006