1. Templates, Image Pyramids, and Filter Banks Computer Vision James Hays, Brown Slides: Hoiem and others

2. Reminder Project 1 due Friday

3. Fourier Bases This change of basis is the Fourier Transform Teases away fast vs. slow changes in the image.

4. Fourier Bases in Matlab, check out: imagesc(log(abs(fftshift(fft2(im)))));

5. Man-made Scene

6. Can change spectrum, then reconstruct

7. Low and High Pass filtering

8. What is the spatial representation of the hard cutoff in the frequency domain? Sinc Filter Frequency Domain Spatial Domain

9. Review 1.Match the spatial domain image to the Fourier magnitude image 1 54 A 3 2 C B D E

10. Today’s class Template matching Image Pyramids Filter banks and texture

11. Template matching Goal: find in image Main challenge: What is a good similarity or distance measure between two patches? – Correlation – Zero-mean correlation – Sum Square Difference – Normalized Cross Correlation

12. Matching with filters Goal: find in image Method 0: filter the image with eye patch Input Filtered Image What went wrong? f = image g = filter

13. Matching with filters Goal: find in image Method 1: filter the image with zero-mean eye Input Filtered Image (scaled) Thresholded Image True detections False detections mean of f

14. Matching with filters Goal: find in image Method 2: SSD Input 1- sqrt(SSD) Thresholded Image True detections

15. Matching with filters Can SSD be implemented with linear filters?

16. Matching with filters Goal: find in image Method 2: SSD Input 1- sqrt(SSD) What’s the potential downside of SSD?

17. Matching with filters Goal: find in image Method 3: Normalized cross-correlation Matlab: normxcorr2(template, im) mean image patch mean template

18. Matching with filters Goal: find in image Method 3: Normalized cross-correlation Input Normalized X-Correlation Thresholded Image True detections

19. Matching with filters Goal: find in image Method 3: Normalized cross-correlation Input Normalized X-Correlation Thresholded Image True detections

20. Q: What is the best method to use? A: Depends SSD: faster, sensitive to overall intensity Normalized cross-correlation: slower, invariant to local average intensity and contrast But really, neither of these baselines are representative of modern recognition.

21. Q: What if we want to find larger or smaller eyes? A: Image Pyramid

22. Review of Sampling Low-Pass Filtered Image Image Gaussian Filter Sample Low-Res Image

23. Gaussian pyramid Source: Forsyth

24. Template Matching with Image Pyramids Input: Image, Template 1.Match template at current scale 2.Downsample image 3.Repeat 1-2 until image is very small 4.Take responses above some threshold, perhaps with non-maxima suppression

25. Coarse-to-fine Image Registration 1.Compute Gaussian pyramid 2.Align with coarse pyramid 3.Successively align with finer pyramids – Search smaller range Why is this faster? Are we guaranteed to get the same result?

26. 2D edge detection filters is the Laplacian operator: Laplacian of Gaussian Gaussianderivative of Gaussian

27. Laplacian filter Gaussian unit impulse Laplacian of Gaussian Source: Lazebnik

28. Computing Gaussian/Laplacian Pyramid http://sepwww.stanford.edu/~morgan/texturematch/paper_html/node3.html Can we reconstruct the original from the laplacian pyramid?

29. Laplacian pyramid Source: Forsyth

30. Why do we get different, distance-dependent interpretations of hybrid images? ? Perception

31. Hybrid Image

32. Hybrid Image in Laplacian Pyramid High frequency  Low frequency

33. Question Can you align the images using the FFT?

34. Image representation Pixels: great for spatial resolution, poor access to frequency Fourier transform: great for frequency, not for spatial info Pyramids/filter banks: balance between spatial and frequency information

35. Major uses of image pyramids Compression Object detection – Scale search – Features Detecting stable interest points Registration – Course-to-fine

36. Application: Representing Texture Source: Forsyth

37. Texture and Material http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/

38. Texture and Orientation http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/

39. Texture and Scale http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/

40. What is texture? Regular or stochastic patterns caused by bumps, grooves, and/or markings

41. How can we represent texture? Compute responses of blobs and edges at various orientations and scales

42. Overcomplete representation: filter banks LM Filter Bank Code for filter banks: www.robots.ox.ac.uk/~vgg/research/texclass/filters.html

43. Filter banks Process image with each filter and keep responses (or squared/abs responses)

44. How can we represent texture? Measure responses of blobs and edges at various orientations and scales Idea 1: Record simple statistics (e.g., mean, std.) of absolute filter responses

45. Can you match the texture to the response? Mean abs responses Filters A B C 1 2 3

46. Representing texture by mean abs response Mean abs responses Filters

47. Representing texture Idea 2: take vectors of filter responses at each pixel and cluster them, then take histograms (more on in later weeks)

48. How is it that a 4MP image can be compressed to a few hundred KB without a noticeable change? Compression

49. Lossy Image Compression (JPEG) Block-based Discrete Cosine Transform (DCT) Slides: Efros

50. Using DCT in JPEG The first coefficient B(0,0) is the DC component, the average intensity The top-left coeffs represent low frequencies, the bottom right – high frequencies

51. Image compression using DCT Quantize – More coarsely for high frequencies (which also tend to have smaller values) – Many quantized high frequency values will be zero Encode – Can decode with inverse dct Quantization table Filter responses Quantized values

52. JPEG Compression Summary 1.Convert image to YCrCb 2.Subsample color by factor of 2 – People have bad resolution for color 3.Split into blocks (8x8, typically), subtract 128 4.For each block a.Compute DCT coefficients b.Coarsely quantize Many high frequency components will become zero c.Encode (e.g., with Huffman coding) http://en.wikipedia.org/wiki/YCbCr http://en.wikipedia.org/wiki/JPEG

53. Lossless compression (PNG) 1.Predict that a pixel’s value based on its upper-left neighborhood 2.Store difference of predicted and actual value 3.Pkzip it (DEFLATE algorithm)

54. Denoising Additive Gaussian Noise Gaussian Filter

55. Smoothing with larger standard deviations suppresses noise, but also blurs the image Reducing Gaussian noise Source: S. Lazebnik

56. Reducing salt-and-pepper noise by Gaussian smoothing 3x35x57x7

57. Alternative idea: Median filtering A median filter operates over a window by selecting the median intensity in the window Is median filtering linear? Source: K. Grauman

58. Median filter What advantage does median filtering have over Gaussian filtering? – Robustness to outliers Source: K. Grauman

59. Median filter Salt-and-pepper noise Median filtered Source: M. Hebert MATLAB: medfilt2(image, [h w])

60. Median vs. Gaussian filtering 3x35x57x7 Gaussian Median

61. Other non-linear filters Weighted median (pixels further from center count less) Clipped mean (average, ignoring few brightest and darkest pixels) Bilateral filtering (weight by spatial distance and intensity difference) http://vision.ai.uiuc.edu/?p=1455Image: Bilateral filtering

62. Review of last three days

63. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0 0000000000 0000000000 000 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Credit: S. Seitz Review: Image filtering 111 111 111

64. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 010 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz

65. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 01020 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz

66. Filtering in spatial domain 01 -202 01 * =

67. Filtering in frequency domain FFT Inverse FFT =

68. Review of Last 3 Days Filtering in spatial domain – Slide filter over image and take dot product at each position – Remember linearity (for linear filters) – Examples 1D: [-1 0 1], [0 0 0 0 0.5 1 1 1 0.5 0 0 0] 1D: [0.25 0.5 0.25], [0 0 0 0 0.5 1 1 1 0.5 0 0 0] 2D: [1 0 0 ; 0 2 0 ; 0 0 1]/4

69. Review of Last 3 Days Filtering in frequency domain – Can be faster than filtering in spatial domain (for large filters) – Can help understand effect of filter – Algorithm: 1.Convert image and filter to fft (fft2 in matlab) 2.Pointwise-multiply ffts 3.Convert result to spatial domain with ifft2

70. Review of Last 3 Days Linear filters for basic processing – Edge filter (high-pass) – Gaussian filter (low-pass) FFT of Gaussian [-1 1] FFT of Gradient Filter Gaussian

71. Review of Last 3 Days Derivative of Gaussian

72. Review of Last 3 Days Applications of filters – Template matching (SSD or Normxcorr2) SSD can be done with linear filters, is sensitive to overall intensity – Gaussian pyramid Coarse-to-fine search, multi-scale detection – Laplacian pyramid Teases apart different frequency bands while keeping spatial information Can be used for compositing in graphics – Downsampling Need to sufficiently low-pass before downsampling

73. Next Lectures Image representation (e.g. SIFT) and matching across multiple views (e.g. Stereo, Structure from Motion).