Linear filters. CS8690 Computer Vision University of Missouri at Columbia What is Image Filtering? Modify the pixels in an image based on some function.

Slides:



Advertisements
Similar presentations
Linear Filtering – Part II Selim Aksoy Department of Computer Engineering Bilkent University
Advertisements

3-D Computational Vision CSc Image Processing II - Fourier Transform.
Spatial Filtering (Chapter 3)
Pyramids and Texture. Scaled representations Big bars and little bars are both interesting Spots and hands vs. stripes and hairs Inefficient to detect.
Linear Filtering – Part II Selim Aksoy Department of Computer Engineering Bilkent University
Computational Photography CSE 590 Tamara Berg Filtering & Pyramids.
Edge detection Goal: Identify sudden changes (discontinuities) in an image Intuitively, most semantic and shape information from the image can be encoded.
Lecture 4 Edge Detection
Lecture 4 Linear Filters and Convolution
Computer Vision Group Edge Detection Giacomo Boracchi 5/12/2007
Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth Differentiation and convolution Recall Now this is linear and shift invariant,
Edge detection. Edge Detection in Images Finding the contour of objects in a scene.
Image processing. Image operations Operations on an image –Linear filtering –Non-linear filtering –Transformations –Noise removal –Segmentation.
Computer Vision - A Modern Approach
Immagini e filtri lineari. Image Filtering Modifying the pixels in an image based on some function of a local neighborhood of the pixels
Linear filtering. Overview: Linear filtering Linear filters Definition and properties Examples Gaussian smoothing Separability Applications Denoising.
Linear filters and edges. Linear Filters General process: Form new image whose pixels are a weighted sum of original pixel values, using the same set.
Introduction to Computer Vision CS / ECE 181B  Handout #4 : Available this afternoon  Midterm: May 6, 2004  HW #2 due tomorrow  Ack: Prof. Matthew.
CS443: Digital Imaging and Multimedia Filters Spring 2008 Ahmed Elgammal Dept. of Computer Science Rutgers University Spring 2008 Ahmed Elgammal Dept.
Lecture 2: Image filtering
Computer Vision - A Modern Approach Set: Pyramids and Texture Slides by D.A. Forsyth Scaled representations Big bars (resp. spots, hands, etc.) and little.
Linear filtering.
Linear Filtering About modifying pixels based on neighborhood. Local methods simplest. Linear means linear combination of neighbors. Linear methods simplest.
Computer Vision : CISC 4/689
Computer Vision Linear filters and edges Marc Pollefeys COMP 256.
Fourier Analysis : Rendering and Image Processing Alexei Efros.
Basic Image Processing January 26, 30 and February 1.
Most slides from Steve Seitz
Computer Vision Spring ,-685 Instructor: S. Narasimhan Wean 5409 T-R 10:30am – 11:50am.
2D Image Fourier Spectrum.
Filtering Course web page: vision.cis.udel.edu/cv March 5, 2003  Lecture 9.
Applications of Image Filters Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 02/04/10.
Linear Filters and Edges
Machine Vision ENT 273 Image Filters Hema C.R. Lecture 5.
CS559: Computer Graphics Lecture 3: Digital Image Representation Li Zhang Spring 2008.
EECS 274 Computer Vision Pyramid and Texture. Filter, pyramid and texture Frequency domain Fourier transform Gaussian pyramid Wavelets Texture Reading:
Lecture 03 Area Based Image Processing Lecture 03 Area Based Image Processing Mata kuliah: T Computer Vision Tahun: 2010.
Image Processing Edge detection Filtering: Noise suppresion.
Lecture 6: Edge Detection CAP 5415: Computer Vision Fall 2008.
EECS 274 Computer Vision Linear Filters and Edges.
Computer Vision Spring ,-685 Instructor: S. Narasimhan Wean 5403 T-R 3:00pm – 4:20pm.
Linear filtering. Motivation: Noise reduction Given a camera and a still scene, how can you reduce noise? Take lots of images and average them! What’s.
Edge detection Goal: Identify sudden changes (discontinuities) in an image Intuitively, most semantic and shape information from the image can be encoded.
Computer Vision : CISC 4/689 Note to self David Jacob’s notes: Convolution Correlation Derivatives Separability.
Edges.
Image hole-filling. Agenda Project 2: Will be up tomorrow Due in 2 weeks Fourier – finish up Hole-filling (texture synthesis) Image blending.
Linear filtering. Motivation: Image denoising How can we reduce noise in a photograph?
Instructor: Mircea Nicolescu Lecture 5 CS 485 / 685 Computer Vision.
Last Lecture photomatix.com. Today Image Processing: from basic concepts to latest techniques Filtering Edge detection Re-sampling and aliasing Image.
Sliding Window Filters Longin Jan Latecki October 9, 2002.
Non-linear filtering Example: Median filter Replaces pixel value by median value over neighborhood Generates no new gray levels.
Filters– Chapter 6. Filter Difference between a Filter and a Point Operation is that a Filter utilizes a neighborhood of pixels from the input image to.
Edges Edges = jumps in brightness/color Brightness jumps marked in white.
Spatial Filtering (Chapter 3) CS474/674 - Prof. Bebis.
Linear Filters and Edges Chapters 7 and 8
Linear Filtering – Part II
Linear Filters and Edges Chapters 7 and 8
Linear Filters and Edges Chapters 7 and 8
Department of Computer Engineering
Edge Detection CS 678 Spring 2018.
Edge detection Goal: Identify sudden changes (discontinuities) in an image Intuitively, most semantic and shape information from the image can be encoded.
Gaussian Pyramid and Texture
Phase and Amplitude in Fourier Transforms, Meaning of frequencies
Basic Image Processing
Most slides from Steve Seitz
Linear Filtering CS 678 Spring 2018.
Department of Computer Engineering
Most slides from Steve Seitz
Templates and Image Pyramids
Presentation transcript:

Linear filters

CS8690 Computer Vision University of Missouri at Columbia What is Image Filtering? Modify the pixels in an image based on some function of a local neighborhood of the pixels Some function

CS8690 Computer Vision University of Missouri at Columbia Linear Filtering Linear case is simplest and most useful –Replace each pixel with a linear combination of its neighbors. The prescription for the linear combination is called the convolution kernel. Linear case is simplest and most useful –Replace each pixel with a linear combination of its neighbors. The prescription for the linear combination is called the convolution kernel kernel

CS8690 Computer Vision University of Missouri at Columbia  c 22 + c 22 f(i,j) f(.) o (i,j) = f(.) c 12 + c 12 f(i-1,j) f(.) c 13 + c 13 f(i-1,j+1) + f(.) c 21 c 21 f(i,j-1) f(.) c 23 + c 23 f(i,j+1) + f(.) c 31 c 31 f(i+1,j-1) f(.) c 32 + c 32 f(i+1,j) f(.) c 33 + c 33 f(i+1,j+1) f(.) c 11 c 11 f(i-1,j-1) Convolution

CS8690 Computer Vision University of Missouri at Columbia Linear Filter = Convolution

CS8690 Computer Vision University of Missouri at Columbia Filtering Examples

CS8690 Computer Vision University of Missouri at Columbia Filtering Examples

CS8690 Computer Vision University of Missouri at Columbia Filtering Examples

CS8690 Computer Vision University of Missouri at Columbia Smoothing With Gaussian Gaussian Averaging

CS8690 Computer Vision University of Missouri at Columbia Linear Filters General process: –Form new image whose pixels are a weighted sum of original pixel values, using the same set of weights at each point. Properties –Output is a linear function of the input –Output is a shift-invariant function of the input (i.e. shift the input image two pixels to the left, the output is shifted two pixels to the left) General process: –Form new image whose pixels are a weighted sum of original pixel values, using the same set of weights at each point. Properties –Output is a linear function of the input –Output is a shift-invariant function of the input (i.e. shift the input image two pixels to the left, the output is shifted two pixels to the left) Example: smoothing by averaging –form the average of pixels in a neighbourhood Example: smoothing with a Gaussian –form a weighted average of pixels in a neighbourhood Example: finding a derivative –form a weighted average of pixels in a neighbourhood

CS8690 Computer Vision University of Missouri at Columbia Convolution Represent these weights as an image, H H is usually called the kernel Operation is called convolution –it’s associative Represent these weights as an image, H H is usually called the kernel Operation is called convolution –it’s associative Result is: Notice wierd order of indices –all examples can be put in this form –it’s a result of the derivation expressing any shift-invariant linear operator as a convolution.

CS8690 Computer Vision University of Missouri at Columbia Example: Smoothing by Averaging

CS8690 Computer Vision University of Missouri at Columbia Smoothing with a Gaussian Smoothing with an average actually doesn’t compare at all well with a defocussed lens –Most obvious difference is that a single point of light viewed in a defocussed lens looks like a fuzzy blob; but the averaging process would give a little square. Smoothing with an average actually doesn’t compare at all well with a defocussed lens –Most obvious difference is that a single point of light viewed in a defocussed lens looks like a fuzzy blob; but the averaging process would give a little square. A Gaussian gives a good model of a fuzzy blob

CS8690 Computer Vision University of Missouri at Columbia An Isotropic Gaussian The picture shows a smoothing kernel proportional to (which is a reasonable model of a circularly symmetric fuzzy blob)

CS8690 Computer Vision University of Missouri at Columbia Differentiation and convolution Recall Now this is linear and shift invariant, so must be the result of a convolution. Recall Now this is linear and shift invariant, so must be the result of a convolution. We could approximate this as (which is obviously a convolution; it’s not a very good way to do things, as we shall see)

CS8690 Computer Vision University of Missouri at Columbia Finite differences

CS8690 Computer Vision University of Missouri at Columbia Noise Simplest noise model –independent stationary additive Gaussian noise –the noise value at each pixel is given by an independent draw from the same normal probability distribution Simplest noise model –independent stationary additive Gaussian noise –the noise value at each pixel is given by an independent draw from the same normal probability distribution Issues –this model allows noise values that could be greater than maximum camera output or less than zero –for small standard deviations, this isn’t too much of a problem - it’s a fairly good model –independence may not be justified (e.g. damage to lens) –may not be stationary (e.g. thermal gradients in the ccd)

CS8690 Computer Vision University of Missouri at Columbia sigma=1

CS8690 Computer Vision University of Missouri at Columbia sigma=16

CS8690 Computer Vision University of Missouri at Columbia Finite differences and noise Finite difference filters respond strongly to noise –obvious reason: image noise results in pixels that look very different from their neighbours Generally, the larger the noise the stronger the response Finite difference filters respond strongly to noise –obvious reason: image noise results in pixels that look very different from their neighbours Generally, the larger the noise the stronger the response What is to be done? –intuitively, most pixels in images look quite a lot like their neighbours –this is true even at an edge; along the edge they’re similar, across the edge they’re not –suggests that smoothing the image should help, by forcing pixels different to their neighbours (=noise pixels?) to look more like neighbours

CS8690 Computer Vision University of Missouri at Columbia Finite differences responding to noise Increasing noise -> (this is zero mean additive gaussian noise)

CS8690 Computer Vision University of Missouri at Columbia The response of a linear filter to noise Do only stationary independent additive Gaussian noise with zero mean (non-zero mean is easily dealt with) Mean: –output is a weighted sum of inputs –so we want mean of a weighted sum of zero mean normal random variables –must be zero Do only stationary independent additive Gaussian noise with zero mean (non-zero mean is easily dealt with) Mean: –output is a weighted sum of inputs –so we want mean of a weighted sum of zero mean normal random variables –must be zero Variance: –recall variance of a sum of random variables is sum of their variances variance of constant times random variable is constant^2 times variance –then if  is noise variance and kernel is K, variance of response is

CS8690 Computer Vision University of Missouri at Columbia Filter responses are correlated over scales similar to the scale of the filter Filtered noise is sometimes useful –looks like some natural textures, can be used to simulate fire, etc. over scales similar to the scale of the filter Filtered noise is sometimes useful –looks like some natural textures, can be used to simulate fire, etc.

CS8690 Computer Vision University of Missouri at Columbia

CS8690 Computer Vision University of Missouri at Columbia

CS8690 Computer Vision University of Missouri at Columbia

CS8690 Computer Vision University of Missouri at Columbia Smoothing reduces noise Generally expect pixels to “be like” their neighbours –surfaces turn slowly –relatively few reflectance changes Generally expect noise processes to be independent from pixel to pixel Generally expect pixels to “be like” their neighbours –surfaces turn slowly –relatively few reflectance changes Generally expect noise processes to be independent from pixel to pixel Implies that smoothing suppresses noise, for appropriate noise models Scale –the parameter in the symmetric Gaussian –as this parameter goes up, more pixels are involved in the average –and the image gets more blurred –and noise is more effectively suppressed

CS8690 Computer Vision University of Missouri at Columbia The effects of smoothing Each row shows smoothing with gaussians of different width; each column shows different realisations of an image of gaussian noise.

CS8690 Computer Vision University of Missouri at Columbia Some other useful filtering techniques Median filter Anisotropic diffusion Median filter Anisotropic diffusion

CS8690 Computer Vision University of Missouri at Columbia Median filters : principle non-linear filter method : n 1. rank-order neighbourhood intensities n 2. take middle value no new grey levels emerge...

CS8690 Computer Vision University of Missouri at Columbia Median filters : odd-man-out advantage of this type of filter is its “odd-man-out” effect e.g. 1,1,1,7,1,1,1,1  ?,1,1,1.1,1,1,?

CS8690 Computer Vision University of Missouri at Columbia Median filters : example filters have width 5 : 

CS8690 Computer Vision University of Missouri at Columbia Median filters : analysis median completely discards the spike, linear filter always responds to all aspects median filter preserves discontinuities, linear filter produces rounding-off effects DON’T become all too optimistic 

CS8690 Computer Vision University of Missouri at Columbia Median filter : images  3 x 3 median filter : sharpens edges, destroys edge cusps and protrusions

CS8690 Computer Vision University of Missouri at Columbia Median filters : Gauss revisited Comparison with Gaussian : e.g. upper lip smoother, eye better preserved  3 x 3 median filter : sharpens edges, destroys edge cusps and protrusions

CS8690 Computer Vision University of Missouri at Columbia Example of median 10 times 3 X 3 median patchy effect important details lost (e.g. ear-ring) 

Pyramids

CS8690 Computer Vision University of Missouri at Columbia Linear filters Gaussian blurring Finite differences Composition of linear filters = linear filter Edge detection Corners, etc.

CS8690 Computer Vision University of Missouri at Columbia Scaled representations Big bars (resp. spots, hands, etc.) and little bars are both interesting –Stripes and hairs, say Inefficient to detect big bars with big filters –And there is superfluous detail in the filter kernel Big bars (resp. spots, hands, etc.) and little bars are both interesting –Stripes and hairs, say Inefficient to detect big bars with big filters –And there is superfluous detail in the filter kernel Alternative: –Apply filters of fixed size to images of different sizes –Typically, a collection of images whose edge length changes by a factor of 2 (or root 2) –This is a pyramid (or Gaussian pyramid) by visual analogy

CS8690 Computer Vision University of Missouri at Columbia A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose

CS8690 Computer Vision University of Missouri at Columbia Aliasing Can’t shrink an image by taking every second pixel If we do, characteristic errors appear –In the next few slides –Typically, small phenomena look bigger; fast phenomena can look slower –Common phenomenon Wagon wheels rolling the wrong way in movies Checkerboards misrepresented in ray tracing Striped shirts look funny on colour television Can’t shrink an image by taking every second pixel If we do, characteristic errors appear –In the next few slides –Typically, small phenomena look bigger; fast phenomena can look slower –Common phenomenon Wagon wheels rolling the wrong way in movies Checkerboards misrepresented in ray tracing Striped shirts look funny on colour television

CS8690 Computer Vision University of Missouri at Columbia Resample the checkerboard by taking one sample at each circle. In the case of the top left board, new representation is reasonable. Top right also yields a reasonable representation. Bottom left is all black (dubious) and bottom right has checks that are too big.

CS8690 Computer Vision University of Missouri at Columbia Constructing a pyramid by taking every second pixel leads to layers that badly misrepresent the top layer

CS8690 Computer Vision University of Missouri at Columbia Open questions What causes the tendency of differentiation to emphasize noise? In what precise respects are discrete images different from continuous images? How do we avoid aliasing? General thread: a language for fast changes The Fourier Transform What causes the tendency of differentiation to emphasize noise? In what precise respects are discrete images different from continuous images? How do we avoid aliasing? General thread: a language for fast changes The Fourier Transform

CS8690 Computer Vision University of Missouri at Columbia The Fourier Transform Represent function on a new basis –Think of functions as vectors, with many components –We now apply a linear transformation to transform the basis dot product with each basis element In the expression, u and v select the basis element, so a function of x and y becomes a function of u and v basis elements have the form Represent function on a new basis –Think of functions as vectors, with many components –We now apply a linear transformation to transform the basis dot product with each basis element In the expression, u and v select the basis element, so a function of x and y becomes a function of u and v basis elements have the form vectorized image Fourier transform base,. transformed image

CS8690 Computer Vision University of Missouri at Columbia To get some sense of what basis elements look like, we plot a basis element --- or rather, its real part --- as a function of x,y for some fixed u, v. We get a function that is constant when (ux+vy) is constant. The magnitude of the vector (u, v) gives a frequency, and its direction gives an orientation. The function is a sinusoid with this frequency along the direction, and constant perpendicular to the direction.

CS8690 Computer Vision University of Missouri at Columbia Fourier basis element example, real part F u,v (x,y) F u,v (x,y)=const. for (ux+vy)=const. Vector (u,v) Magnitude gives frequency Direction gives orientation.

CS8690 Computer Vision University of Missouri at Columbia Here u and v are larger than in the previous slide.

CS8690 Computer Vision University of Missouri at Columbia And larger still...

CS8690 Computer Vision University of Missouri at Columbia Phase and Magnitude Fourier transform of a real function is complex –difficult to plot, visualize –instead, we can think of the phase and magnitude of the transform Phase is the phase of the complex transform Magnitude is the magnitude of the complex transform Fourier transform of a real function is complex –difficult to plot, visualize –instead, we can think of the phase and magnitude of the transform Phase is the phase of the complex transform Magnitude is the magnitude of the complex transform Curious fact –all natural images have about the same magnitude transform –hence, phase seems to matter, but magnitude largely doesn’t Demonstration –Take two pictures, swap the phase transforms, compute the inverse - what does the result look like?

CS8690 Computer Vision University of Missouri at Columbia

CS8690 Computer Vision University of Missouri at Columbia This is the magnitude transform of the cheetah pic

CS8690 Computer Vision University of Missouri at Columbia This is the phase transform of the cheetah pic

CS8690 Computer Vision University of Missouri at Columbia

CS8690 Computer Vision University of Missouri at Columbia This is the magnitude transform of the zebra pic

CS8690 Computer Vision University of Missouri at Columbia This is the phase transform of the zebra pic

CS8690 Computer Vision University of Missouri at Columbia Reconstruction with zebra phase, cheetah magnitude

CS8690 Computer Vision University of Missouri at Columbia Reconstruction with cheetah phase, zebra magnitude

CS8690 Computer Vision University of Missouri at Columbia Various Fourier Transform Pairs Important facts –The Fourier transform is linear –There is an inverse FT –scale function down  scale transform up i.e. high frequency = small details –The FT of a Gaussian is a Gaussian. compare to box function transform Important facts –The Fourier transform is linear –There is an inverse FT –scale function down  scale transform up i.e. high frequency = small details –The FT of a Gaussian is a Gaussian. compare to box function transform

CS8690 Computer Vision University of Missouri at Columbia Convolution theorem The convolution theorem –The Fourier transform of the convolution of two functions is the product of their Fourier transforms –The Fourier transform of the product of two functions is the convolution of the Fourier transforms The convolution theorem –The Fourier transform of the convolution of two functions is the product of their Fourier transforms –The Fourier transform of the product of two functions is the convolution of the Fourier transforms

CS8690 Computer Vision University of Missouri at Columbia More on Fourier transform Basis functions of Fourier transformare eigenfunctions of linear systems linear system

CS8690 Computer Vision University of Missouri at Columbia Sampling Go from continuous world to discrete world, from function to vector Samples are typically measured on regular grid Go from continuous world to discrete world, from function to vector Samples are typically measured on regular grid

CS8690 Computer Vision University of Missouri at Columbia Sampling in 2D does the same thing, only in 2D. We’ll assume that these sample points are on a regular grid, and can place one at each integer point for convenience.

CS8690 Computer Vision University of Missouri at Columbia Delta function limit to infinity of constant area function: limit to infinity of constant area function:

CS8690 Computer Vision University of Missouri at Columbia A continuous model for a sampled function We want to be able to approximate integrals sensibly Leads to –the delta function –model on right We want to be able to approximate integrals sensibly Leads to –the delta function –model on right

CS8690 Computer Vision University of Missouri at Columbia The Fourier transform of a sampled signal

CS8690 Computer Vision University of Missouri at Columbia

CS8690 Computer Vision University of Missouri at Columbia

CS8690 Computer Vision University of Missouri at Columbia Smoothing as low-pass filtering The message of the FT is that high frequencies lead to trouble with sampling. Solution: suppress high frequencies before sampling –multiply the FT of the signal with something that suppresses high frequencies –or convolve with a low- pass filter The message of the FT is that high frequencies lead to trouble with sampling. Solution: suppress high frequencies before sampling –multiply the FT of the signal with something that suppresses high frequencies –or convolve with a low- pass filter A filter whose FT is a box is bad, because the filter kernel has infinite support Common solution: use a Gaussian –multiplying FT by Gaussian is equivalent to convolving image with Gaussian.

CS8690 Computer Vision University of Missouri at Columbia Sampling without smoothing. Top row shows the images, sampled at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

CS8690 Computer Vision University of Missouri at Columbia Sampling with smoothing. Top row shows the images. We get the next image by smoothing the image with a Gaussian with sigma 1 pixel, then sampling at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

CS8690 Computer Vision University of Missouri at Columbia Sampling with smoothing. Top row shows the images. We get the next image by smoothing the image with a Gaussian with sigma 1.4 pixels, then sampling at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

CS8690 Computer Vision University of Missouri at Columbia Applications of scaled representations Search for correspondence –look at coarse scales, then refine with finer scales Edge tracking –a “good” edge at a fine scale has parents at a coarser scale Control of detail and computational cost in matching –e.g. finding stripes –terribly important in texture representation Search for correspondence –look at coarse scales, then refine with finer scales Edge tracking –a “good” edge at a fine scale has parents at a coarser scale Control of detail and computational cost in matching –e.g. finding stripes –terribly important in texture representation

Gaussian Pyramid Low-Pass Filtered Image Image Gaussian Filter Downsample Low-Res Image

CS8690 Computer Vision University of Missouri at Columbia The Gaussian pyramid Smooth with gaussians, because –a gaussian*gaussian=another gaussian Synthesis –smooth and sample Analysis –take the top image Gaussians are low pass filters, so representation is redundant Smooth with gaussians, because –a gaussian*gaussian=another gaussian Synthesis –smooth and sample Analysis –take the top image Gaussians are low pass filters, so representation is redundant

CS8690 Computer Vision University of Missouri at Columbia Example: CMU face detection

CS8690 Computer Vision University of Missouri at Columbia The Gaussian pyramid

CS8690 Computer Vision University of Missouri at Columbia