CS 4487/9587 Algorithms for Image Analysis Image Processing Basics Lecture 3 Lena Gorelick, substituting for Yuri Boykov Acknowledgements: slides from Steven Seitz, Aleosha Efros, David Forsyth, and Gonzalez & Woods

CS 4487/9587 Algorithms for Image Analysis Image Processing Basics Domain Transformation Resize, Rotate etc. Range Transformation Point Processing gamma correction window-center correction histogram equalization Pixel location Pixel Intensity Window = 800 Center = 1160

CS 4487/9587 Algorithms for Image Analysis Image Processing Basics Domain Transformation Resize, Rotate etc. Range Transformation Point Processing gamma correction window-center correction histogram equalization Neighborhood Processing (Filtering)

Neighborhood Processing (or filtering) What is wrong with this image? How can we remove noise? Look spatially around each pixel - neighborhood Can we use Point Processing? Courtesy of Carlo Tomasi Courtesy of Neel Joshi Readings: Forsyth & Ponce, chapters 8.1-8.2

Neighborhood Processing (or filtering) Lets reshuffle all pixels within the image Point Processing will have the same effect Gets intensity of a single pixel as an input Unaware of spatial information Neighborhood Processing Takes “spatial information” into account Images contain “spatial information” Has Spatial Information Looks noisy Readings: Forsyth & Ponce, chapters 8.1-8.2

Neighborhood Processing (filtering) Linear image transforms Assume 1D function

Neighborhood Processing (filtering) Linear image transforms Assume 1D function

Neighborhood Processing (filtering) Linear shift-invariant filters matrix M A commonly used pattern Represented by a kernel (or a mask) For 2k+1 size kernel Shift - Invariant The same in each row (i - 1) i (i + 1)

Neighborhood Processing (filtering) 2D linear transforms Assume 2D function Concatenate the columns using a “raster-scan” order

Neighborhood Processing (filtering) 2D filtering A 2D function Can be filtered by a 2D filter (kernel) To get a new image This is called a cross-correlation operation Or a sliding dot-product

Neighborhood Processing (filtering) 2D filtering Cross-correlation in which the filter is flipped horizontally and vertically is called convolution

Neighborhood Processing (filtering) Convolution vs. Cross-Correlation If the kernel is symmetric convolution = cross-correlation

Neighborhood Processing 2D filtering Noise Types of noise: Salt and pepper noise Impulse noise Gaussian noise Due to transmission errors dead CCD pixels specks on lens can be specific to a sensor Salt and pepper and impulse noise can be due to transmission errors (e.g., from deep space probe), dead CCD pixels, specks on lens We’re going to focus on Gaussian noise first. If you had a sensor that was a little noisy and measuring the same thing over and over, how would you reduce the noise?

Neighborhood Processing Practical Noise Reduction How can we remove noise? Replace each pixel with the average of a kxk window around it 100 130 110 120 90 80 104

Neighborhood Processing (filtering) Mean filtering 90 Replace each pixel with the average of a kxk window around it

Neighborhood Processing (filtering) Mean filtering 90 10 20 30 40 60 90 50 80 Replace each pixel with the average of a kxk window around it What happens if we use a larger filter window?

Neighborhood Processing (filtering) Effect of mean filters 3x3 5x5 7x7 Salt & Pepper noise Gaussian noise Demo with photoshop Side effect - blur

Neighborhood Processing (filtering) Mean kernel What’s the kernel for a 3x3 mean filter? 90

Neighborhood Processing (filtering) Mean kernel What’s the kernel for a 3x3 mean filter? 90 1/9

Neighborhood Processing (filtering) Mean kernel What’s the kernel for a 3x3 mean filter? Equal weight to all pixels within the neighborhood 90 1 1/9

Neighborhood Processing (filtering) Gaussian Filtering A Gaussian kernel gives less weight to pixels further from the center of the window 90 1 2 4 is a discrete approximation of a Gaussian function:

Neighborhood Processing (filtering) Mean vs. Gaussian filtering Input image Gaussian filtering Mean Filtering Gaussian filter, 20x20, σ = 5 Mean filter, 20x20

Neighborhood Processing (filtering) Mean vs. Gaussian filtering

Neighborhood Processing (filtering) Gaussian Filtering Low-pass filter Smooth color variation (low frequency) is preserved Sharp edges (high frequency) are removed

Neighborhood Processing (filtering) Median filters A Median Filter operates over a window by selecting the median intensity in the window. Image credit: Wikipedia – page on Median Filter

Neighborhood Processing (filtering) Median filters Is a median filter a kind of convolution? No, median filter is an example of non-linear filtering

Salt&Pepper Noise

Gaussian noise

Neighborhood Processing (filtering) Median filters What advantage does a median filter have over a mean filter? Better at removing Salt & Pepper noise Disadvantage: Slow Better at salt’n’pepper noise Not convolution: try a region with 1’s and a 2, and then 1’s and a 3

Neighborhood Processing (filtering) Derivatives and Convolution Image Derivatives and Gradients Used for Edge/Corner Detection Computed with Finite Differences Filters Laplacian of Gaussians (LoG) Filter Used for Edge/Blob Detection and Image Enhancement Approximated using Difference of Gaussians

Reading: Forsyth & Ponce, 8.1-8.2 First Derivative Recall Sharp changes in gray level of the input image correspond to “peaks or valleys” of the first-derivative of the input signal. http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Reading: Forsyth & Ponce, 8.1-8.2 First Derivative and convolution How can we approximate it for a discrete function? Is this operation shift-invariant? Is it linear? ?

Reading: Forsyth & Ponce, 8.1-8.2 First Derivative and convolution Finite Difference x x+h x x-h x x+h/2 x-h/2 Slide credit: Wikipedia

Reading: Forsyth & Ponce, 8.1-8.2 First Derivative and convolution Finite Difference – The order of an error can be derived using Taylor Theorem Slide credit: Wikipedia

Reading: Forsyth & Ponce, 8.1-8.2 First Derivative and convolution Pixel Size Using Finite Central Difference We need a kernel such that 1 -1

Neighborhood Processing (filtering) Finite differences – responds to edges Now the figure on the right is signed; darker is negative, lighter is positive, and mid grey is zero. I always ask 1) which derivative (y or x) is this? 2) Have I got the sign of the kernel right? (i.e. is it d/dx or -d/dx). Dark = negative White = positive Gray = 0

Increasing noise -> (this is zero mean additive gaussian noise) Neighborhood Processing (filtering) Finite differences - responding to noise Increasing noise -> (this is zero mean additive gaussian noise)

Neighborhood Processing (filtering) Finite differences and noise Finite difference filters respond strongly to noise Noisy pixels look very different from their neighbours The larger the noise  the stronger is the response How can we eliminate the response to noise? Most pixels in images look similar to their neighbours (even at an edge) Smooth the image (mean/gaussian filtering)

Neighborhood Processing (filtering) Smoothing and Differentiation Smoothing before differentiation = two convolutions Convolution is associative Important to point out that the derivative of gaussian kernels I’ve illustrated “look like” the effects that they’re trying to identify.

Neighborhood Processing (filtering) Smoothing and Differentiation 1 pixel 3 pixels 7 pixels The scale of the smoothing filter affects derivative estimates, and also the semantics of the edges recovered.

Sobel kernels Yet another approximation frequently used Neighborhood Processing (filtering) Sobel kernels Yet another approximation frequently used 1 -1 2 -2 1 2 -1 -2

Neighborhood Processing (filtering) Image Gradients Recall for a function of two variables The gradient at a point (x,y) Gradient Magnitude Gradient Orientation direction of the “steepest ascend” orthogonal to object boundaries in the image x y small image gradients in low textured areas

Neighborhood Processing (filtering) Image Gradient for Edge Detection Typical application where image gradients are used is image edge detection find points with large image gradients Canny edge detector suppresses non-extrema Gradient points

Reading: Forsyth & Ponce, 8.1-8.2 Second derivatives and convolution Peaks or valleys of the first-derivative of the input signal, correspond to “zero-crossings” of the second-derivative of the input signal. http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) Second derivatives and convolution http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) Second derivatives and convolution Better localized edges But more sensitive to noise http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) Second derivatives and convolution http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) Second Image Derivatives Laplace operator “divergence of gradient” rotationally invariant second derivative for 2D functions is the rate at which the average value of ƒ over spheres centered atp, deviates from ƒ(p) as the radius of the sphere grows. -1 2 -1 2 -1 4 Finite Difference Second Order Derivative in x Finite Difference Second Order Derivative in y

Neighborhood Processing (filtering) Second Image Derivatives Laplacian Zero Crossing Used for edge detection (alternative to computing Gradient extrema) image intensity Non-decreasing function  slow increase, faster increase, slow increase Positive gradient Positive Magnitude of gradient magnitude of image gradient Laplacian (2nd derivative) of the image http://homepages.inf.ed.ac.uk/rbf/HIPR2/zeros.htm

Neighborhood Processing (filtering) Laplacian Filtering -Zero on uniform regions -Positive on one side of an edge -Negative on the other side  -Zero at some point in between on the edge itself  band-pass filter (Suppresses both high and low frequencies) http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) Laplacian of a Gaussian (LoG) Smooth before differentiation (remember associative property of convolution) http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) Laplacian of a Gaussian (LoG) Suppresses both high and low frequencies http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) Laplacian of a Gaussian (LoG) http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) Laplacian of a Gaussian (LoG) http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) Laplacian of a Gaussian (LoG) Can be approximated by a difference of two Gaussians (DoG) http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) DoG vs. LoG Separable (product decomposition)  more efficient Can explain band-pass filter since Gaussian is a low-pass filter. http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) LoG for Blob Detection Cross correlation with a filter can be viewed as comparing a little “picture” of what you want to find against all local regions in the image. Scale of blob (size ; radius in pixels) is determined by the sigma parameter of the LoG filter. http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) LoG for Blob Detection http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering) Unsharp masking What does blurring take away? - = unsharp mask +a = ? They are the same for filters that have both horizontal and vertical symmetry. unsharp mask

Neighborhood Processing (filtering) Unsharp masking = ? They are the same for filters that have both horizontal and vertical symmetry. unsharp mask

Reading: Forsyth & Ponce ch.7.5 Filters and Templates Recall: filtering = cross-correlation = dot-product Measures the angle between two vectors: the filter and the image window  Measures visual similarity (x,y) Filter Mask Input Image f(x,y) g(x,y) Output Image Correlation

Neighborhood Processing (filtering) Template Matching Since we divide by the norm Normalized cross-correlation Bring the template and the image to a uniform scale: subtract the mean and divide by the variance http://scien.stanford.edu/pages/labsite/2008/psych221/projects/08/MariaJabon/index.htm

Neighborhood Processing (filtering) Template Matching Normalized cross-correlation Extremely time consuming due to sliding window Slide credit: Matlab manual

Neighborhood Processing (filtering) Template Matching Slide credit: OpenCV manual