Image Processing IB Paper 8 – Part A Ognjen Arandjelović http://mi.eng.cam.ac.uk/~oa214/

– Image Essentials, Sampling –

Image Processing Paradigm We can think of image processing as a black box that takes an input image and produces an output image. Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Feature extraction, for example, does not fall under the umbrella of image processing.

Image Processing Image processing deals with computer-based manipulation of digital images. These include: geometric operations (various morphs) brightness and contrast correction colour enhancement, segmentation, denoising. Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Image Processing Applications Fingerprint enhancement for analysis and recognition: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Original fingerprint image After denoising and contrast enhancement After binarization and morphological processing

Image Processing Applications Enhancement of CCTV footage: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Still from original CCTV footage After brightness and contrast enhancement After denoising

Image Processing Applications Achieving illumination and pose invariance for automatic face recognition: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. After face detection and cropping After adaptive high-pass filtering Still from original authentication video

Discretization An inherent cause of information loss when representing physical signals on a computer: Face geometry Two types: Level / numerical It is impossible to represent infinite numbers (e.g. Pi) using a finite amount of storage space Temporal or spatial It is impossible to sample a signal at infinite frequency or at infinite number of locations Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Discretization Under a Magnifying Glass An example on an image of a face: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Pixels = spatial discretization Values = numerical discretization

Numerical Quantization The luminance of each pixel can be represented only with finite precision. This loss of information is called quantization noise. Face geometry We can compute the average quantization noise energy per pixel: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Minimal luminance difference that can be represented

Numerical Quantization – An Example Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Spatial Discretization – An Example Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

1D Sampling In 1D, we model sampling of a continuous function as multiplication by a train of delta functions: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

1D Sampling – Frequency Domain Multiplication in the spatial domain goes into convolution in the frequency domain: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. A train of delta functions Fourier spectrum of the original 1D signal Fourier spectrum of the sampled 1D signal

2D Sampling Much like in 1D, in 2D we model sampling of a continuous function as multiplication by a comb of delta functions: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. A comb of delta functions Image as a 2D function / surface

2D Sampling – Frequency Domain Much like in 1D, in 2D we model sampling of a continuous function as multiplication by a comb of delta functions: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. A comb of delta functions Fourier spectrum of the sampled 2D signal Fourier spectrum of a 2D signal

Geometric Transformations Geometric transformations warp images without (in principle) changing the value of the corresponding 2D signal. Examples include: Scaling Cropping Rotation Arbitrary morphs Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Geometric Transformations – Apps Image registration for face recognition (correction of mild pose variations): Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Geometric Transformations – Apps Image warping for mosaicing of photographs: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. How to perform seamless concatenation of images?

Geometric Transformations – Main Idea All geometric transformations of images at their core concern the problem of resampling. As the original (not sampled) 2D signal cannot be accessed, we need to first reconstruct it from the set of samples we have i.e. pixels. Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Signal Reconstruction Remember that in the case of a band-limited signal, we can obtain a perfect reconstruction using the ideal low-pass filter: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Frequency spectrum of the sampled signal Ideal low-pass filter Perfectly reconstructed signal What is the problem with this?

The Ideal Low-Pass Filter Recall the Fourier transform of the ideal low-pass filter (i.e. the pulse function): Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Fourier transform Ideal low-pass filter The sinc function N.B.

The Ideal Low-Pass Filter in 2D Recall the Fourier transform of the ideal low-pass filter (i.e. the pulse function): Fourier transform Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Ideal 2D low-pass filter The 2D sinc function

Reconstruction in Spatial Domain Using duality, we can see that the original signal can be reconstructed by convolving its sampled version with a sinc function: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

The Ideal LPF in the Spatial Domain The reason why we cannot use the ideal LPF in image processing is of computational nature: This value, for example depends on all samples! Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. The idea is to use something more manageable, by making the new sample dependent only on its neighbourhood.

Linear Interpolation In linear interpolation, the signal between two samples is approximated by a straight line: Clearly not perfect Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. We can thus say that the constraint is that the line should pass through the two sample points – there are 2 DOFs (a point on the line and the slope).

Linear Interpolation We can then write an expression for the value of the function at an arbitrary location between two original samples: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Value at the new sample Value of “pixel a” Value of “pixel b”

Bi-Linear Interpolation Bi-linear interpolation is simply a 2D extension of linear interpolation: Original sampling locations Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. What is the value at this location?

Bi-Linear Interpolation One way of thinking about bi-linear interpolation is as fitting of a plane between neighbouring non-collinear sampling points: Original sampling locations What is the value at this point? Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Bi-Linear Interpolation Alternatively, you can think of bi-linear interpolation as weighted average of the new sample’s neighbours: Original sampling locations What is the value at this point? Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Cubic Interpolation We have seen that linear interpolation is far from perfect. Cubic interpolation does better: Much better! Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Cubic Interpolation Constraints In cubic interpolation we have the following constraints for segment between x2 and x3, given 4 consecutive samples x1, x2, x3, x4: C0 Continuity: Value at x2 and x3 should be exact (as with linear interpolation) C1 Continuity: Gradient at x2 should be (x3-x1) / 2 and at x3 should be (x4-x2) / 2 Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

The Gradient Constraint The see why the gradient at say x2 should be (x3-x1) / 2, consider 1st order Taylor series expansions around x2: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Now subtract (1) from (2):

Cubic Interpolation in 1D The expression for cubic interpolation is thus more complicated: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Bi-Cubic Interpolation The expression for bi-cubic interpolation gets very messy, so we do not give it here. The principle, however, is the same – you can think of it as cubic interpolation in the x direction, followed by cubic interpolation in the y direction. Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Bi-Linear vs. Bi-Cubic Interpolation A summary of key differences: Cubic produces less smoothing of the signal than linear Cubic is about 4 times more computationally demanding and is hence seldom used for resampling of large images Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Impulse Response of Linear Interp. We can analyse the performance of the two interpolators by looking at their impulse responses: An impulse Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Impulse response of the linear interpolation process

Impulse Response of Linear Interp. By taking the Fourier transform of impulse responses of the two interpolators, we can see how they affect different frequencies: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Frequency response of cubic interpolation is much flatter at higher frequencies – closer to the ideal LPF.

Image Rescaling Now that we know how to reconstruct the original signal from a discrete set of samples, we can easily manipulate images in various ways. New sampling locations (use interpolation) Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Original sampling locations Image resizing as resampling – the original 5 х 5 image is resampled to 6 х 6.

Image Rescaling Example To examine the effects of resampling, let us look at a small rectangular patch extracted from CCTV footage: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Magnified patch – 50 х 90 pixels

Image Rescaling Example Let us now compare the results of bi-linear and bi-cubic interpolation-based magnification for a factor of 2: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Bi-cubic Bi-linear Note that bi-linear magnification produces a much smoother result.

Rescaling Comparison – An Example Quantitative insight can be gained by looking at the image difference of two results: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. As expected from theory, the difference is mainly in the high-frequency content.

Downsampling Caveats Remember that the original signal can be reconstructed from a set of samples using a LPF only if it is band-limited: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. A train of delta functions Fourier spectrum of the original 1D signal Fourier spectrum of the sampled 1D signal No overlap of spectrum ‘replicas’ (this overlap is called aliasing) It is crucial to ensure that the signal is band-limited before downsampling.

Downsampling Caveats – Aliasing Let’s take a look at what happens with downsampling without considering the aliasing problem: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Downsampling Original image (184 х 184 pixels) Resulting image (40 х 40 pixels) Note strange downsampling artefacts.

Downsampling Caveats – Aliasing To ensure that high frequencies are suppressed, the original image should first be LP filtered: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Gaussian Smoothing Original image (184 х 184 pixels) Low-pass smoothed image

Downsampling Caveats – Aliasing Now we can downsample as before, without worrying about aliasing: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Downsampling Low-pass smoothed image Resulting image (40 х 40 pixels)

Downsampling Caveats – Aliasing Compare the results (or try switching anti-aliasing off in your Acrobat Reader!): Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Without LP filtering With LP filtering

Image Rotation Now that we know how to reconstruct the original signal from a discrete set of samples, we can easily manipulate images in various ways. New sampling locations (use interpolation) Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Original sampling locations Image rotation as resampling – the original 5 х 5 image is rotated ~30 degrees clockwise.

Image Rotation – New Samples The main difference to rescaling is that the locations of new samples are now slightly more difficult to compute. Recall that these are related to the original locations by the rotation matrix: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Where θ is the angle of rotation.

Image Rotation – Image Size Note that the size of the resulting, rotated image is in general different than the size of the original image. If the input image is of size H х W pixels, then the output image is of size H' х W' pixels, where: Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. The ceiling operator

Image Rotation – Image Size Note that the size of the resulting, rotated image is in general different than the size of the original image. Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important.

Image Rotation – An Example Results shown on entire feature length movies including, Run Lola Run, Groundhog Day For example: We might want to search for Lola’s apartment. Or in groundhog day for the clock. If you have seen the movie You know the clock is very important. Original fingerprint (480 х 400 pixels) Fingerprint rotated by 60 degrees (586 х 615 pixels)

– That is All for Today –