With Applications in Image Processing The Story of Wavelets Polikar, (C) 2007 With Applications in Image Processing Guest Lecture Dr. Robi Polikar

Outline Polikar, (C) 2007 Review of Fourier based methods Analyzing non-stationary signals Short-time Fourier transform Continuous wavelet transform Discrete wavelet transform Applications Discontinuity detection Signal denoising Compression 2-D wavelet transform Image analysis and compression Polikar, (C) 2007

What is a Transform and Why Do we Need One ? Transform: A mathematical operation that takes a function or sequence and maps it into another one Transforms are good things because… The transform of a function may give additional /hidden information about the original function, which may not be available /obvious otherwise The transform of an equation may be easier to solve than the original equation (recall Laplace transforms for “Defeques”) The transform of a function/sequence may require less storage, hence provide data compression / reduction An operation may be easier to apply on the transformed function, rather than the original function (recall convolution) Polikar, (C) 2007

Properties of Transforms Most useful transforms are: Linear: we can pull out constants, and apply superposition One-to-one: different functions have different transforms Invertible:for each transform T, there is an inverse transform T-1 using which the original function f can be recovered (kind of – sort of the undo button…) Continuous transform: map functions to functions Discrete transform: map sequences to sequences Semi-discrete transform: relate functions with sequences Polikar, (C) 2007 T f F T-1

What Does a Transform Look Like…? Complex function representation through simple building blocks Compressed representation through using only a few blocks (called basis functions / kernels) Sinusoids as building blocks: Fourier transform Frequency domain representation of the function Polikar, (C) 2007

What Transforms Are Available? Fourier series Continuous Fourier transform Discrete Fourier transform Laplace transform Z-transform Polikar, (C) 2007

How Does FT Work Anyway? Polikar, (C) 2007 Recall that FT uses complex exponentials (sinusoids) as building blocks. For each frequency of complex exponential, the sinusoid at that frequency is compared to the signal. If the signal consists of that frequency, the correlation is high  large FT coefficients. If the signal does not have any spectral component at a frequency, the correlation at that frequency is low / zero,  small / zero FT coefficient. Polikar, (C) 2007

FT At Work Polikar, (C) 2007

FT At Work F Polikar, (C) 2007 F F

FT At Work Polikar, (C) 2007 F

Stationary and Non-stationary Signals FT identifies all spectral components present in the signal, however it does not provide any information regarding the temporal (time) localization of these components. Why? Stationary signals consist of spectral components that do not change in time all spectral components exist at all times no need to know any time information FT works well for stationary signals However, non-stationary signals consists of time varying spectral components How do we find out which spectral component appears when? FT only provides what spectral components exist , not where in time they are located. Need some other ways to determine time localization of spectral components Polikar, (C) 2007

Stationary and Non-stationary Signals Stationary signals’ spectral characteristics do not change with time Non-stationary signals have time varying spectra Polikar, (C) 2007 Concatenation

Non-stationary Signals 5 Hz 20 Hz 50 Hz Polikar, (C) 2007 Perfect knowledge of what frequencies exist, but no information about where these frequencies are located in time

FT Shortcomings Polikar, (C) 2007 Complex exponentials stretch out to infinity in time They analyze the signal globally, not locally Hence, FT can only tell what frequencies exist in the entire signal, but cannot tell, at what time instances these frequencies occur In order to obtain time localization of the spectral components, the signal need to be analyzed locally HOW ? Polikar, (C) 2007

Short Time Fourier Transform (STFT) Choose a window function of finite length Put the window on top of the signal at t=0 Truncate the signal using this window Compute the FT of the truncated signal, save. Slide the window to the right by a small amount Go to step 3, until window reaches the end of the signal For each time location where the window is centered, we obtain a different FT Hence, each FT provides the spectral information of a separate time-slice of the signal, providing simultaneous time and frequency information Polikar, (C) 2007

STFT Example 300 Hz 200 Hz 100Hz 50Hz Polikar, (C) 2007

STFT Example Polikar, (C) 2007

STFT Example Polikar, (C) 2007 a=0.01

STFT Example Polikar, (C) 2007 a=0.001

STFT Example Polikar, (C) 2007 a=0.0001

STFT Example Polikar, (C) 2007 a=0.00001

Time – Frequency Resolution Time – frequency resolution problem with STFT Analysis window dictates both time and frequency resolutions, once and for all Narrow window  Good time resolution Narrow band (wide window)  Good frequency resolution When do we need good time resolution, when do we need good frequency resolution? Polikar, (C) 2007

Heisenberg Principle Polikar, (C) 2007 Time resolution: How well two spikes in time can be separated from each other in the transform domain Frequency resolution: How well two spectral components can be separated from each other in the transform domain Both time and frequency resolutions cannot be arbitrarily high!!!  We cannot precisely know at what time instance a frequency component is located. We can only know what interval of frequencies are present in which time intervals

The Wavelet Transform Polikar, (C) 2007 Overcomes the preset resolution problem of the STFT by using a variable length window Analysis windows of different lengths are used for different frequencies: Analysis of high frequencies Use narrower windows for better time resolution Analysis of low frequencies  Use wider windows for better frequency resolution This works well, if the signal to be analyzed mainly consists of slowly varying characteristics with occasional short high frequency bursts. Heisenberg principle still holds!!! The function used to window the signal is called the wavelet Polikar, (C) 2007

Polikar, (C) 2007 1:44,500,000 1:2,500,000 1:375,500 1:62,500

Continuous Wavelet Transform translation Mother wavelet Polikar, (C) 2007 Normalization factor Scaling: Changes the support of the wavelet based on the scale (frequency) CWT of x(t) at scale a and translation b Note: low scale  high frequency

Why Wavelet? Polikar, (C) 2007 We require that the wavelet functions, at a minimum, satisfy the following: Polikar, (C) 2007 Wave… …let

Time-Frequency Grid in CWT Scale Polikar, (C) 2007 a=1/4 4Δf Δt/4 a=1/2 Δt/2 2Δf a=1 Δt Δf a=2 2Δt Δf/2 4Δt Time Δt * Δf ≥ ½

Matlab Demos on CWT Polikar, (C) 2007

Discretization of Time & Scale Parameters Recall that, if we use orthonormal basis functions as our mother wavelets, then we can reconstruct the original signal by where W(a,b) is the CWT of x(t) Q: Can we discretize the mother wavelet a,b(t) in such a way that a finite number of such discrete wavelets can still form an orthonormal basis (which in turn allows us to reconstruct the original signal)? If yes, how often do we need to sample the translation and scale parameters to be able to reconstruct the signal? A: Yes, but it depends on the choice of the wavelet! Polikar, (C) 2007

Dyadic Grid Polikar, (C) 2007 We do not have to use a uniform sampling rate for the translation parameters, since we do not need as high time sampling rate when the scale is high (low frequency). Let’s consider the following sampling grid: Polikar, (C) 2007 b where a is sampled on a log scale, and b is sampled at a higher rate when a is small, that is, where a0 and b0 are constants, and j,k are integers. log a

An example on MRA Polikar, (C) 2007 f(t) f0(t) f2(t) Let’s suppose we wish to approximate the function f(t) using a simpler function that has constant values over set intervals of 2j, such that the error ||f(t)-fj(t)|| is minimum for any j. The best approx. would be the average of the function over the interval length. In general, then… f(t) j=- Polikar, (C) 2007 j=-2 j=-1 f0(t) j=0 j=1 f2(t) j=2 j=4 j=5 j=

An Example (cont.) Polikar, (C) 2007 As we go from a value of j to a higher value, we obtain a coarser approximation of the function, hence some detail is lost. Let’s denote the detail lost at each approximation level with a function g(t): Polikar, (C) 2007 Detail function Approximation function That is, the original function can be reconstructed simply by adding all the details (!)

Scaling & Wavelet Functions The scaling function Smoothed, scaling-function-dependent approximation of x(t) at scale j Approximation coefficients at scale j Polikar, (C) 2007 The wavelet function Wavelet-function-dependent detail of x(t) at scale j Detail coefficients at scale j Furthermore: & Wavelet Reconstruction

Putting It All Together (cont.) We can start reconstruction from any level (scale) in between: Polikar, (C) 2007 Approximation coefficients at some arbitrary scale j0 Detail coefficients at scale j0 and below Wavelet-function-dependent details of x(t) at scales j0 and below Smoothed, scaling-function-dependent approximation of x(t) at the scale j0 The essence of wavelet transform: A signal can be obtained from (or decomposed into) an approximation at a specific level, and sum of details lost up to that level.

Discrete Wavelet Transform Implementation G H 2 + x[n] Decomposition Reconstruction ~ Polikar, (C) 2007 2 Down-sampling Up-sampling G Half band high pass filter Half band low pass filter H 2 2-level DWT decomposition. The decomposition can be continues as long as there are enough samples for down-sampling.

DWT - Demystified Polikar, (C) 2007 g[n] h[n] g[n] h[n] g[n] h[n] Length: 512 B: 0 ~  |H(jw)| w /2 -/2 g[n] h[n] Length: 256 B: 0 ~ /2 Hz Polikar, (C) 2007 Length: 256 B: /2 ~  Hz 2 2 a1 d1: Level 1 DWT Coeff. |G(jw)| w  - /2 -/2 g[n] h[n] Length: 128 B: 0 ~  /4 Hz Length: 128 B: /4 ~ /2 Hz 2 2 a2 d2: Level 2 DWT Coeff. g[n] h[n] 2 2 Length: 64 B: 0 ~ /8 Hz Length: 64 B: /8 ~ /4 Hz …a3…. d3: Level 3 DWT Coeff. Level 3 approximation Coefficients

Matlab Implementation Command Line Single level decomposition and reconstruction [cA1 cD1]=dwt(x, ‘db4’) single level decomposition x=idwt(cA1, cD1, ‘db4’):3 single level inverse transform Direct reconstruction of approximation and detail signals D1=upcoef(‘d’, cD1, ‘db4’, 1, N): one-level reconstruction of detail signal from detail coefficient A1=upcoef(‘a’, cA1, ‘db4’, 1, N): one level reconstruction of approximation signal from approximation coefficients Multi-level decomposition & reconstruction [C L]=wavedec(x, K, ‘db4’): k-level decomposition A0=waverec(C, L, ‘db4’): reconstruct the signal Extracting coefficients cA3=appcoef(C, L, ‘db4’, 3); cD5=detcoef(C, L, 5); Single branch reconstruction A3=wrcoef(‘a’, C, L, ‘db4’, 3); Reconstruct level 3 approx. signal D2=wrcoef(‘d’, C, L, ‘db4’, 2); Reconstruct level 2 detail signal Polikar, (C) 2007 L=[lA3 lD3 lD2 lD1 lsignal] C= A3 D3 D2 D1 lA3 lD3 lD2 lD1

Quadrature Mirror Filters It can be shown that that is, h[] and g[] filters are related to each other: in fact, that is, h[] and g[] are mirrors of each other, with every other coefficient negated. Such filters are called quadrature mirror filters. For example, Daubechies wavelets with 4 vanishing moments….. Polikar, (C) 2007

DB-4 Wavelets Polikar, (C) 2007 ~ ~ L: filter length (8, in this case)

Implementation of DWT on MATLAB Choose wavelet and number of levels Load signal Hit Analyze button s=a5+d5+…+d1 Polikar, (C) 2007 Approx. coef. at level 5 Level 1 coeff. Highest freq. (Wavedemo_signal1)

Details vs. Approximations Polikar, (C) 2007

Applications Polikar, (C) 2007 Detect discontinuities

Applications Polikar, (C) 2007 Detect hidden discontinuities

Applications Polikar, (C) 2007 Simple denoising

Compression Polikar, (C) 2007 DWT is commonly used for compression, since most DWT are very small, can be zeroed-out! Polikar, (C) 2007

Compression Polikar, (C) 2007

1D-DWT2D-DWT Polikar, (C) 2007 Recall fundamental concepts of 1D-DWT Provides time-scale (frequency) representation of non-stationary signals Based on multiresolution approximation (MRA) Approximate a function at various resolutions using a scaling function, (t) Keep track of details lost using wavelet functions, (t) Reconstruct the original signal by adding approximation and detail coeff. Implemented by using a series of lowpass and highpass filters Lowpass filters are associated with the scaling function and provide approximation Highpass filters are associated with the wavelet function and provide detail lost in approximating the signal Polikar, (C) 2007

2-D DWT How do we generalize these concepts to 2D? 2D functions  images f(x,y)  I[m,n] intensity function What does it mean to take 2D-DWT of an image? How do we interpret? How can we represent an image as a function? How do we define low frequency / high frequency in an image? How to we compute it? Why would we want to take 2D-DWT of an image anyway? Compression Denoising Feature extraction Polikar, (C) 2007

2D Scaling/Wavelet Functions We start by defining a two-dimensional scaling and wavelet functions: If (t) is orthogonal to its own translates, is also orthogonal to its own translates. Then, if fo(x,y) is the projection of f(x,y) on the space Vo generated by s(x,y): Polikar, (C) 2007

2D-DWT Just like in 1D we generated an approximation of the 2D function f(x,y). Now, how do we compute the detail lost in approximating this function? Unlike 1D case there will be three functions representing the details lost: Details lost along the horizontal direction Details lost along the vertical direction Details lost along the diagonal direction 1D  Two sets of coeff.; a(k,n) & d (k,n) 2D Four sets of coefficients: a(k,n), b(k, n), c(k, n) & d(k,n) Polikar, (C) 2007

Four Faces of 2D-DWT Polikar, (C) 2007 One level of 2D DWT reconstruction: Polikar, (C) 2007 Approximation coefficients Detail coefficients along the horizontal direction Detail coefficients along the vertical direction Detail coefficients along the diagonal direction

Implementation of 2D-DWT INPUT IMAGE …… ROWS COLUMNS H ~ 2 1 G 1 2 LL LH HL HH LHH LLH LHL LLL Polikar, (C) 2007

Up and Down … Up and Down Polikar, (C) 2007 2 1 1 2 2 1 1 2 Downsample columns along the rows: For each row, keep the even indexed columns, discard the odd indexed columns 2 1 Polikar, (C) 2007 Downsample columns along the rows: For each column, keep the even indexed rows, discard the odd indexed rows 1 2 Upsample columns along the rows: For each row, insert zeros at between every other sample (column) 2 1 Upsample rows along the columns: For each column, insert zeros at between every other sample (row) 1 2

Implementing 2D-DWT Decomposition COLUMN j ROW i Polikar, (C) 2007

Reconstruction Polikar, (C) 2007 LL H 1 2 2 1 H LH G 1 2 HL H 1 2 G 1 2 H 2 1 H Polikar, (C) 2007 LH 1 2 G ORIGINAL IMAGE HL 1 2 H 2 1 G HH 1 2 G

2-D DWT ON MATLAB Polikar, (C) 2007 Load Image Choose (must be .mat file) Choose wavelet type Hit Analyze Polikar, (C) 2007 Choose display options

2D Wavelets Other uses for Wavelets in image processing (as seen in later slides) Image Compression The Discrete wavelet transform decomposes an image into a set of successfully smaller orthogonal images. Often it is possible to coarsely quantize or eliminate low valued coefficients without sacrificing the integrity of the image Polikar, (C) 2007 5MB 5MB 2.5MB

2D Wavelets Polikar, (C) 2007 2) Image Enhancement Since the DWT decomposes the image into components of different size, position and orientation, you can alter the coefficients before reconstruction so that you can attenuate certain characteristics of the image. Polikar, (C) 2007

2D Wavelets Polikar, (C) 2007 3) Image Fusion Image fusion combines 2 or more registered images of the same object into a single object, that in most cases is better than the original. This is helpful in the medical imaging field where multiple images from different machines. The images are combined in the transform domain by taking the highest amplitude coefficient then performing the inverse on the new fused image. Polikar, (C) 2007 Run Image Fusion GUI

Image Denoising Polikar, (C) 2007

Project Idea Polikar, (C) 2007 Implement JPEG 2000 A much improved form of JPEG compression that primarily uses the wavelet based compression idea we just discussed. Polikar, (C) 2007