Fourier Transformations Jeff Edmonds York University COSC 6111 Change from Time to Polynomial Basis Evaluating & Interpolating FFT in nlogn Time Roots of Unity Same FFT Code & Butterfly Inverse FFT Sin and Cos Basis FFT Butterfly Polynomial Multiplication Integer Multiplication

Fourier Transformation Fourier Transform are a change of basis from the time basis to sine/cosine basis JPG or polynomial basis Purposes: Signal Processing Compressing data (eg images with.jpg) Multiplying integers in n logn loglogn time. Error correcting codes. …. The Scientist and Engineer's Guide to Digital Signal Processing By Steven W. Smith, Ph.D. Amazingly once you include complex numbers, the FFT code for sine/cosines and for polynomials are the SAME.

Fourier Transformation Instead of using sine and cosines as the basis, Polynomial Basis

Fourier Transformation Instead of using sine and cosines as the basis, We use polynomials. Polynomial Basis

Change of Basis: T([a 1,a 2,…,a d ]) = [A 1,A 2,…,A d ] Changes the basis used to describe an object. The Time basis of a vector space: A tuple of basis objects Linearly independent Spans the space uniquely  v  [a 1,a 2,…,a d ], v = a 1 w 1 +a 2 w 2 +… + a d w d The Polynomial basis of a vector space: A tuple of basis objects Linearly independent Spans the space uniquely  v  [A 1,A 2,…,A d ], v = A 1 W 1 +A 2 W 2 +… + A d W d Polynomial Basis Fourier Transformation

[3,2] v = [a 1,a 2 ] =[1 1 / 5,3 2 / 5 ] [A 1,A 2 ] = Change of Basis: T([a 1,a 2,…,a d ]) = [A 1,A 2,…,A d ] Changes the basis used to describe an object.  v  [a 1,a 2,…,a d ], v = a 1 w 1 +a 2 w 2 +… + a d w d  v  [A 1,A 2,…,A d ], v = A 1 W 1 +A 2 W 2 +… + A d W d =[w 1,w 2 ] =[, ] Standard Basis v = Polynomial Basis New Basis =[W 1,W 2 ] = [, ] W 1 [1] W 1 [2] W 2 [1] W 2 [2] [ ][ ] = [ ] W 1 [1] W 1 [2] W 2 [1] W 2 [2] a1a1 a2a2 A1A1 A2A2 Fourier Transformation

 f = a 0 ∙B 0 +a 1 ∙B 1 +a 2 ∙B 2 +… + a n-1 ∙B n-1 =[B 0,B 1,..] f = a 0 =1 1 / 5 a 1 =3 2 / 5 Polynomial Basis =[, ] Change of Basis: T([y[0],y[1],…, y[n-1]]) = [a 1,a 2,…,a n-1 ] Changes the basis used to describe an object. A discrete function x f(x) f(x) = a 0 +a 1 x +a 2 x 2 + …+ a n-1 x n-1 coefficients of the polynomial [a 1,a 2,…,a n-1 ] = Polynomial Basis

Have you seen Taylor Expansions of a Function? They show that functions f(x) can be expressed by specifying the coefficients of the polynomial. F(x) = a 0 +a 1 x +a 2 x 2 +a 3 x 3 + … Eg: f(x) = 1/(1-x) F(x) = 1+x +x 2 +x 3 + … Fourier Transformation Polynomial Basis

Have you seen Taylor Expansions of a Function? They show that functions f(x) can be expressed by specifying the coefficients of the polynomial. F(x) = a 0 +a 1 x +a 2 x 2 +a 3 x 3 + … Fourier Transformation Polynomial Basis

Have you seen Taylor Expansions of a Function? They show that functions f(x) can be expressed by specifying the coefficients of the polynomial. F(x) = a 0 +a 1 x +a 2 x 2 +a 3 x 3 + … Fourier Transformation Polynomial Basis

Change of Basis: T([y[0],y[1],…, y[n-1]]) = [a 1,a 2,…,a n-1 ] Changes the basis used to describe an object. Time Basis =[, ] f = Fourier Transformation y[0]=3 y[1]=2 =[I 0,I 1,…] =[?,?] The time basis x I j [x] zero one j  y  [y[0],y[1],…,y[n-1]], y = y[0]I 0 +y[1]I 1 +… + y[n-1]I n-1 Polynomial Basis A discrete function x f(x) the value f(j) of the function at x=j. y[j] = the value f(j) of the function at x=j.

Fourier Transformation Instead of evaluating the polynomials on j = 0, 1, 2,…, n-1 We evaluate them on x 0, x 1, x 2, …, x n-1 These x j are special fixed values. Later we set x j = e 2  i j/n Polynomial Basis the value f(j) of the function at x=j. y[j] = the value f(j) of the function at x=j.

Change of Basis: T([y[0],y[1],…, y[n-1]]) = [a 1,a 2,…,a n-1 ] Changes the basis used to describe an object. Time Basis =[, ] f = Fourier Transformation y[0]=3 y[1]=2 =[I 0,I 1,…] =[?,?] The time basis x I j [x] zero one xjxj  y  [y[0],y[1],…,y[n-1]], y = y[0]I 0 +y[1]I 1 +… + y[n-1]I n-1 Polynomial Basis A discrete function x f(x) the value f(x j ) of the function at x=j. y[j] = the value f(j) of the function at x=j. x 0 x 1 x 2 x 3 x 4 … x n-1

A Fourier Transform is a change in basis. It changes the representation of a function from the coefficients of the polynomial f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 This amounts to evaluating f at these points. Evaluating & Interpolating x to the value f(x i ) at key values x i. x 0 x 1 x 2 x 3 x 4 … x n-1 y 0 y 1 y 2 y 3 y 4 … y n-1 Fourier Transformation y i = f(x i )

A Fourier Transform is a change in basis. It changes the representation of a function Fourier Transformation from the coefficients of the polynomial f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 This amounts to evaluating f at these points. (x 0 ) 0 (x 0 ) 1 (x 0 ) 2 (x 0 ) 3 … (x 0 ) n-1 a 0 a 1 a 2 a 3 … a n-1 y 0 y 1 y 2 y 3 … y n-1 = (x 1 ) 0 (x 1 ) 1 (x 1 ) 2 (x 1 ) 3 … (x 1 ) n-1 (x n-1 ) 0 (x n-1 ) 1 (x n-1 ) 2 (x n-1 ) 3 …(x n-1 ) n-1 (x 2 ) 0 (x 2 ) 1 (x 2 ) 2 (x 2 ) 3 … (x 2 ) n-1 (x 3 ) 0 (x 3 ) 1 (x 3 ) 2 (x 3 ) 3 … (x 3 ) n-1 Vandermonde matrix Invertible if x i distinct. Evaluating & Interpolating y i = f(x i )

Fourier Transformation to the coefficients of the polynomial f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 This amounts to interpolating these points. An Inverse Fourier Transform is the reverse. It changes the representation of a function Evaluating & Interpolating x from the value f(x i ) at key values x i. x 0 x 1 x 2 x 3 x 4 … x n-1 y 0 y 1 y 2 y 3 y 4 … y n-1 y i = f(x i )

Fourier Transformation to the coefficients of the polynomial f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 This amounts to interpolating these points. Given a set of n points in the plane with distinct x-coordinates, there is exactly one (n-1)-degree polynomial going through all these points. An Inverse Fourier Transform is the reverse. It changes the representation of a function Evaluating & Interpolating

f(x) = a 0 +x(a 1 +x(a 2 + … + x(a n-1 ) … )) Fourier Transformation My input: (a 0,a 1,a 2,…,a n-1 ) & x My output: f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n n-1 multiplications? n multiplications and n additions. Given n distinct values (x 0,x 1,x 2,…,x n-1 ). Can they all be done in less than n×O(n) time? Evaluating & Interpolating

Fourier Transformation My input: (a 0,a 1,a 2,…,a n-1 ) & x My output: f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 Given n distinct values (x 0,x 1,x 2,…,x n-1 ) Can they all be done in less than n×O(n) time? (x 0 ) 0 (x 0 ) 1 (x 0 ) 2 (x 0 ) 3 … (x 0 ) n-1 a 0 a 1 a 2 a 3 … a n-1 y 0 y 1 y 2 y 3 … y n-1 = (x 1 ) 0 (x 1 ) 1 (x 1 ) 2 (x 1 ) 3 … (x 1 ) n-1 (x n-1 ) 0 (x n-1 ) 1 (x n-1 ) 2 (x n-1 ) 3 …(x n-1 ) n-1 (x 2 ) 0 (x 2 ) 1 (x 2 ) 2 (x 2 ) 3 … (x 2 ) n-1 (x 3 ) 0 (x 3 ) 1 (x 3 ) 2 (x 3 ) 3 … (x 3 ) n-1 y i = f(x i ) I hope so Evaluating & Interpolating

The Fast Fourier Transform (FFT) is a very efficient algorithm for performing a discrete Fourier transform FFT principle first used by Gauss in 18?? (But was not interesting without computers) FFT algorithm published by Cooley & Tukey in 1965 In 1969, the 2048 point analysis of a seismic trace took 13 ½ hours. Using the FFT, the same task on the same machine took 2.4 seconds! Fast Fourier Transformation FFT nlogn Time

Error (ppm) DFTFFT Not only do you get faster speed + in place memory processing but fewer calculations means less round off errors Maybe I should take CSE6111 after all! Fast Fourier Transformation FFT nlogn Time

N DFT (N 2 ) FFT (1.5N log N)faster 321, , ,3841, ,5363, ,1446, ,048,57615, ,194,30433, ,777,21673, Discrete Fourier Transform is too slow for real time! Fast Fourier Transformation FFT nlogn Time

Divide & Conquer - Friends - Recursion. Trust your friends to solve any subinstance: as long as smaller and is an instance to the same problem. Fast Fourier Transformation My instance My friend’s Instance My friend’s Instance My friend’s Instance FFT nlogn Time

f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 Fast Fourier Transformation My input: (start with one x) (a 0,a 1,a 2,…,a n-1 ) & x = a 0 +a 2 x 2 +a 4 x 4 + … + a n-2 x n-2 + a 1 x+a 3 x 3 +a 5 x 5 + … + a n-1 x n-1 = a 0 +a 2 x 2 +a 4 x 4 + … + a n-2 x n-2 + x( a 1 +a 3 x 2 +a 5 x 4 + … + a n-1 x n-2 ) = f even (x 2 ) + x( f odd (x 2 ) ) My output: f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 1 st friend’s input? f even : (a 0,a 2,a 4,…,a n-2 ) & ? 2 nd friend’s input? f odd : (a 1,a 3,a 5,…,a n-1 ) & ? f(x) = f even (x 2 ) + x f odd (x 2 ) f even (z) = a 0 +a 2 z+a 4 z 2 +a 6 z 3 + … + a n-2 z n/2-1 FFT nlogn Time

Fast Fourier Transformation My input: (start with one x) (a 0,a 1,a 2,…,a n-1 ) & x My output: f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 1 st friend’s input? f even : (a 0,a 2,a 4,…,a n-2 ) & x 2 2 nd friend’s input? f odd : (a 1,a 3,a 5,…,a n-1 ) & x 2 1 st friend’s output: y even = f even (x 2 ) 2nd friend’s output: y odd = f odd (x 2 ) f(x) = f even (x 2 ) + x f odd (x 2 ) My output: f(x) = y even + x y odd T(n) = 2 T(n/2) + O(1) = O(n) Ok. So it takes O(n) time to evaluate. FFT nlogn Time

Fast Fourier Transformation My input: (a 0,a 1,a 2,…,a n-1 ) (x 0,x 1,x 2,…,x n-1 ) My output: (y 0,y 1,y 2,…,y n-1 ) y i = f(x i ) 1 st friend’s input? f even : (a 0,a 2,a 4,…,a n-2 ) (x 0 2,x 1 2,x 2 2,…,x n-1 2 ) 2 nd friend’s input? f odd : (a 1,a 3,a 5,…,a n-1 ) (x 0 2,x 1 2,x 2 2,…,x n-1 2 ) 1 st friend’s output:  i y = f even (x i 2 ) 2 nd friend’s output:  i y = f odd (x i 2 ) My output:  i f(x i ) = y + x i y T(n) = 2 T(n/2) + O(n) = O(n log n) Wow! That was easy. FFT nlogn Time

Fast Fourier Transformation My input: (a 0,a 1,a 2,…,a n-1 ) (x 0,x 1,x 2,…,x n-1 ) 1 st friend’s input? f even : (a 0,a 2,a 4,…,a n-2 ) (x 0 2,x 1 2,x 2 2,…,x n-1 2 ) Oops n coefficients n values of x n / 2 coefficients n values of x Does not meet precondition! FFT nlogn Time

Fast Fourier Transformation My input: (a 0,a 1,a 2,…,a n-1 ) (x 0,x 1,x 2,…,x n-1 ) My output: (y 0,y 1,y 2,…,y n-1 ) y i = f(x i ) 1 st friend’s input? f even : (a 0,a 2,a 4,…,a n-2 ) (x 0 2,x 1 2,x 2 2,…,x n/2-1 2 ) 2 nd friend’s input? f odd : (a 1,a 3,a 5,…,a n-1 ) (x 0 2,x 1 2,x 2 2,…,x n/2-1 2 ) My output:  i f(x i ) = y + x i y T(n) = 4 T(n/2) + O(n) That’s no good. 3 rd friend’s input? f even : (a 0,a 2,a 4,…,a n-2 ) (x n/2 2,x n/2+1 2,…,x n-1 2 ) 4 th friend’s input? f odd : (a 1,a 3,a 5,…,a n-1 ) (x n/2 2,x n/2+1 2,…,x n-1 2 ) = O(n 2 ) FFT nlogn Time

Fast Fourier Transformation The values (x 0,x 1,x 2,…,x n-1 ) are said to be special if: There are n distinct values. When you square each of them: The set collapses to n / 2 distinct values. Eg: …, -3, -2, -1, 1, 2, 3, … square each of them …, 9, 4, 1, 1, 4, 9, … collapse the set 1, 4, 9, … half as many elements. Roots of Unity

Fast Fourier Transformation My input: (a 0,a 1,a 2,…,a n-1 ) Special (x 0,x 1,x 2,…,x n-1 ) My output: (y 0,y 1,y 2,…,y n-1 ) y i = f(x i ) 1 st friend’s input? f even : (a 0,a 2,a 4,…,a n-2 ) (x 0 2,x 1 2,x 2 2,…,x n-1 2 ) n / 2 distinct values 2 nd friend’s input? f odd : (a 1,a 3,a 5,…,a n-1 ) (x 0 2,x 1 2,x 2 2,…,x n-1 2 ) n / 2 distinct values 1 st friend’s output:  i y = f even (x i 2 ) 2 nd friend’s output:  i y = f odd (x i 2 ) My output:  i f(x i ) = y + x i y T(n) = 2 T(n/2) + O(n) = O(n log n) That’s better -3 3 f even (9) f odd (9) 9 9 f even (9)f odd (9) 3 f(3) -3 f(-3) Roots of Unity

Fast Fourier Transformation My input: (a 0,a 1,a 2,…,a n-1 ) Special (x 0,x 1,x 2,…,x n-1 ) My output: (y 0,y 1,y 2,…,y n-1 ) y i = f(x i ) 1 st friend’s input? f even : (a 0,a 2,a 4,…,a n-2 ) (x 0 2,x 1 2,x 2 2,…,x n-1 2 ) n / 2 distinct values 2 nd friend’s input? f odd : (a 1,a 3,a 5,…,a n-1 ) (x 0 2,x 1 2,x 2 2,…,x n-1 2 ) n / 2 distinct values To meet precondition these also need to be special Roots of Unity

Fast Fourier Transformation The values (x 0,x 1,x 2,…,x n-1 ) are said to be special if: There are n distinct values. When you square each of them: The set collapses to n / 2 distinct values. Which are also special Eg: …, -3, -2, -1, 1, 2, 3, … square each of them …, 9, 4, 1, 1, 4, 9, … collapse the set 1, 4, 9, … But these are not special. square each of them 2, 16, 81, … Roots of Unity

Fast Fourier Transformation The values (x 0,x 1,x 2,…,x n-1 ) are said to be special if: There are n distinct values. When you square each of them: The set collapses to n / 2 distinct values. Which are also special Eg: -i, -1, 1, i square each of them -1, 1, 1, -1 collapse the set -1, 1 square each of them 1, 1 collapse the set 1  = -i,-1, 1, i are said to be 4 th roots of unity Because  4 = 1 Roots of Unity

Fast Fourier Transformation  is said to be an n th root of unity (in a field) if  n = 1 (There should be n solutions of this polynomial) Fermat’s Little Theorem:  b≠0 b p-1 = mod p 1 says every nonzero element is an n th root of unity when n=p-1. Roots of Unity

Fast Fourier Transformation  is said to be an n th root of unity (in a field) if  n = 1  is said to be a generator of the field if the numbers 1, ,  2, …,  n-1 are all distinct 1, ,  2, …,  n-1 are then special (when n is even)  n/2+n/2-1,…,  n/2+3,  n/2+2,  n/2+1,  n/2+0,  0,  1,  2,  3,…,  n/2-1 square each of them  n+n-2, …,  n+6,  n+4,  n+2,  n+0,  0,  2,  4,  6, …,  n-2 use  n = 1  n-2, …,  6,  4,  2,  0,  0,  2,  4,  6, …,  n-2 collapse the set  0,  2,  4,  6, …,  n-2 We need these to be n / 2 special values. 1 st half 2 nd half Roots of Unity

 16 1  16 0  16 2  16 3  16 4  16 5  16 6  16 7  16 8  16 9       =  = 1 Fast Fourier Transformation 16 th roots of unity -1 = i -i (  n/2 ) 2 = 1 (  n/4 ) 2 =  n/2 = -1 (  3n/4 ) 2 =  n/2 = -1 These could be Z mod 17 or complex numbers ××r r θ re θi = rcosθ + irsinθ re θi × se αi = (rs)e (θ+α)i Roots of Unity

Fast Fourier Transformation Goal: Proof f(θ) = g(θ) f(0) = g(0) f’(0) = g’(0) f’’(θ) = -f(θ) g’’(θ) = -g(θ) Proof by induction (over the reals) that f(θ) = g(θ) f(θ)g(θ) For this θ, f(θ) = g(θ) and f’(θ) = g’(θ) For next θ+ , f(θ+  ) = g(θ+  ) f’’(θ) = -f(θ) = -g(θ) =g’’(θ) For next θ+ , f’(θ+  ) = g’(θ+  ) Roots of Unity

Fast Fourier Transformation f(θ) = re θi g(θ) = rcosθ + irsinθ Goal: Proof f(θ) = g(θ) f(0) = re 0i = r g(0) = rcos0 + irsin0 = r f’(θ) = ire θi g’(θ) = -rsinθ + ircosθ f(0) = g(0) f’(0) = ire 0i = ir g’(0) = -rsin0 + ircos0 = ir f’(0) = g’(0) f’’(θ) = -re θi g’’(θ) = -rcosθ - rsinθ = -f(θ)= -g(θ) Roots of Unity

 16 1  16 0  16 2  16 4  16 5  16 6  16 7  16 8  16 9       Fast Fourier Transformation 16 th roots of unity square each of them  16 3 and collapse Roots of Unity

 16 0  16 2  16 4  16 6  16 8    Fast Fourier Transformation 16 th roots of unity Are these special? square each of them and collapse Roots of Unity

8080 8181 8282 8383 8484 8585 8686 8787 Fast Fourier Transformation 8 th roots of unity Are these special? square each of them and collapse Roots of Unity

4040 4141 4242 4343 Fast Fourier Transformation 4 th roots of unity Are these special? square each of them and collapse Roots of Unity

2020 2121 Fast Fourier Transformation 2 th roots of unity Are these special? = 1 square each of them and collapse Roots of Unity

Fast Fourier Transformation My input: (a 0,a 1,a 2,…,a n-1 ) (n th roots of unity  n i ) My output: (y 0,y 1,y 2,…,y n-1 ) y i = f(  n i ) 1 st friend’s input? f even : (a 0,a 2,a 4,…,a n-2 ) ( n / 2 th roots of unity  n/2 i ) 2 nd friend’s input? f odd : (a 1,a 3,a 5,…,a n-1 ) ( n / 2 th roots of unity  n/2 i ) 1 st friend’s output:  i y = f even (  n/2 i ) 2 nd friend’s output:  i y = f odd (  n/2 i ) My output:  i f(x i ) = y + x i y T(n) = 2 T(n/2) + O(n) = O(n log n) Excellent Roots of Unity

Fourier Transformation Algorithm FFT(y, , n): Input: y = [a 0,a 1,a 2,…,a n-1 ] (Time Domain)  = e 2  i 1/n (n th root of unity) n = # of samples (2 r ) Output: Y = [y 0,y 1,y 2,…,y n-1 ] (Frequency Domain) % Separate even and odd indices a even = [a 0,a 2,a 4,…,a n-2 ] a odd = [a 1,a 1,a 5,…,a n-1 ] % Recurse y even =FFT(a even,  2, n / 2 ) (  2 = e 2  i 2/n ) y odd =FFT(a odd,  2, n / 2 ) %Combining For i = 0 to n / 2 -1 y[i] = y even [i] +  i ∙y odd [i] y[i+ n / 2 ] = y even [i] +  i+n/2 ∙y odd [i] Return(Y) FFT Code

A inverse Fourier transform changes the representation of a function x 0 x 1 x 2 x 3 x 4 … x n-1 y 0 y 1 y 2 y 3 y 4 … y n-1 y i = f(x i ) from the value f(x i ) at key values x i. to the coefficients of the polynomial f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 This amounts to interpolating these points. Inverse FFT Fast Fourier Transformation

A Fourier Transform is a change in basis. It changes the representation of a function Fast Fourier Transformation from the coefficients of the polynomial f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 This amounts to evaluating f at these points. (x 0 ) 0 (x 0 ) 1 (x 0 ) 2 (x 0 ) 3 … (x 0 ) n-1 a 0 a 1 a 2 a 3 … a n-1 y 0 y 1 y 2 y 3 … y n-1 = (x 1 ) 0 (x 1 ) 1 (x 1 ) 2 (x 1 ) 3 … (x 1 ) n-1 (x n-1 ) 0 (x n-1 ) 1 (x n-1 ) 2 (x n-1 ) 3 …(x n-1 ) n-1 (x 2 ) 0 (x 2 ) 1 (x 2 ) 2 (x 2 ) 3 … (x 2 ) n-1 (x 3 ) 0 (x 3 ) 1 (x 3 ) 2 (x 3 ) 3 … (x 3 ) n-1 (x i ) j y i = f(x i ) Inverse FFT

x i =  i (x 0 ) 0 (x 0 ) 1 (x 0 ) 2 (x 0 ) 3 … (x 0 ) n-1 a 0 a 1 a 2 a 3 … a n-1 y 0 y 1 y 2 y 3 … y n-1 = (x 1 ) 0 (x 1 ) 1 (x 1 ) 2 (x 1 ) 3 … (x 1 ) n-1 (x n-1 ) 0 (x n-1 ) 1 (x n-1 ) 2 (x n-1 ) 3 …(x n-1 ) n-1 (x 2 ) 0 (x 2 ) 1 (x 2 ) 2 (x 2 ) 3 … (x 2 ) n-1 (x 3 ) 0 (x 3 ) 1 (x 3 ) 2 (x 3 ) 3 … (x 3 ) n-1 (x i ) j (  0 ) 0 (  0 ) 1 (  0 ) 2 (  0 ) 3 … (  0 ) n-1 a 0 a 1 a 2 a 3 … a n-1 y 0 y 1 y 2 y 3 … y n-1 = (  1 ) 0 (  1 ) 1 (  1 ) 2 (  1 ) 3 … (  1 ) n-1 (  2 ) 0 (  2 ) 1 (  2 ) 2 (  2 ) 3 … (  2 ) n-1 (  3 ) 0 (  3 ) 1 (  3 ) 2 (  3 ) 3 … (  3 ) n-1 (  n-1 ) 0 (  n-1 ) 1 (  n-1 ) 2 (  n-1 ) 3 … (  n-1 ) n-1 (i)j(i)j Inverse FFT

(  i ) j =  ij Vandermonde matrix (  0 ) 0 (  0 ) 1 (  0 ) 2 (  0 ) 3 … (  0 ) n-1 a 0 a 1 a 2 a 3 … a n-1 y 0 y 1 y 2 y 3 … y n-1 = (  1 ) 0 (  1 ) 1 (  1 ) 2 (  1 ) 3 … (  1 ) n-1 (  2 ) 0 (  2 ) 1 (  2 ) 2 (  2 ) 3 … (  2 ) n-1 (  3 ) 0 (  3 ) 1 (  3 ) 2 (  3 ) 3 … (  3 ) n-1 (  n-1 ) 0 (  n-1 ) 1 (  n-1 ) 2 (  n-1 ) 3 … (  n-1 ) n-1 (i)j(i)j  0  0  0  0 …  0 a 0 a 1 a 2 a 3 … a n-1 y 0 y 1 y 2 y 3 … y n-1 =  0  1  2  3 …  n-1  0  2  4  6 …  2n-2  0  3  6  9 …  3n-3  0  n-1  2n-2  3n-3 …  (n-1)(n-1) ijij Inverse FFT

V  a = y a = V  -1 y Vandermonde matrix  0  0  0  0 …  0 a 0 a 1 a 2 a 3 … a n-1 y 0 y 1 y 2 y 3 … y n-1 =  0  1  2  3 …  n-1  0  2  4  6 …  2n-2  0  3  6  9 …  3n-3  0  n-1  2n-2  3n-3 …  (n-1)(n-1) ijij V  -1 = 1 / n V  -1 Inverse FFT

Vandermonde matrix  0  0  0  0 …  0 a 0 a 1 a 2 a 3 … a n-1 y 0 y 1 y 2 y 3 … y n-1 =  0  1  2  3 …  n-1  0  2  4  6 …  2n-2  0  3  6  9 …  3n-3  0  n-1  2n-2  3n-3 …  (n-1)(n-1) ijij V  -1 = 1 / n V  -1  0  0  0  0 …  0 y 0 y 1 y 2 y 3 … y n-1 a 0 a 1 a 2 a 3 … a n-1 =  0  -1  -2  -3 …  -(n-1)  0  -2  -4  -6 …  -(2n-2)  0  -3  -6  -9 …  -3(n-3)  0  -(n-1)  -(2n-2)  -(3n-3) …  -(n-1)(n-1)  -ij 1/n1/n Inverse FFT

If  is an n th root of unity and a generator of the field  16 1  16 0  16 2  16 3  16 4  16 5  16 6  16 7  16 8  16 9       =  = 1 -1 = r Fast Fourier Transformation Inverse FFT

The inverse  -1  of  is –then  -1 =  n-1 –And is also an n th root of unity and a generator of the field –Proof:  -1 =  n-1 =  n =1  16 1  16 0  16 2  16 3  16 4  16 5  16 6  16 7  16 8  16 9       =  = 1 -1 = r Fast Fourier Transformation Inverse FFT

Cancellation Property:  16 1  16 0  16 2  16 3  16 4  16 5  16 6  16 7  16 8  16 9       =  = 1 -1 = r Fast Fourier Transformation Inverse FFT

Vandermonde matrix  0  0  0  0 …  0 a 0 a 1 a 2 a 3 … a n-1 y 0 y 1 y 2 y 3 … y n-1 =  0  1  2  3 …  n-1  0  2  4  6 …  2n-2  0  3  6  9 …  3n-3  0  n-1  2n-2  3n-3 …  (n-1)(n-1) ijij V  -1 = 1 / n V  -1  0  0  0  0 …  0 y 0 y 1 y 2 y 3 … y n-1 a 0 a 1 a 2 a 3 … a n-1 =  0  -1  -2  -3 …  -(n-1)  0  -2  -4  -6 …  -(2n-2)  0  -3  -6  -9 …  -3(n-3)  0  -(n-1)  -(2n-2)  -(3n-3) …  -(n-1)(n-1)  -ij 1/n1/n Inverse FFT

The FFT and inverse FFT really are inverse operations Proof: Let A=D -1 D. We want to show that A=I, where If i=j, then If i and j are different, then Fast Fourier Transformation Inverse FFT

Fast Fourier Transformation The FFT and inverse FFT can use the same hardware FFT Input: Output: Inverse FFT Input: Output: Inverse FFT

Modifies DFT frequency coefficient calculations: ReX[ k ] = x[n] cos(2πkn/N) 0 < k < N/2 x[i] ε Real ImX[ k ] = - x[n] sin(2πkn/N) Uses complex and polar numbers as a shorthand: X k = ReX[ k ] + i ImX[ k ] X k = x n e – i2πkn/N = x n ω kn Ʃ Ʃ n=0 N-1 Complex Ʃ n=0 N-1 r·e iθ = r·cosθ + i r·sinθ = r θ Ʃ n=0 N-1 ω = e – i2π/N N = 2 r Fast Fourier Transformation Sin & Cos basis

1. Convert your N real sampled values to complex numbers by adding 0i to them 2. Feed this as the input to the FFT 3.Remove FFT output’s redundant information (i.e. all frequencies above N/2) x n = x n + 0i 0 < n < N-1 0 N/2 N-1 “Negative” Frequency ReX Even Symmetry About N/2 (f s /2) 0 N/2 ImX Odd Symmetry About N/2 (f s /2) “Negative” Frequency N-1 Fast Fourier Transformation Sin & Cos basis

Fast Fourier Transformation FFT Butterfly

(  0 ) 0 (  0 ) 1 (  0 ) 2 (  0 ) 3 … (  0 ) n-1 (  1 ) 0 (  1 ) 1 (  1 ) 2 (  1 ) 3 … (  1 ) n-1 (  2 ) 0 (  2 ) 1 (  2 ) 2 (  2 ) 3 … (  2 ) n-1 (  3 ) 0 (  3 ) 1 (  3 ) 2 (  3 ) 3 … (  3 ) n-1 (  n-1 ) 0 (  n-1 ) 1 (  n-1 ) 2 (  n-1 ) 3 …(  n-1 ) n-1... = x 0 x 1 x 2 x 3 … x N-1 X 0 X 1 X 2 X 3 … X N-1 Behold the Vandermonde matrix! But that’s O(N 2 ) !! The Ugly Math for the FFT

= x 0 x 1 x 2 x 3 … x N-1 X 0 X 1 X 2 X 3 … X N-1 But if I multiply the exponents...  0  n-1  2n-2  3n-3 …  (n-1)(n-1)  0  0  0  0 …  0  0  1  2  3 …  n-1  0  2  4  6 …  2n-2  0  3  6  9 …  3n-3... But that’s still O(N 2 ) !! The Ugly Math for the FFT

Just watch! For example, if N=8 and I use the N roots of unity... 0000000000000000 0123456701234567  0  2  4  6  8  10  12  14  0  3  6  9  12  15  18  21  0  4  8  12  16  20  24  28  0  5  10  15  20  25  30  35  0  6  12  18  24  30  36  42  0  7  14  21  28  35  42  49 = x0x1x2x3x4x5x6x7x0x1x2x3x4x5x6x7 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 44 00 11 22 33 55 66 77  p+4 = -  p  p =  p mod 8  =  8 = e -i2π/8  0 = 1  4 = -1

= x0x1x2x3x4x5x6x7x0x1x2x3x4x5x6x7 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X   2  3 -  -  2 -  3 1  2 -  2 1  2 -  2 1  3 -  2  -  3  2 -    2 -  3  -  2   2   2   3 -  2 -   3  2  Now the 2 nd half of each row either equals the 1 st half or its negative

= x0x1x2x3x4x5x6x7x0x1x2x3x4x5x6x7 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X  3 -  2  -  3  2 -   3 -  2 -   3  2  1   2  3 -  -  2 -    2 -  3  -  2  3 1  2 -  2 1  2 -   2   2  2 x4x4 x0x  x 0 and x 4 have identical coefficients (ignoring sign) as do: x 1 and x 5 x 2 and x 6 x 3 and x 7

= x0x1x2x3x4x5x6x7x0x1x2x3x4x5x6x7 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X  3 -  2  -  3  2 -   3 -  2 -   3  2  1   2  3 -  -  2 -    2 -  3  -  2  3 1  2 -  2 1  2 -   2   2  2 x1x1 x5x5

= x0x1x2x3x4x5x6x7x0x1x2x3x4x5x6x7 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7  x 0 and x 4 have identical coefficients (ignoring sign) as do: x 1 and x 5 x 2 and x 6 x 3 and x  3 -  2  -  3  2 -   3 -  2 -   3  2  1   2  3 -  -  2 -    2 -  3  -  2  3 1  2 -  2 1  2 -   2   2  2 x2x2 x6x

= x0x1x2x3x4x5x6x7x0x1x2x3x4x5x6x7 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7  x 0 and x 4 have identical coefficients (ignoring sign) as do: x 1 and x 5 x 2 and x 6 x 3 and x  3 -  2  -  3  2 -   3 -  2 -   3  2  1   2  3 -  -  2 -    2 -  3  -  2  3 1  2 -  2 1  2 -   2   2  2 x3x3 x7x

Now rewrite the matrix as equations in terms of: x 0 ± x 4, x 2 ± x 6, x 1 ± x 5, x 3 ± x 7 Oh my! Half the columns are gone. What’s next? (x 0 + x 4 ) + (x 2 + x 6 ) + (x 1 + x 5 ) + (x 3 + x 7 ) = X 0 (x 0 - x 4 ) +  2 (x 2 - x 6 ) +  (x 1 - x 5 ) +  3 (x 3 - x 7 ) = X 1 (x 0 + x 4 ) - (x 2 + x 6 ) +  2 (x 1 + x 5 ) -  2 (x 3 + x 7 ) = X 2 (x 0 - x 4 ) -  2 (x 2 - x 6 ) +  3 (x 1 - x 5 ) +  (x 3 - x 7 ) = X 3 (x 0 + x 4 ) + (x 2 + x 6 ) - (x 1 + x 5 ) - (x 3 + x 7 ) = X 4 (x 0 - x 4 ) +  2 (x 2 - x 6 ) -  (x 1 - x 5 ) -  3 (x 3 - x 7 ) = X 5 (x 0 + x 4 ) - (x 2 + x 6 ) -  2 (x 1 + x 5 ) +  2 (x 3 + x 7 ) = X 6 (x 0 - x 4 ) -  2 (x 2 - x 6 ) -  3 (x 1 - x 5 ) -  (x 3 - x 7 ) = X 7

Think signal flow and construct the equations using the butterfly operator: Ex. for (x 0 + x 4 ) + (x 2 + x 6 ) + (x 1 + x 5 ) + (x 3 + x 7 ) = X 0 (x 0 - x 4 ) +  2 (x 2 - x 6 ) +  (x 1 - x 5 ) +  3 (x 3 - x 7 ) = X 1 (x 0 + x 4 ) - (x 2 + x 6 ) +  2 (x 1 + x 5 ) -  2 (x 3 + x 7 ) = X 2 (x 0 - x 4 ) -  2 (x 2 - x 6 ) +  3 (x 1 - x 5 ) +  (x 3 - x 7 ) = X 3 (x 0 + x 4 ) + (x 2 + x 6 ) - (x 1 + x 5 ) - (x 3 + x 7 ) = X 4 (x 0 - x 4 ) +  2 (x 2 - x 6 ) -  (x 1 - x 5 ) -  3 (x 3 - x 7 ) = X 5 (x 0 + x 4 ) - (x 2 + x 6 ) -  2 (x 1 + x 5 ) +  2 (x 3 + x 7 ) = X 6 (x 0 - x 4 ) -  2 (x 2 - x 6 ) -  3 (x 1 - x 5 ) -  (x 3 - x 7 ) = X 7 Ʃ Ʃ ωpωp xoxo x4x4 xo + ωp x4xo + ωp x4 x o - ω p x 4 x 0 ± x 4 ( ω p = 1)

Note - the butterfly has a shorthand notation of: ωk ωk xoxo x4x4 xo + ωp x4xo + ωp x4 xo - ωp x4xo - ωp x4 (x 0 + x 4 ) + (x 2 + x 6 ) + (x 1 + x 5 ) + (x 3 + x 7 ) = X 0 (x 0 - x 4 ) +  2 (x 2 - x 6 ) +  (x 1 - x 5 ) +  3 (x 3 - x 7 ) = X 1 (x 0 + x 4 ) - (x 2 + x 6 ) +  2 (x 1 + x 5 ) -  2 (x 3 + x 7 ) = X 2 (x 0 - x 4 ) -  2 (x 2 - x 6 ) +  3 (x 1 - x 5 ) +  (x 3 - x 7 ) = X 3 (x 0 + x 4 ) + (x 2 + x 6 ) - (x 1 + x 5 ) - (x 3 + x 7 ) = X 4 (x 0 - x 4 ) +  2 (x 2 - x 6 ) -  (x 1 - x 5 ) -  3 (x 3 - x 7 ) = X 5 (x 0 + x 4 ) - (x 2 + x 6 ) -  2 (x 1 + x 5 ) +  2 (x 3 + x 7 ) = X 6 (x 0 - x 4 ) -  2 (x 2 - x 6 ) -  3 (x 1 - x 5 ) -  (x 3 - x 7 ) = X 7

Damn you and your re “cursed” friends! Using Bit Reverse Order and a tree of butterflies, my Decimation in Time Algorithm can solve this in O (N log N) No friends this time. They’d just be overhead invading my stack space. If you’re lying, I’ll claim your soul! Decimation in Time & Bit Reverse Order ( (rearranging the order of the N samples)

STAGE 1 2 – Point Butterfly 2 combined 2-Point Butterflies 4 combined 2-Point Butterflies STAGE 3STAGE 2 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 FFT Block Diagram

X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8282 8383 8080 8080 8080 8080 8282 8080 8080 8080 8282 Stepping Through the FFT  8 0 = 1  8 2 =  4 1

X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 8181 8282 8383 8080 8080 8080 8080 8282 8080 8080 8080 8282 Stepping Through the FFT x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7

X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8282 8383 8080 8080 8080 8080 8282 8080 8080 8080 8282 Stepping Through the FFT

8080 8080 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8282 8383 8080 8080 8080 8282 8080 8080 8282 Stepping Through the FFT

8080 8080 8080 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8282 8383 8080 8080 8080 8282 8080 8282 N/2 Stepping Through the FFT

8080 8080 8080 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8282 8383 8080 8080 8080 8282 8080 8282 N/2 Stepping Through the FFT

8080 8080 8080 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8282 8383 8080 8080 8080 8282 8080 8282 N/2 Stepping Through the FFT

8080 8080 8080 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8282 8383 8080 8080 8080 8282 8080 8282 N/2 Stepping Through the FFT

8080 8080 8080 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8282 8383 8080 8080 8080 8282 8080 8282 N/2 + N/2 Stepping Through the FFT

8080 8080 8080 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8282 8383 8080 8080 8080 8282 N/2 + N/2 Stepping Through the FFT 8080 8282

8080 8080 8080 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8282 8383 8080 8080 8080 8282 N/2 + N/2 Stepping Through the FFT 8080 8282

8080 8080 8080 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8383 8080 8080 8080 8282 N/2 + N/2 8282 Stepping Through the FFT 8080 8282 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7

8080 8080 X0X1X2X3X4X5X6X7X0X1X2X3X4X5X6X7 x0x4x2x6x1x5x3x7x0x4x2x6x1x5x3x7 8181 8383 8080 8080 8080 8080 8282 N/2 + N/2 + N/2  O (N log N) 8282 Stepping Through the FFT 8080 8282

Algorithm FFT (ReX, ImX) Input: ReX[ ], ImX[ ] = real, imaginary parts of the time samples Output: ReX[ ], ImX[ ] = cosine, sine coefficients of frequency domain N = SizeOf( ReX ) PutInBitReverseOrder (ReX, ImX) % time domain decomposition % frequency domain synthesis (done in place) for k = 1 to log 2 N % Loop for each stage Wre = 1; Wim = 0; θ = 2π/ 2 k % Initialize stage constants for j = 1 to 2 k-1 % Loop for each sub DFT for i = j-1 to N-1 step 2 k % Loop for each butterfly ip = i + 2 k-1 tmpRe = ReX[ip]·Wre - ImX[ip]·Wim tmpIm = ReX[ip]·Wim + ImX[ip]·Wre ReX[ip] = ReX[ i ] - tmpRe ImX[ip] = ImX[ i ] - tempIm ReX[ i ] = ReX[ i ] + tempRe ImX[ i ] = ImX[ i ] + tempIm next i tempRe = Wre Wre = tmpRe·cos(θ) + Wim·sin(θ) Wim = - tmpRe·sin(θ) + Wim· cos(θ) next j next k return (ReX, ImX) % ReX[ ],ImX[ ] return freq coeffs 0 to N-1

Polynomial Multiplication f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 g(x) = b 0 +b 1 x +b 2 x 2 + … + b n-1 x n-1 [f×g](x) = c 0 +c 1 x +c 2 x 2 + … +c 2n-2 x 2n-2 x 5 coefficient: c 5 = a 0 ×b 5 +a 1 ×b 4 + a 2 ×b 3 + … + a 5 ×b 0 Time = O(n 2 ) Too much Convolution

Polynomial Multiplication f(x) = a 0 +a 1 x +a 2 x 2 + … + a n-1 x n-1 g(x) = b 0 +b 1 x +b 2 x 2 + … + b n-1 x n-1 [f×g](x) = c 0 +c 1 x +c 2 x 2 + … +c 2n-2 x 2n-2 Coefficient Domain a j Evaluation Domain y i [a 0,a 1,a 2,…,a n-1 ] [b 0,b 1,b 2,…,b n-1 ] Fast Fourier Transform takes O(nlogn) time! y i = f(x i ) z i = g(x i ) y i ×z i = [g×f](x i ) Multipling values pointwise takes O(n) time! [c 0,c 1,c 2,…,c n-1 ]

Multiplying Big Integers X = 11… (N bits) Y = 10… X×Y = 10… The high school algorithm takes O(N 2 ) bit operations. Can we do it faster? I hope so See Recursion for one way to do it faster. This is another.

Grade School Revisited: How To Multiply Two Numbers Multiplying Big Integers

X = 0011 … Break into m = O(log N) bit blocks m

Multiplying Big Integers X = 0000 … Break into m = O(log N) bit blocks Pad with zero 2N bits to hold product n blocks where n is a power of 2, ie n=2 r. Let p be a prime log p ≥ block size = m p-1 is divisible by n, so Z mod p has n n th roots of unity. View each block as a finite field element in Z mod p. (no actual work) n=2 r blocks O(log p) a n … a 7 a 6 a 5 a 4 a 3 a 2 a 1 a 0 m

Multiplying Big Integers X = 0000 … View as coefficients of a polynomial. Note X = f(2 m ). Same for Y = g(2 m ). Multiply g×f using FFT in time O(nlogn). Note X×Y = [g×f](2 m ). Evaluate [g×f](2 m ) in time O(n) operations, but each op could be on O(n) bit numbers for a total of O(n 2 ) time. f(x) = a n-1 x n-1 + … + a 5 x 5 + a 4 x 4 + a 3 x 3 + a 2 x 2 + a 1 x + a 0 m g(x) = b n-1 x n-1 + … + b 5 x 5 + b 4 x 4 + b 3 x 3 + b 2 x 2 + b 1 x + b 0

Multiplying Big Integers X×Y = Evaluate [g×f](2 m ) in time O(n). [g×f](x) = c n-1 x n-1 + … + c 5 x 5 + c 4 x 4 + c 3 x 3 + c 2 x 2 + c 1 x + c 0 m O(log p) Some texts say the c i can just be shifted and joined. Problem: The field elements may be too big.

Multiplying Big Integers Evaluate [g×f](2 m ) in time O(n). [g×f](x) = c n x n + … + c 5 x 5 + c 4 x 4 + c 3 x 3 + c 2 x 2 + c 1 x + c 0 m O(log p) Shift each c i by im. Add X×Y = Adding n numbers each n bits long takes O(n 2 ) but here the numbers are sparse.

Multiplying Big Integers Evaluate [g×f](2 m ) in time O(n). [g×f](x) = c n x n + … + c 5 x 5 + c 4 x 4 + c 3 x 3 + c 2 x 2 + c 1 x + c 0 m O(log p) Shift each c i by im. Add X×Y = At each point, at most two numbers overlap  Carry is at most one  O(N) bit operations.

Multiplying Big Integers X = 11… (N bits) Y = 10… X×Y = 10… Suppose N is really really big. How many bit operations are needed? O(N logN) O(N logN loglogN) O(N logN loglogN logloglogN loglogloglogN …) FFT time Time stated in text Time as far as I can see

Multiplying Big Integers X = … … Input size = N bits Field element size = N’ = log(N) bits # a i = n = N / N’ # of field ops = O(nlogn) Time for × field op = ? N’ X’ = Input size = N’ bits Field element size = N’’ = log(N’) bits # a i = n’ = N’ / N’’ # of field ops = O(n’logn’) Time for × field op = ? Total time: N’’ And so on … O(N’ logN’ loglogN’ logloglogN’ …)

Multiplying Big Integers X = … … Input size = N bits Field element size = N’ = log(N) bits # a i = n = N / N’ # of field ops = O(nlogn) Time for × field op = ? Total time: = O( n logn ) × O(N’ logN’ loglogN’ logloglogN’ …) = O( N / N’ log N / N’ ) × O(N’ loglogN logloglogN loglogloglogN …) = O( N logN loglogN logloglogN loglogloglogN …) N’ O(N’ logN’ loglogN’ logloglogN’ …)

The End