1. Princeton University COS 423 Theory of Algorithms Spring 2002 Kevin Wayne Fast Fourier Transform Jean Baptiste Joseph Fourier (1768-1830) These lecture slides are adapted from CLRS.

2. 2 Fast Fourier Transform Applications. n Perhaps single algorithmic discovery that has had the greatest practical impact in history. n Optics, acoustics, quantum physics, telecommunications, systems theory, signal processing, speech recognition, data compression. n Progress in these areas limited by lack of fast algorithms. History. n Cooley-Tukey (1965) revolutionized all of these areas. n Danielson-Lanczos (1942) efficient algorithm. n Runge-König (1924) laid theoretical groundwork. n Gauss (1805, 1866) describes similar algorithm. n Importance not realized until advent of digital computers.

3. 3 Polynomials: Coefficient Representation Degree n polynomial. Addition: O(n) ops. Evaluation: O(n) using Horner's method. Multiplication (convolution): O(n 2 ).

4. 4 Polynomials: Point-Value Representation Degree n polynomial. n Uniquely specified by knowing p(x) at n different values of x. x y = p(x) xjxj yjyj

5. 5 Polynomials: Point-Value Representation Degree n polynomial. Addition: O(n). Multiplication: O(n), but need 2n points. Evaluation: O(n 2 ) using Lagrange's formula.

6. 6 Best of Both Worlds Can we get "fast" multiplication and evaluation?  Yes! Convert back and forth between two representations. coefficient Representation O(n 2 ) Multiplication O(n) Evaluation point-valueO(n)O(n 2 ) FFTO(n log n) O(n) point-value multiplication O(n 2 ) coefficient multiplication O(n log n) evaluation FFT interpolation inverse FFT O(n log n)

7. 7 Converting Between Representations: Naïve Solution Evaluation (coefficient to point-value). n Given a polynomial p(x) =a 0 + a 2 x 1 +... + a n-1 x n-1, choose n distinct points {x 0, x 1,..., x n-1 } and compute y k = p(x k ), for each k using Horner's method. n O(n 2 ). Interpolation (point-value to coefficient). n Given n distinct points {x 0, x 1,..., x n-1 } and y k = p(x k ), compute the coefficients {a 0, a 1,..., a n-1 } by solving the following linear system of equations. n Note Vandermonde matrix is invertible iff x k are distinct. n O(n 3 ).

8. 8 Fast Interpolation: Key Idea Key idea: choose {x 0, x 1,..., x n-1 } to make computation easier! n Set x k = x j ?

9. 9 Fast Interpolation: Key Idea Key idea: choose {x 0, x 1,..., x n-1 } to make computation easier! n Set x k = x j ? n Use negative numbers: set x k = -x j so that (x k ) 2 = (x j ) 2. – set x k = -x n/2 + k – E = p even (x 2 ) = a 0 + a 2 17 2 + a 4 17 4 + a 6 17 6 +... + a n-2 17 n-2 – O = x p odd (x 2 ) = a 1 17 + a 3 17 3 + a 5 17 5 + a 7 17 7 +... + a n-1 17 n - 1 – y 0 = E + O, y n/2 = E - O

10. 10 Fast Interpolation: Key Idea Key idea: choose {x 0, x 1,..., x n-1 } to make computation easier! n Set x k = x j ? n Use negative numbers: set x k = -x j so that (x k ) 2 = (x j ) 2. – set x k = -x n/2 + k n Use complex numbers: set x k =  k where  is n th root of unity. – (x k ) 2 = (-x n/2 + k ) 2 – (x k ) 4 = (-x n/4 + k ) 4 – (x k ) 8 = (-x n/8 + k ) 8

11. 11 Roots of Unity An n th root of unity is a complex number z such that z n = 1. n  = e 2  i / n = principal n th root of unity. n e i t = cos t + i sin t. n i 2 = -1. n There are exactly n roots of unity:  k, k = 0, 1,..., n-1.  0 = 1 11  2 = i 33  4 = -1 55  6 = -i 77

12. 12 Roots of Unity: Properties L1: Let  be the principal n th root of unity. If n > 0, then  n/2 = -1. n Proof:  = e 2  i / n   n/2 = e  i = -1. (Euler's formula) L2: Let n > 0 be even, and let  and be the principal n th and (n/2) th roots of unity. Then (  k ) 2 = k. n Proof: (  k ) 2 = e (2k)2  i / n = e (k) 2  i / (n / 2) = k. L3: Let n > 0 be even. Then, the squares of the n complex n th roots of unity are the n/2 complex (n/2) th roots of unity. n Proof: If we square all of the n th roots of unity, then each (n/2) th root is obtained exactly twice since: – L1   k + n / 2 = -  k – thus, (  k + n / 2 ) 2 = (  k ) 2 – L2  both of these = k –  k + n / 2 and  k have the same square

13. 13 Divide-and-Conquer Given degree n polynomial p(x) = a 0 + a 1 x 1 + a 2 x 2 +... + a n-1 x n-1. n Assume n is a power of 2, and let  be the principal n th root of unity. n Define even and odd polynomials: – p even (x) := a 0 + a 2 x 1 + a 4 x 2 + a 6 x 3 +... + a n-2 x n/2 - 1 – p odd (x) := a 1 + a 3 x 1 + a 5 x 2 + a 7 x 3 +... + a n-1 x n/2 - 1 – p(x) = p even (x 2 ) + x p odd (x 2 ) n Reduces problem of – evaluating degree n polynomial p(x) at  0,  1,...,  n-1 to – evaluating two degree n/2 polynomials at: (  0 ) 2, (  1 ) 2,..., (  n-1 ) 2. n L3  p even (x) and p odd (x) only evaluated at n/2 complex (n/2) th roots of unity.

14. 14 FFT Algorithm if (n == 1) // n is a power of 2 return a 0   e 2  i / n (e 0,e 1,e 2,...,e n/2-1 )  FFT(n/2, a 0,a 2,a 4,...,a n-2 ) (d 0,d 1,d 2,...,d n/2-1 )  FFT(n/2, a 1,a 3,a 5,...,a n-1 ) for k = 0 to n/2 - 1 y k  e k +  k d k y k+n/2  e k -  k d k return (y 0,y 1,y 2,...,y n-1 ) FFT (n, a 0, a 1, a 2,..., a n-1 ) O(n) complex multiplies if we pre-compute  k.

15. 15 Recursion Tree a 0, a 1, a 2, a 3, a 4, a 5, a 6, a 7 a 1, a 3, a 5, a 7 a 0, a 2, a 4, a 6 a 3, a 7 a 1, a 5 a 0, a 4 a 2, a 6 a0a0 a4a4 a2a2 a6a6 a1a1 a5a5 a3a3 a7a7 "bit-reversed" order

16. 16 Proof of Correctness Proof of correctness. Need to show y k = p(  k ) for each k = 0,..., n-1, where  is the principal n th root of unity. n Base case. n = 1   = 1. Algorithm returns y 0 = a 0 = a 0  0. n Induction step. Assume algorithm correct for n / 2. – let be the principal (n/2) th root of unity – e k = p even ( k ) = p even (  2k ) by Lemma 2 – d k = p odd ( k ) = p odd (  2k ) by Lemma 2 – recall p(x) = p even (x 2 ) + x p odd (x 2 )

17. 17 Best of Both Worlds Can we get "fast" multiplication and evaluation?  Yes! Convert back and forth between two representations. coefficient Representation O(n 2 ) Multiplication O(n) Evaluation point-valueO(n)O(n 2 ) FFTO(n log n) O(n) point-value multiplication O(n 2 ) coefficient multiplication O(n log n) evaluation FFT interpolation inverse FFT O(n log n)

18. 18 Forward FFT: given {a 0, a 1,..., a n-1 }, compute {y 0, y 1,..., y n-1 }. Inverse FFT: given {y 0, y 1,..., y n-1 } compute {a 0, a 1,..., a n-1 }. Inverse FFT

19. 19 Great news: same algorithm as FFT, except use  -1 as "principal" n th root of unity (and divide by n). Inverse FFT

20. 20 Inverse FFT: Proof of Correctness Summation lemma. Let  be a primitive n th root of unity. Then n If k is a multiple of n then  k = 1. n Each n th root of unity  k is a root of x n - 1 = (x - 1) (1 + x + x 2 +... + x n-1 ), if  k  1 we have: 1 +  k +  k(2) +... +  k(n-1) = 0. Claim: F n and F n -1 are inverses.

21. 21 Inverse FFT: Algorithm if (n == 1) // n is a power of 2 return a 0   e -2  i/n (e 0,e 1,e 2,...,e n/2-1 )  FFT(n/2, a 0,a 2,a 4,...,a n-2 ) (d 0,d 1,d 2,...,d n/2-1 )  FFT(n/2, a 1,a 3,a 5,...,a n-1 ) for k = 0 to n/2 - 1 y k  (e k +  k d k ) / n y k+n/2  (e k -  k d k ) / n return (y 0,y 1,y 2,...,y n-1 ) IFFT (n, a 0, a 1, a 2,..., a n-1 )

22. 22 Best of Both Worlds Can we get "fast" multiplication and evaluation?  Yes! Convert back and forth between two representations. coefficient Representation O(n 2 ) Multiplication O(n) Evaluation point-valueO(n)O(n 2 ) FFTO(n log n) O(n) point-value multiplication O(n 2 ) coefficient multiplication O(n log n)evaluation FFT interpolation inverse FFT O(n log n)

23. 23 Integer Arithmetic Multiply two n-digit integers: a = a n-1... a 1 a 0 and b = b n-1... b 1 b 0. n Form two degree n polynomials. n Note: a = p(10), b = q(10). n Compute product using FFT in O(n log n) steps. n Evaluate r(10) = a  b. n Problem: O(n log n) complex arithmetic steps. Solution. n Strassen (1968): carry out arithmetic to suitable precision. – T(n) = O(n T(log n))  T(n) = O(n log n (log log n) 1+  ) n Schönhage-Strassen (1971): use modular arithmetic. – T(n) = O(n log n log log n)

24. Princeton University COS 423 Theory of Algorithms Spring 2002 Kevin Wayne Extra Slides

25. 25 Polynomials: Representation and Addition Degree n polynomial. n Coefficient representation. n Point-value representation. Addition. n O(n) for either representation.

26. 26 Polynomials: Evaluation and Multiplication Multiplication (convolution). n Brute force. O(n 2 ). n Use 2n points. O(n). Evaluation. n Horner's method. O(n). n Lagrange's formula. O(n 2 ).