1. ESE 250: Digital Audio Basics Week 4 February 5, 2013 The Frequency Domain 1ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

2. Course Map 2ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

3. Musical Representation With this compact notation – Could communicate a sound to pianist – Much more compact than 44KHz time-sample amplitudes (fewer bits to represent) – Represent frequencies 3ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

4. Physics review Consider a sound signal that varies as a sine wave in time: What is the relationship between and ? – rad/s, Hz 4ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

5. Time-domain Digital representation Week2: Quantization in TIME and VALUE Question: Other ways to represent this signal? 5ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

6. Another-domain representation? Consider a sound signal that varies as a sine wave in time: Note that a sine wave is periodic, this implies redundancy that we could exploit to better compress it (week3 – compression) At least we could save memory by representing only one period. But can we do even better? Question: What information do we need to represent the signal? – Amplitude, frequency, phase Question: How much memory to store each of these? 6ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

7. Another-domain representation? Remember week 2 lab: Inputs: – Amplitude: 1 – Frequency: various values, 220Hz, etc… – Phase: 0 – Sampling rate: 48,000 kHz Output: Periodic sine function sampled in time, with 48,000 samples for 1 second of sound Question: Why did we need to create the large LVM files? Why not just store the little information we started with? 7ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

8. Storage advantage Assume each of amplitude, frequency and phase are stored as 16bit integers. How much memory to store the sine in lab2? – Only storing the info above? – With an LVM file containing 48,000 samples? Percent savings? 8ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

9. Frequency-domain T = π, A = 3: s(t) = A*sin(2π*f*t) = 3*sin(2*t) 9ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

10. Frequency-domain Of course, not all signals are this simple For example, consider s(t) = sin(2t) + sin(t)/2 Question: What will the frequency representation look like? 10ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

11. Frequency-domain How about the time-domain? – Plot sin(2t) – Plot sin(t)/2 – Sum: sin(2t)+ sin(t)/2 Notice how it was easier to plot the frequency domain representation 11

12. Frequency-domain Another example What function is this? 12ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

13. Frequency-domain So far, we have seen how a signal written as a sum of sines of different frequencies can have a frequency domain representation. Each sine component is more or less important depending on its coefficient But can any arbitrary signal be represented as a sum of sines? – No. But the idea has potential, let’s explore it! 13ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

14. More background Fourier series: – Any periodic signal can be represented as a sum of simple periodic functions: sin and cos sin(nt) and cos(nt) For n = 1, 2, 3, … These are called the harmonics of the signal 14 Jean Baptiste Joseph Fourier, wikipedia ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

15. Fourier Series – more formally Given a (well-behaved) periodic function f(t), we can always write it as: Where: These two equations give us the and coefficients of the Fourier series based on the input function. These coefficients form the frequency domain representation We will not prove the equations If you’re interested: Office Hours, ESE 325, Math 312, … 15ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

16. Fourier Series – Why does it work? The cos(nx) and sin(nx) functions form an orthogonal basis: they allow us to represent any periodic signal by taking a linear combination of the basis components without interfering with one another 16

17. Orthogonal basis Choose a basis Order the vectors in it All vectors in the space can be represented as sequences of coordinates, or ordered coefficients of the basis vectors Example: 3D space – Basis: [i j k] – Linear combination: xi + yj + zk – Coordinate representation: [x y z] Example: Quadratic polynomials – Basis: [x 2 x 1] – Linear combination: ax 2 + bx + c – Coordinate representation: [a b c] 17ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

18. Fourier Series Consider the xy-plane: We can address a point by its two coordinates (x,y) The blue point is located at (10, 3) In Fourier series, we have a basis with infinite dimension, so we sum up over an infinite number of harmonics Harmonics are 1, cos(t), cos(2t), cos(3t), …, sin(t), sin(2t), … 18 0 5 10 0 5 10 ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

19. Example of another orthogonal basis Although we have an infinite number of basis elements, we don’t need to use all of them The components with the largest coefficients are the most significant The more components we add, the closer to the function we get. i.e.: As, Examples follow 19ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

20. Fourier Series – Sawtooth wave 20 (falstad.com/fourier/)

21. ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah Fourier Series – Sawtooth wave 21 (falstad.com/fourier/)

22. Interlude 22ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

23. 23 Where Are We Heading After Today? Week 2 – Received signal is discrete-time-stamped quantized – q = PCM[ r ] = quant L [Sample T s [r] ] Week 3 – Quantized Signal is Coded – c =code[ q ] Week 4 – Sampled signal not coded directly First linearly transformed into frequency domain – Q = DFT[ q ] [Painter & Spanias. Proc.IEEE, 88(4):451–512, 2000] q SampleCode Store/ Transmit DecodeProduce r(t)r(t)p(t)p(t) Generic Digital Signal Processor q c c Q Psychoacoustic Audio Coder ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

24. What now? In the first half of the lecture we have introduced the idea of frequency domain In the second half of the lecture, we will see how to extend the idea of Fourier Series to the class of signals we are interested in: Not necessarily periodic and quantized 24ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

25. Frequency-domain Fourier Series deal with periodic signals Are all signals periodic? Is your favorite mp3 song periodic? What do we do with these? 25ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

26. Frequency-domain Certainly, we could loop a song and make the signal periodic, but this is not a good solution Mathematicians have extended the notion of Fourier Series into that of Fourier Transforms by making the period of the signal go to infinity: 26ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

27. Discrete Fourier Transforms Fourier Transforms are nice, but we want to store and process our signals with computers We therefore extend Fourier Transforms into Discrete Fourier Transforms, or DFT The signal is now discrete: The signal contains N samples: Remember Euler’s formula: 27ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah The frequency domain signal also has N-1 elements: The time domain signal can be reconstructed from

28. Discrete Fourier Transforms A DFT transforms N samples of a signal in time domain into a (periodic) frequency representation with N samples So we don’t have to deal with real signals anymore We work with sampled signals (quantized in time), and the frequency representation we get is also quantized in time! 28ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah (sepwww.stanford.edu/oldsep/hale/FftLab.html)

29. Discrete Fourier Transforms A smaller sampling period -> More points to represent the signal larger N -> More harmonics used in DFT N harmonics -> Smaller error compared to actual analog signal we capture/produce DFTs are extensively used in practice, since computers can handle them 29ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

30. Discrete Fourier Transform exercise Complete the table below using the DFT equation shown above 30ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah (The bottom is redundant because the DFT of a real signal is symmetrical) 7 samples -> 7 harmonics

31. 31 Approximate Reconstruction – Always achievable A signal sampled in time can be approximated arbitrarily closely from the time-sampled values With a DFT, each sample gives us knowledge of one harmonic Each harmonic is a component used in the reconstruction of the signal The more harmonics we use, the better the reconstruction Approximating the Sampled Signal { Cos[0t], Sin[1t], Cos[1t], Sin[2t], Cos[2t], Sin[3t], Cos[3t] } ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

32. 32 Usually Computed, Not “Solved” 7 Samples; 7 Harmonics 11 Samples 15 Samples; 15 Harmonics ; 11 Harmonics DFT ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

33. ESE 250 – Spring’10 DeHon & Koditschek33 Yet Another Sampled (Real) Signal t v Measured Data: Sampled Signal:

34. 34 Approximate Reconstruction – although always achievable – may require a lot of samples – to get good performance Sometimes time is better than frequency Some Signals Dislike Some Harmonics 15 Samples & Harmonics 21 Samples & Harmonics 31 Samples & Harmonics ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

35. Saving resources However, N can get very large e.g., with a sampling rate of 48,000Hz How big is N for a 4min song? How many operations does this translate to? – To compute one frequency component? – To compute all N frequency components? This is not practical. Instead, we use a window of values to which we apply the transform – Typical size: …, 512, 1024, 2048, … 35ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

36. A Window Operation In the example below, we traverse the signal but only look at 64 samples at a time 36ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah Time Freq (sepwww.stanford.edu/oldsep/hale/FftLab.html)

37. A Window Operation The following shows a DFT (on the right) applied to a sinusoidal signal (on the left). The DFT is only applied to part of the signal, the window, which is shown in the middle When we increase the window size, the information of the DFT is more accurate 37ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah 8-point DFT (http://math.mit.edu/mathlets/mathlets/discrete-fourier-transform/) 16-point DFT32-point DFT

38. Moonlight Sonata in Time and Frequency 38ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

39. Big ideas A signal in time-domain can also be represented in the frequency domain. – X(f) = Transform(x(t)) This representation may be more efficient: – For storage – For signal processing – For understanding the wave (e.g. can cut-off some frequencies) We go from a signal represented by values in time to one represented by coefficients of harmonics Fourier Series, for continuous periodic signals FT: Fourier Transform, for continuous signals DFT: Discrete Fourier Transform, for discrete signals – DFTs are extensively used in practice – Finite amount of data, easy to implement for computers 39ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

40. Admin Lab 3 due Wednesday (tomorrow) On Thursday, Lab 4: – You will be looking at signals in the frequency domain in LabView – You will create your own synthesizer! 40ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah

41. Announcement Hackathon and bootcamp – If you are interested The bootcamp will be this Saturday, noon to six, in Detkin lab. The Hackathon will begin around 5-6 the weekend of the 15-17th 41ESE 250 - Spring'13 DeHon, Kod, Kadric, Wilson-Shah