ESE 250: Digital Audio Basics Week 4 February 5, 2013 The Frequency Domain 1ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Course Map 2ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Musical Representation With this compact notation – Could communicate a sound to pianist – Much more compact than 44KHz time-sample amplitudes (fewer bits to represent) – Represent frequencies 3ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Physics review Consider a sound signal that varies as a sine wave in time: What is the relationship between and ? – rad/s, Hz 4ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Time-domain Digital representation Week2: Quantization in TIME and VALUE Question: Other ways to represent this signal? 5ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Another-domain representation? Consider a sound signal that varies as a sine wave in time: Note that a sine wave is periodic, this implies redundancy that we could exploit to better compress it (week3 – compression) At least we could save memory by representing only one period. But can we do even better? Question: What information do we need to represent the signal? – Amplitude, frequency, phase Question: How much memory to store each of these? 6ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Another-domain representation? Remember week 2 lab: Inputs: – Amplitude: 1 – Frequency: various values, 220Hz, etc… – Phase: 0 – Sampling rate: 48,000 kHz Output: Periodic sine function sampled in time, with 48,000 samples for 1 second of sound Question: Why did we need to create the large LVM files? Why not just store the little information we started with? 7ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Storage advantage Assume each of amplitude, frequency and phase are stored as 16bit integers. How much memory to store the sine in lab2? – Only storing the info above? – With an LVM file containing 48,000 samples? Percent savings? 8ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Frequency-domain T = π, A = 3: s(t) = A*sin(2π*f*t) = 3*sin(2*t) 9ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Frequency-domain Of course, not all signals are this simple For example, consider s(t) = sin(2t) + sin(t)/2 Question: What will the frequency representation look like? 10ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Frequency-domain How about the time-domain? – Plot sin(2t) – Plot sin(t)/2 – Sum: sin(2t)+ sin(t)/2 Notice how it was easier to plot the frequency domain representation 11

Frequency-domain Another example What function is this? 12ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Frequency-domain So far, we have seen how a signal written as a sum of sines of different frequencies can have a frequency domain representation. Each sine component is more or less important depending on its coefficient But can any arbitrary signal be represented as a sum of sines? – No. But the idea has potential, let’s explore it! 13ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

More background Fourier series: – Any periodic signal can be represented as a sum of simple periodic functions: sin and cos sin(nt) and cos(nt) For n = 1, 2, 3, … These are called the harmonics of the signal 14 Jean Baptiste Joseph Fourier, wikipedia ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Fourier Series – more formally Given a (well-behaved) periodic function f(t), we can always write it as: Where: These two equations give us the and coefficients of the Fourier series based on the input function. These coefficients form the frequency domain representation We will not prove the equations If you’re interested: Office Hours, ESE 325, Math 312, … 15ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Fourier Series – Why does it work? The cos(nx) and sin(nx) functions form an orthogonal basis: they allow us to represent any periodic signal by taking a linear combination of the basis components without interfering with one another 16

Orthogonal basis Choose a basis Order the vectors in it All vectors in the space can be represented as sequences of coordinates, or ordered coefficients of the basis vectors Example: 3D space – Basis: [i j k] – Linear combination: xi + yj + zk – Coordinate representation: [x y z] Example: Quadratic polynomials – Basis: [x 2 x 1] – Linear combination: ax 2 + bx + c – Coordinate representation: [a b c] 17ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Fourier Series Consider the xy-plane: We can address a point by its two coordinates (x,y) The blue point is located at (10, 3) In Fourier series, we have a basis with infinite dimension, so we sum up over an infinite number of harmonics Harmonics are 1, cos(t), cos(2t), cos(3t), …, sin(t), sin(2t), … ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Example of another orthogonal basis Although we have an infinite number of basis elements, we don’t need to use all of them The components with the largest coefficients are the most significant The more components we add, the closer to the function we get. i.e.: As, Examples follow 19ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Fourier Series – Sawtooth wave 20 (falstad.com/fourier/)

ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah Fourier Series – Sawtooth wave 21 (falstad.com/fourier/)

Interlude 22ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

23 Where Are We Heading After Today? Week 2 – Received signal is discrete-time-stamped quantized – q = PCM[ r ] = quant L [Sample T s [r] ] Week 3 – Quantized Signal is Coded – c =code[ q ] Week 4 – Sampled signal not coded directly First linearly transformed into frequency domain – Q = DFT[ q ] [Painter & Spanias. Proc.IEEE, 88(4):451–512, 2000] q SampleCode Store/ Transmit DecodeProduce r(t)r(t)p(t)p(t) Generic Digital Signal Processor q c c Q Psychoacoustic Audio Coder ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

What now? In the first half of the lecture we have introduced the idea of frequency domain In the second half of the lecture, we will see how to extend the idea of Fourier Series to the class of signals we are interested in: Not necessarily periodic and quantized 24ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Frequency-domain Fourier Series deal with periodic signals Are all signals periodic? Is your favorite mp3 song periodic? What do we do with these? 25ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Frequency-domain Certainly, we could loop a song and make the signal periodic, but this is not a good solution Mathematicians have extended the notion of Fourier Series into that of Fourier Transforms by making the period of the signal go to infinity: 26ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Discrete Fourier Transforms Fourier Transforms are nice, but we want to store and process our signals with computers We therefore extend Fourier Transforms into Discrete Fourier Transforms, or DFT The signal is now discrete: The signal contains N samples: Remember Euler’s formula: 27ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah The frequency domain signal also has N-1 elements: The time domain signal can be reconstructed from

Discrete Fourier Transforms A DFT transforms N samples of a signal in time domain into a (periodic) frequency representation with N samples So we don’t have to deal with real signals anymore We work with sampled signals (quantized in time), and the frequency representation we get is also quantized in time! 28ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah (sepwww.stanford.edu/oldsep/hale/FftLab.html)

Discrete Fourier Transforms A smaller sampling period -> More points to represent the signal larger N -> More harmonics used in DFT N harmonics -> Smaller error compared to actual analog signal we capture/produce DFTs are extensively used in practice, since computers can handle them 29ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Discrete Fourier Transform exercise Complete the table below using the DFT equation shown above 30ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah (The bottom is redundant because the DFT of a real signal is symmetrical) 7 samples -> 7 harmonics

31 Approximate Reconstruction – Always achievable A signal sampled in time can be approximated arbitrarily closely from the time-sampled values With a DFT, each sample gives us knowledge of one harmonic Each harmonic is a component used in the reconstruction of the signal The more harmonics we use, the better the reconstruction Approximating the Sampled Signal { Cos[0t], Sin[1t], Cos[1t], Sin[2t], Cos[2t], Sin[3t], Cos[3t] } ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

32 Usually Computed, Not “Solved” 7 Samples; 7 Harmonics 11 Samples 15 Samples; 15 Harmonics ; 11 Harmonics DFT ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

ESE 250 – Spring’10 DeHon & Koditschek33 Yet Another Sampled (Real) Signal t v Measured Data: Sampled Signal:

34 Approximate Reconstruction – although always achievable – may require a lot of samples – to get good performance Sometimes time is better than frequency Some Signals Dislike Some Harmonics 15 Samples & Harmonics 21 Samples & Harmonics 31 Samples & Harmonics ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Saving resources However, N can get very large e.g., with a sampling rate of 48,000Hz How big is N for a 4min song? How many operations does this translate to? – To compute one frequency component? – To compute all N frequency components? This is not practical. Instead, we use a window of values to which we apply the transform – Typical size: …, 512, 1024, 2048, … 35ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

A Window Operation In the example below, we traverse the signal but only look at 64 samples at a time 36ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah Time Freq (sepwww.stanford.edu/oldsep/hale/FftLab.html)

A Window Operation The following shows a DFT (on the right) applied to a sinusoidal signal (on the left). The DFT is only applied to part of the signal, the window, which is shown in the middle When we increase the window size, the information of the DFT is more accurate 37ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah 8-point DFT ( 16-point DFT32-point DFT

Moonlight Sonata in Time and Frequency 38ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Big ideas A signal in time-domain can also be represented in the frequency domain. – X(f) = Transform(x(t)) This representation may be more efficient: – For storage – For signal processing – For understanding the wave (e.g. can cut-off some frequencies) We go from a signal represented by values in time to one represented by coefficients of harmonics Fourier Series, for continuous periodic signals FT: Fourier Transform, for continuous signals DFT: Discrete Fourier Transform, for discrete signals – DFTs are extensively used in practice – Finite amount of data, easy to implement for computers 39ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Admin Lab 3 due Wednesday (tomorrow) On Thursday, Lab 4: – You will be looking at signals in the frequency domain in LabView – You will create your own synthesizer! 40ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah

Announcement Hackathon and bootcamp – If you are interested The bootcamp will be this Saturday, noon to six, in Detkin lab. The Hackathon will begin around 5-6 the weekend of the 15-17th 41ESE Spring'13 DeHon, Kod, Kadric, Wilson-Shah