2. Sound Synthesis Part IV: Subtractive & Granular synthesis, Physical modelling

3. Plan Previously... Subtractive synthesis Granular Synthesis Analysis Physical modelling Summary

4. Previously,... Sound is defined by: –Loudness (amplitude + high freq. Comp.) –Pitch (fundamental frequency) –Timbre (all other freq. Comp + time envelope) Visible in a spectrogram:

5. Types of synthesis Sound Synthesis Additive synthesis Distortion techniques Subtractive synthesis Granular synthesis Analysis based Physical modelling

6. Simple Instrument Helmholtz model –Waveform –Constant frequency –Envelope Envelope feeds varying amplitude to the oscillator. ASD Envelope AMP FREQ PHASE AMP ATTACK DURATION DECAY

7. Additive Synthesis FREQ +

8. Pros & cons Advantages –Can reproduce a spectrogram bin by bin. –Theoretically, can generate any sound –High quality synthesis can be achieved Limitations –Requires large amount of data for generation –Sound analysis is only valid for a short range of pitch and loudness! –Not intuitive to control

9. Types of synthesis Sound Synthesis Additive synthesis Distortion techniques Subtractive synthesis Granular synthesis Analysis based Physical modelling

10. Distortion (aka nonlinear) Synthesis Additive synthesis is powerful & flexible BUT requires many simple oscillators. Plethoric and non-intuitive controls on the sound spectrum.  Using fewer oscillators, with more components.  Complex spectra can be obtained by distorting simple oscillators.

11. FM Synthesis: Instrument Design Envelopes functions F1 & F2 change the timbre of the sound (John Chowning, 1973) A) Bell-like timbre B) Wood drum sound C) Brass-like timbre fm + fc A fo F1 F2 1/d fm*I 1/d (John Chowning, 1973)

12. Pros & cons Advantages –Modulation index is a simple & effective way to control timbre. –Computationally very efficient. –Very easy to generate inharmonic spectra. Limitations –Difficult to synthesise natural sounds.

13. Waveshaping: Instrument Design Instrument for clarinet-like timbre From Jean-Claude Risset’s Introductory Catalogue of Computer Synthesized Sounds. fm + ASD Envelope 256 F(x) * A 255 0.64 0.085

14. Pros & cons Advantages –Distortion index: efficient way to control timbre –Computationally very efficient –Possible to compute transfer characteristic analytically. Limitations: –Difficult to synthesise natural sounds.

15. Types of synthesis Sound Synthesis Additive synthesis Distortion techniques Subtractive synthesis Granular synthesis Analysis based Physical modelling

16. Subtractive Synthesis Idea: “Start with a source with a broad spectrum (noise, pulse) and filter out the unwanted portions of the spectrum.” Main sources: –Noise –Pulse generator SOURCE FILTER

17. Sources: Pulse waveform Significant amplitude only for a short time (pulse width) Repeated periodically  rich spectrum Spectrum depends on shape and width –Narrow pulse  high frequencies. Square, triangular, etc. But, band-limited is preferable!

18. Source: Pulse Waveform (cont’d) Common band-limited pulse (BUZZ): –Shape depends on ration f0/fs –Harmonics of fundamental only up to Nyquist frequency (to avoid aliasing) –Number of harmonics N = int(fs/2f0) Examples: –f0=440Hz, fs=40kHz  N = 45 –F0=1046Hz, fs=40kHz  N = 19 N FREQ AMP BUZZ

19. Pulse Waveform (cont’d)

20. Filters Chosen to carve out or reduce frequency components. Two zones: –Pass Band: components are preserved –Stop Band: components are reduced/removed –And usually a smooth transition between them Examples: –Low pass filters will preserve low frequencies –High pass will preserve high frequencies (change fundamental!). –Band pass select a frequency band...

21. Filters: low & high pass

22. Filters (cont’d) Definition: “The cut-off frequency of a filter is where the power transmitted drops by -3dB.” Band-pass filters are defined by –A center frequency: fc –A Bandwidth: BW OR –Lower and upper cut-off frequencies: fl,fu –The sharpness is called a Quality Factor: Q=f0/BW The opposite of a Band-pass is a Band-reject filter.

23. Filters: Band-pass

24. Timbre: Envelope [flashback] Arrows indicate formants. This slide indicates two speech vowels (i and u) Formants not only determine timbre but helps distinguishing vowels. (used in speech recognition)

25. Pros & cons Advantages: –Filtering allows to control formants naturally. –Computationally efficient –Can generate natural sounds (but requires skillful design!) Limitations –Difficult to produce natural sounds! –Instability and noise (when using analog implementations)

26. Types of synthesis Sound Synthesis Additive synthesis Distortion techniques Subtractive synthesis Granular synthesis Analysis based Physical modelling

27. Granular Synthesis Question: “How can we decompose a sound into a number of micro-sounds, and then use them as fundamental building blocks in sound synthesis?” Assumption: “Every sound can be represented as a combination of audio grains (micro-sounds) distributed in time”  granular synthesis (Micro-sounds = grains = audio particles)

28. How does it work? According to Dodge & Jersey (1997) “Grains are the fundamental compositional elements to weave the sound” Grains are defined as small bursts (typically 5-100 ms.) of sound energy encased in an envelope.

29. Two categories GRANULAR SYNTHESIS SYNCHRONO US ASYNCHRONO US Grains triggered at timed interval (central clock)  Sound with pitch Grains triggered at random intervals  No pitch, perceived as “cloud”.

30. A grain of Sound... A grain can be generated using a simple Helmholtz instrument.  Very narrow time envelope - small d! f0 A 1/d ENVELOPE OSCILLATOR

31. Synchronous Grain Synthesis In synchronous mode, grains are triggered at constant time intervals (with constant triggering frequency). Synthetic sounds can be characterized by a pitch sensation. The pitch corresponds to the triggering frequency not the frequency of the wave inside the grain!

32. Synchronous a) Classical additive b) Granular synchronous

33. Synchronous Grain Synthesis The spectral envelope of the waveform inside the grain determines the spectral envelope of the sound. Example: Repeating at 220Hz rate (A3) a grain enclosing a 3kHz sine wave yields a spectrum containing the harmonics of 220Hz –Harmonics nearest 3kHz will receive most emphasis –Increasing the rate to 261.6Hz (C4) (or any other frequency) changes the spacing between the harmonics, but the peak of the spectral envelope still appears at 3kHz.  Synchronous Granular Synthesis imparts a fixed formant to a sound.

34. Asynchronous Grain Synthesis In Asynchronous mode, grains are triggered with a random frequency, yielding a noise-like sound without any pitch. Sounds can be described as “clouds”, “audio textures”, audio “wall papers” All about timbre...

35. Asynchronous a)Granular harmonic cloud (additive)b) Granular textural cloud

36. Clouds are not all alike...

37. Sound clouds (cont’d) t f t f A) cumulusB) stratus

38. Properties of a sound cloud Sound clouds can be described by: –Start time and duration –Bandwidth of the cloud –Amplitude envelope of the cloud –Density of grains –Grain envelope/duration –Grain waveform –Spatial dispersion

39. Examples of sound clouds t f t f B) glissando t f B) stratus t f t f A) constant B) variableC) cumulus

40. Grain waveform The Grain content can be seen like a colour – it represents the timbre. –Monochromatic –Polychromatic –Transchromatic

41. Density of grains Density of grains = number of grains per second (stochastic!) –Very sparse (eg. 47) –Sparse (eg. 230) –Dense (eg. 4700) –Variable (eg. 2300) (sparse  dense)

42. Spatial Dispersion The way grains are spread in the cloud –Deterministic –Random

43. Grain Envelope The way the grains’ envelopes and duration are distributed: –Deterministic –Random This affects loudness.

44. Pros & cons Advantages –Flexible sound processing capabilities –Possible to synthesize interesting unnatural sounds (“clouds”) –Possible to implement special effects (pitch shifting, morphing) Limitations –Not always easy to control in a predictable way –Can be computationally demanding

45. Types of synthesis Sound Synthesis Additive synthesis Distortion techniques Subtractive synthesis Granular synthesis Analysis based Physical modelling

46. Analysis-based Sound Synthesis Analysis of sound properties & psychoacoustics lead to sound synthesis techniques (eg, additive synth) What about speech? –Pitch and timbre do not really cover the important properties of speech!

47. Speech ?... One way to look at it: Speech is composed of: –Phones (P): single sounds –Diphones (D): pairs of two sounds (co- articulation) Usually, for languages |D| << |P|^2 –Spanish: 800 –German: 2,500 PiPj Dij

48. Example: British vowels David Detering (1997) studied (British) vowels Plots first two formants (F1 and F2)

49. English (British vs. American)

50. Approaches to speech synthesis Concatenative: string bits of recorded speech) Unit selection: large database of recorded speech, split into phones, diphones, syllabes, etc. Indexed by pitch and chained at run-time Diphone synthesis: small database of all sound (phones) transitions. Formant synthesis: additive synthesis + acoustic model (physical modelling) Hidden Markov Models More...

51. Applications Assistance for disabled persons: –reading text for vision impaired. –Speaking for speech impaired. User interfaces Automated telephonic reception (!) Phones (SIRI?)

52. Types of synthesis Sound Synthesis Additive synthesis Distortion techniques Subtractive synthesis Granular synthesis Analysis based Physical modelling

53. Idea Classical sound synthesis: Physical modelling: ORIGINAL SOUND PARAMETERS SYNTHETIC SOUND MUSICAL INSTRUMENT MATHEMATIC AL MODEL SYNTHETIC SOUND

54. Example: Organ pipe

55. Advantages Theoretically perfect model  will produce exactly the desired sound Can be very memory efficient (no need to store wavetables) Can be used to prototype new instruments before actual production. It is possible to “build” virtually ancient instruments from documents and listen to them.

56. Limitations Requires to know in detail the physical process in the instrument. Accurate models may be computationally demanding. Equations may be difficult to solve (non- linear!) Each instrument requires a lot of work. Physical models require complicated controllers (instrument-specific)

57. Plan Previously... Subtractive synthesis Granular Synthesis Analysis Physical modelling Summary

58. Types of synthesis Sound Synthesis Additive synthesis Distortion techniques Subtractive synthesis Granular synthesis Analysis based Physical modelling

59. Additional Reading C. Dodge, C., & Jerse, T. A. (1997). Computer Music: Synthesis, Composition, and Performance. Schrimer, UK. (see chapters 6, 7 and 8) Deterding, David (1997) The formants of monophthong vowels in Standard Southern British English pronunciation. Journal of the International Phonetic Association, 27, 47-55. G. E. Peterson & H.L.Barney (1952) Control methods used in a study of the vowels. Journal of the Acoustical Society of America, 24, 175-184