Time-scale and pitch modification Algorithms review Alexey Lukin

“Time-scale and pitch modification algorithms” 2/19 The problem Goal: change duration or tonality of musical piece Goal: change duration or tonality of musical piece Naïve approach: Naïve approach: ► (analog) record on tape and change playback speed ► (digital) resample the waveform Alas: pitch and duration change synchronously! Celine DionSpeed up by 20%

“Time-scale and pitch modification algorithms” 3/19 The problem Goal: independent control of times-scale and pitch, timbre should be natural! Goal: independent control of times-scale and pitch, timbre should be natural! Applications: Applications: ► Samplers and virtual instruments ► Production: synchronization of audio and video ► Post-production: pull-up, pull-down ► Entertainment: karaoke (changing key) ► Education: sonic microscope ► More?

“Time-scale and pitch modification algorithms” 4/19 Time domain Time-domain algorithms operate with the waveform, not spectrum Time-domain algorithms operate with the waveform, not spectrum 1.Break the signal into short granules 2.Repeat or discard (or shift) some granules to change duration 3.Resample to change pitch Some pictures in this presentation are taken from Ph.D. thesis of J. Bonada

“Time-scale and pitch modification algorithms” 5/19 Time domain Time-domain algorithms operate with the waveform, not spectrum Time-domain algorithms operate with the waveform, not spectrum 1.Break the signal into short granules 2.Repeat or discard (or shift) some granules to change duration 3.Resample to change pitch Problems: Problems: ► Granules can add in-phase (good) or out-of-phase (bad) ► Transients are duplicated or discarded Guitar+castanetsSlow down to 220% length

“Time-scale and pitch modification algorithms” 6/19 Time domain Solutions: Solutions: ► Ensure that pasted granules are in phase by selecting granule size to be multiple of pitch (requires autocorrelation or pitch analysis) ► Prohibit duplicating and skipping of transient granules (requires detection of transients and advanced scheduling of granules duplication) Fixed granule size Pitch-synchronous granule size (“PSOLA”) Pitch-synchronous granule size, transients detection

“Time-scale and pitch modification algorithms” 7/19 Time domain Pitch-synchronous overlap-add (PSOLA) Pitch-synchronous overlap-add (PSOLA) ► Granules are 2 pitch periods long ► Granules are repeated or discarded ► Requires pitch detection → unstable results for non-pitched or polyphonic material

“Time-scale and pitch modification algorithms” 8/19 Time domain Summary Summary ► Very fast (1…5% CPU) ► Good quality for pitched signals (solo instruments, vocal) ► Poor quality for non-pitched and polyphonic material: Amplitude modulation (out-of-phase overlapping of granules for some parts/instruments) Amplitude modulation (out-of-phase overlapping of granules for some parts/instruments) Repeated or discarded transients (unless special care taken) Repeated or discarded transients (unless special care taken) Implementations Implementations ► Editors, samplers: Audition, Cubase, Logic, Ableton, ACID ► Vocal correctors: Melodyne, Autotune + –

“Time-scale and pitch modification algorithms” 9/19 Vocoders Frequency-domain algorithms operate with a short- time spectrum of the signal Frequency-domain algorithms operate with a short- time spectrum of the signal Idea: build a spectrogram of a signal (using a short-time Fourier transform) and re-synthesize a signal from a spectrogram with a different time stride (hop) Idea: build a spectrogram of a signal (using a short-time Fourier transform) and re-synthesize a signal from a spectrogram with a different time stride (hop) Problem: during synthesis, signal granules can overlap out-of-phase Problem: during synthesis, signal granules can overlap out-of-phase Solution: phase modification Solution: phase modification at each frequency channel called phase unwrapping

“Time-scale and pitch modification algorithms” 10/19 Vocoders Traditional vocoder algorithm: Traditional vocoder algorithm: 1.Calculate shift-time Fourier transform (STFT) of a signal 2.Unwrap phases of each frequency channel (to compensate for change of synthesis stride at step 3), don’t modify magnitudes 3.Synthesize a signal using inverse STFT with a different time stride

“Time-scale and pitch modification algorithms” 11/19 Vocoders Magnitudes do not change Magnitudes do not change Phase unwrapping equations should provide in- phase overlapping of shifted granules at each frequency channel – “horizontal phase coherence” Phase unwrapping equations should provide in- phase overlapping of shifted granules at each frequency channel – “horizontal phase coherence” (phase increment) (phase unwrapping) (synthesis phase)

“Time-scale and pitch modification algorithms” 12/19 Vocoders Phase coherence problem Phase coherence problem ► Horizontal phase coherence is ensured by phase unwrapping ► How about vertical phase coherence (coherence of phases between different frequency bins) ? It is lost! (except cases of integer stretching ratios) This leads to: “Phasiness” due to out-of-phase signals in frequency bins within every signal harmonic “Phasiness” due to out-of-phase signals in frequency bins within every signal harmonic Transients are time-smeared along the whole granule Transients are time-smeared along the whole granule Guitar+castanetsVocoder 220% length

“Time-scale and pitch modification algorithms” 13/19 Vocoders Vertical phase coherence improvement: “phase locking” algorithm locks phases within each spectrum peak Vertical phase coherence improvement: “phase locking” algorithm locks phases within each spectrum peak 1.Divide frequency spectrum into intervals of harmonics 2.Unwrap phase of central (peak) frequency channel 3.Modify phases of other bins accordingly to the phase of the central channel This reduces phasiness, but still doesn’t help transients This reduces phasiness, but still doesn’t help transients No phase lockingPhase locking

“Time-scale and pitch modification algorithms” 14/19 Vocoders How to improve sharpness of transients? How to improve sharpness of transients? ► Frequency resolution of human hearing is not uniform: it is better at low frequencies and worse at high frequencies ► So, we can use longer STFT windows at bass (for getting better frequency resolution) and shorter windows at treble Just phase locking Phase locking and multiple window sizes

“Time-scale and pitch modification algorithms” 15/19 Vocoders How to improve sharpness of transients? How to improve sharpness of transients? ► We can directly paste transients to output without stretching (and phase modification) ► Unwrapping of steady harmonics through transients Phase locking and multiple window sizes + transients pasted

“Time-scale and pitch modification algorithms” 16/19 Vocoders Summary Summary ► Good quality for complex, polyphonic signals ► Some phasiness (even with phase locking) ► Smearing of transients (unless special care taken) ► Noises sometimes sound unnaturally ► CPU-intensive (but still faster than realtime) Implementations Implementations ► Specialized software: SlowGold, Serato Time’n’Pitch, iZotope Radius + –