1. Microcomputer Systems 2
Digital Systems: Hardware Organization and Design
9/21/2018
Microcomputer Systems 2
Time Stretching & Pitch Shifting of Audio Signals
Architecture of a Respresentative 32 Bit Processor

2. Time Stretching & Pitch Shifting of Audio Signals
Digital Systems: Hardware Organization and Design
9/21/2018
Time Stretching & Pitch Shifting of Audio Signals
Outline
Introduction
Techniques Used for Time Compression/Expansion and Pitch Shifting
Comparison
Timbre and Formants
Architecture of a Respresentative 32 Bit Processor

3. Digital Systems: Hardware Organization and Design
9/21/2018
Outline
Introduction
Frequency Shift vs. Pitch Shift – Audio Examples
Time Compression/Expansion
Techniques Used for Time Compression/Expansion and Pitch Shifting
The Phase Vocoder
Related Topics
Why Phase
Time Domain Harmonic Scaling (TDHS)
More recent approaches
Comparison
Which Method to Use
Pitch Shifting Considerations
Audio Examples
Timbre and Formants
Phase Vocoder and Formants
Time Domain Harmronic scaling and Formants
21 September 2018
Veton Këpuska
Architecture of a Respresentative 32 Bit Processor

4. Introduction Time Stretching & Pitch Shifting
Are two dominant techniques that used for speech and sound manipulation.
Typical applications entail:
Changing the speed of play-back (altering the length of the signal) without altering the pitch of the voice and/or instruments
Changing the pitch of the voice and/or instruments without changing the length of the signal.
21 September 2018
Veton Këpuska

5. Pitch Shifting

6. Pitch Shifting:
As opposed to the process of pitch transposition achieved using (a simple) sample rate conversion, Pitch Shifting is a way to change the pitch of a signal without changing its length.
In practical applications, this is usually achieved by changing the length of a sound using one of the methods discussed next and then performing a sample rate conversion to change the pitch.
21 September 2018
Veton Këpuska

7. Introduction Pitch Shifting is NOT Frequency Shifting:
There exists a certain confusion in terminology in the literature, as Pitch Shifting is often also incorrectly named 'Frequency Shifting'.
A true Frequency Shift (as obtainable by modulating an analytic signal by a complex exponential) will shift the spectrum of a sound, while
Pitch Shifting will dilate it, upholding the harmonic relationship of the sound.
Frequency Shifting yields a metallic, inharmonic sound which may well be an interesting special effect but which is a totally inadequate process for changing the pitch of any harmonic sound except a single sine wave.
21 September 2018
Veton Këpuska

8. Audio Examples of Pitch Shifting vs. Frequency Shifting
Original Sound:
Pitch Shifted:
Frequency Shifted:
21 September 2018
Veton Këpuska

9. Time Compression/Expansion

10. Time Compression/Expansion
Time Compression/Expansion, also known as "Time Stretching" is the reciprocal process to Pitch Shifting.
It leaves the pitch of the signal intact while changing its speed (tempo).
This is a useful application when you wish to change the speed of a voiceover without messing with the timbre of the voice.
21 September 2018
Veton Këpuska

11. Time Compression/Expansion
There are several fairly good methods to do time compression/expansion and pitch shifting but most of them will not perform well on all different kinds of signals and for any desired amount of shift/stretch ratio.
Typically, good algorithms allow pitch shifting up to 5 semitones on average or stretching the length by 130%.
When time stretching and pitch shifting single instrument recordings you might even be able to achieve a 200% time stretch, or a one-octave pitch shift with no audible loss in quality.
21 September 2018
Veton Këpuska

12. Time Compression/Expansion of Speech
Typical Goals
To either speed up or slow down a speech signal while maintaining the approximate pitch
Applications
Change voice mail playback
Court stenographers-play proceedings quicker
Sound effects
Etc…
21 September 2018
Veton Këpuska

13. Techniques Used for Time Compression/Expansion & Pitch Shifting
Option 1 – Change sample rate
If you modify the sample rate, you can change the speed but the pitch is also changed
Increase sample rate = higher pitch (chipmunk sound)
Decrease sample rate = lower pitch (drawn out echo sound)
Option 2 – Decimate or Interpolate Signal
If you change the number of samples, the result is the same as modifying the sample rate
21 September 2018
Veton Këpuska

14. Techniques Used for Time Compression/Expansion & Pitch Shifting
Option 3 – Use more complex methods
This will change the speed of the sample while preserving the pitch data
Short Time Fourier Transform
Short Time Fourier Transform Magnitude
Sinusoidal Synthesis
Linear Prediction Synthesis
21 September 2018
Veton Këpuska

15. Techniques Used for Time Compression/Expansion & Pitch Shifting
Currently, there are two different principal time compression/expansion and pitch shifting schemes employed in most of today's applications:
Phase Vocoder.
Time Domain Harmonic Scaling (TDHS).
21 September 2018
Veton Këpuska

16. Phase Vocoder

17. Phase Vocoder
Phase Vocoder. This method was introduced by Flanagan and Golden in 1966 and digitally implemented by Portnoff ten years later.
Portnoff, M.R. 1981a. "Short-Time Fourier Analysis of Sampled Speech." IEEE Transactions on Acoustics, Speech and Signal Processing ASSP-29(3):
Portnoff, M.R. 1981b. "Time-Scale Modification of Speech Based on Short-Time Fourier Analysis." IEEE Transactions on Acoustics, Speech and Signal Processing ASSP-29(3):
21 September 2018
Veton Këpuska

18. Phase Vocoder
It uses a Short Time Fourier Transform (use abbreviation STFT from here on) to convert the audio signal to the complex Fourier representation.
Since the STFT returns the frequency domain representation of the signal at a fixed frequency grid, the actual frequencies of the partial bins have to be found by converting the relative phase change between two STFT outputs to actual frequency changes.
Note the term 'partial' has nothing to do with the signal harmonics. In fact, a STFT will never readily give you any information about true harmonics if you are not matching the STFT length to the fundamental frequency of the signal – and even then is the frequency domain resolution quite different to what our ear and auditory system perceives.
The timebase of the signal is changed by calculating the frequency changes in the Fourier domain on a different time basis, and then an iSTFT is done to regain the time domain representation of the signal.
21 September 2018
Veton Këpuska

19. Phase Vocoder
Phase vocoder algorithms are used mainly in scientific and educational software products (to show the use and limitations of the Fourier Transform) but have gained in popularity over the past few years due to improvements that made it possible to greatly reduce the artifacts of the "original" phase vocoder algorithm.
The basic phase vocoder suffers from a severe drawback because it introduces a considerable amount of artifacts audible as 'smearing' and 'reverberation' (even at low expansion ratios) due to the “non-synchronized vertical coherence of the sine and cosine basis functions” that are used to change the timebase.
21 September 2018
Veton Këpuska

20. Phase Vocoder
Puckette, Laroche and Dolson have shown that the phasiness can be greatly reduced by picking peaks in the Fourier spectrum and keeping the relative phases around the peaks unchanged. Even though this improves the quality considerably it still renders the result somewhat phasey and diffuse when compared to time domain methods.
Current research focuses on improving the phase vocoder by applying intra-frame sinusoidal sweep and ramp rate correction (Bristow-Johnson and Bogdanowicz) and multi-resolution phase vocoder concepts (Bonada).
21 September 2018
Veton Këpuska

21. Links to Publicly Available Vocoders
Pointers - Phase Vocoder:
The MIT Lab Phase Vocoder
WaveMasher - GPL/Open Source Phase Vocoder by Kenneth Sturgis
Sculptor: A Real Time Phase Vocoder by Nick Bailey
A Phase Vocoder implementation using Matlab 
More reading on the Phase Vocoder
The IRCAM "Super Phase Vocoder“
S.M.Bernsee's Pitch Shifting Using The Fourier Transform article (with C code)
21 September 2018
Veton Këpuska

22. Time Domain Harmonic Scaling (TDHS).

23. Time Domain Harmonic Scaling (TDHS).
Time Domain Harmonic Scaling (TDHS). This is based on a method proposed by Rabiner and Schafer in It is heavily based on a correct estimate of the fundamental frequency of the sound processed.
21 September 2018
Veton Këpuska

24. Theory Short Time Fourier Transform Methods
Chapter 7 in our text (Discrete-Time Speech Signal Processing)
Refer to notes from in class for mathematical theory of operation
I will pick up from where Dr. Kepuska stopped in his notes
21 September 2018
Veton Këpuska

25. How is the Speech/Sound Signal Processed
Link:
Ch7-Short-Time_Fourier_Transform_Analysis_and_Synthesis.ppt
21 September 2018
Veton Këpuska

26. Terminology & Basic Idea
Frame Rate
Window Size
21 September 2018
Veton Këpuska

27. Short Time Fourier Transform
Also called the Fairbanks method
Extract successive short-time segments and then discard the following ones
STFT
Decimate
Samples
IFFT
OLA
Signal
Output
21 September 2018
Veton Këpuska

28. Short Time Fourier Transform
Frame Rate factor L
In frequency domain after taking the STFT, you get
X(nL,ω)
Form a new signal by
Y(nL, ω) = X(snL, ω)
where s = compression factor
Take Inverse Fourier Transform
Use Overlap and Add method to form new signal
21 September 2018
Veton Këpuska

29. Short Time Fourier Transform
X(nL, ω)
Y(nL, ω)
= X(2nL, ω)
21 September 2018
Veton Këpuska

30. Short Time Fourier Transform
New Sequence
Original
Windowed
Sequence
21 September 2018
Veton Këpuska

31. Short Time Fourier Transform
Problems
Pitch Synchronization
It is highly likely that the pitch periods will not line up properly
21 September 2018
Veton Këpuska

32. Short Time Fourier Transform Magnitude
Problems with STFT method relate directly to the linear phase component of the STFT
Time shift = phase change
Alternate approach is to only use the magnitude portion of the STFT—Short Time Fourier Transform Magnitude
21 September 2018
Veton Këpuska

33. Short Time Fourier Transform Magnitude
Compression
With the Fairbanks method, time slices were discarded
Now we can just compress the time slices
Form a new signal by
|Y(nM, ω)| = |X(nL, ω)| where
M = compression factor = L / speed
i.e. for speeding up by two => M = L/2
21 September 2018
Veton Këpuska

34. Short Time Fourier Transform Magnitude
Compression
Take Inverse Fourier Transform
Use Overlap and Add method to form new signal
21 September 2018
Veton Këpuska

35. Short Time Fourier Transform Magnitude
X(nL, ω)
Y(nM, ω)
= X(nL, ω)
M=L/2
21 September 2018
Veton Këpuska

36. Short Time Fourier Transform Magnitude
New Sequence
Original
Windowed
Sequence
21 September 2018
Veton Këpuska

37. Other Methods Sinusoidal Synthesis—Chapter 9
Time-warp the sinewave frequency track and the amplitude function
This technique has been successful with not only speech but also music, biological, and mechanical signals
Problems
Does not maintain the original phase relations
Suffer from reverberance
21 September 2018
Veton Këpuska

38. Other Methods Linear Prediction Synthesis
Use Homomorphic and Linear Prediction results to modify the time base
Book briefly mentions this is possible but ran out of time before I could investigate this process more
21 September 2018
Veton Këpuska

39. Other Methods New Techniques Software
Internet search showed several methods trying to improve on what is out there now
Software
Different software programs that will change speed for you
Adobe Audition is one of the most all encompassing right now
21 September 2018
Veton Këpuska

40. Matlab Code -Prepare the Workspace
%%%%%%%%%%%%%%%%
% Prepare Workspace
close all;
clear all;
window_size_1 = 200;
frame_rate_1 = 100;
%Speed to slow down by
speed = 2;
21 September 2018
Veton Këpuska

41. Matlab Code -Load the Speech Signal
%%%%%%%%%%%%%%%%
% Load Data File
filename = input('Please enter the file name to be used. ');
[sample_data,sample_rate,nbits] = wavread(filename);
loop_time = floor(max(size(sample_data))/frame_rate_1);
sample_data((max(size(sample_data))):(loop_time+1)* frame_rate_1)=0;
21 September 2018
Veton Këpuska

42. Matlab Code -Develop the Window
%%%%%%%%%%%%%%%%
% Create Windows
% Want windows of 25ms
% File sampled at 10,000 samples/sec
% Want a window of size * 25ms(10ms)
triangle_30ms = triang(window_size_1);
%triangle_30ms = hamming(window_size_1);
W0 = sum(triangle_30ms);
21 September 2018
Veton Këpuska

43. Matlab Code -Window the Entire Speech Signal
%%%%%%%%%%%%%%%%
% Window the speech
for i =0:loop_time-1
window_data(:,i+1)=sample_data((frame_rate_1*i)+1:((i+2)* frame_rate_1)).*triangle_30ms;
end
21 September 2018
Veton Këpuska

44. Matlab Code -Perform the Fast Fourier Transform
%%%%%%%%%%%%%%%%
% Create FFT
for i = 1:loop_time
window_data_fft(:,i) = fft(window_data(:,i),1024);
end
21 September 2018
Veton Këpuska

45. Matlab Code -Recreate the Modified Signal
%%%%%%%%%%%%%%%%
% Recreate Original Signal
%Initialize the recreated signals
reconstructed_signal(1:(loop_time+1)*frame_rate_1)=0;
real_reconstructed_signal(1:(loop_time+1)*frame_rate_1)=0;
modified_reconstructed_signal(1:(loop_time+3)*(frame_rate_1/speed))=0;
modified_reconstructed_signal_compressed(1:(loop_time+3)* (frame_rate_1/ speed))=0;
21 September 2018
Veton Këpuska

46. Matlab Code -Recreate the Modified Signal
% Perform the ifft
for i = 1:loop_time
recreated_data_ifft(:,i) = ifft(window_data_fft(:,i),1024);
real_recreated_data_ifft(:,i) = ifft(abs(window_data_fft(:,i)),1024);
truncated_recreated_data_ifft(:,i) = recreated_data_ifft(1:window_size_1,i).*(frame_rate_1/W0);
real_truncated_recreated_data_ifft(:,i) = real_recreated_data_ifft(1:window_size_1,i).*(frame_rate_1/W0);
end
21 September 2018
Veton Këpuska

47. Matlab Code -Recreate the Modified Signal
% Get back to the original signal
for i=0:loop_time-1
reconstructed_signal((frame_rate_1*i)+1:((i+2)*frame_rate_1)) = reconstructed_signal((frame_rate_1*i)+1:((i+2)*frame_rate_1)) + truncated_recreated_data_ifft(:,i+1)';
real_reconstructed_signal((frame_rate_1*i)+1:((i+2)*frame_rate_1)) = real_reconstructed_signal((frame_rate_1*i)+1:((i+2)*frame_rate_1)) + real_truncated_recreated_data_ifft(:,i+1)';
end
21 September 2018
Veton Këpuska

48. Matlab Code -Recreate the Modified Signal
% Get a modified signal by deleting certain parts (STFT)
for i=0:(loop_time-1)/speed
modified_reconstructed_signal((frame_rate_1*i)+1:((i+2)* frame_rate_1)) = modified_reconstructed_signal((frame_rate_1*i)+1:((i+2)*frame_rate_1)) + real_truncated_recreated_data_ifft(:,i*speed+1)';
end
21 September 2018
Veton Këpuska

49. Matlab Code -Recreate the Modified Signal
% Initialize the compressed sequence (STFTM)
modified_reconstructed_signal_compressed(1:frame_rate_1+frame_rate_1/speed+1)=truncated_recreated_data_ifft(frame_rate_1-frame_rate_1/speed:window_size_1,1)';
% Get a modified signal by compressing
for i=0:(loop_time-2)
modified_reconstructed_signal_compressed((frame_rate_1/speed*i)+1:(frame_rate_1/speed*i)+window_size_1) = modified_reconstructed_signal_compressed((frame_rate_1/speed*i)+1:(frame_rate_1/speed*i)+window_size_1) + real_truncated_recreated_data_ifft(:,i+2)';
end
21 September 2018
Veton Këpuska

50. Matlab Code -Plot Results
%%%%%%%%%%%%%%%%
% Plot Results
Figure; subplot(211)
plot(sample_data)
title('Original Speech'); v1=axis;
hold on; subplot(212)
plot(real(modified_reconstructed_signal))
title(['STFT Synthesis w/ Speed = ',num2str(speed),'X']); v2=axis;
if speed > 1
subplot(211); axis(v1)
subplot(212); axis(v1)
else
subplot(211); axis(v2)
subplot(212); axis(v2)
end
21 September 2018
Veton Këpuska

51. Matlab Code -Write Sound Files
%%%%%%%%%%%%%%%%
% Write sound files
wavwrite(modified_reconstructed_signal,sample_rate,nbits,'C:\Classes\ECE_5525\tea party fairbanks 2x.wav')
21 September 2018
Veton Këpuska

52. Examples Baseline Samples
Sample Rate 2X
STFT Sound file
Original File
Sample Rate .5X
STFTM Sound file
21 September 2018
Veton Këpuska

53. Examples STFT—Speed 0.5X
Sound file
21 September 2018
Veton Këpuska

54. Examples STFT—Speed 2X
Sound file
21 September 2018
Veton Këpuska

55. Examples STFT—Speed 4X
Sound file
21 September 2018
Veton Këpuska

56. Examples STFTM—Speed 0.5X
Sound file
21 September 2018
Veton Këpuska

57. Examples STFTM—Speed 2X
Sound file
21 September 2018
Veton Këpuska

58. Examples STFTM—Speed 4X
Sound file
21 September 2018
Veton Këpuska

59. More Results Change in window size
If the window size becomes too small, then a change in pitch will occur
Need window to be 2 to 3 pitch periods long
I generally used 20 – 30 ms windows
21 September 2018
Veton Këpuska

60. More Results Change in frame rate
If the frame rate decreases too much, then there will be too many samples overlapping to get an intelligible signal
21 September 2018
Veton Këpuska

61. More Results Change filter type
Tried Hamming—not much perceptual difference
Using the window energy becomes important here
Frame Rate/W0 is not equal to one
21 September 2018
Veton Këpuska

62. Conclusion Optimum area
Frame rate is one half of the window size
Window size needs to be 2 to 3 pitch periods long
It is possible to easily change the time scale and still maintain the original pitch although the result is not always natural sounding
21 September 2018
Veton Këpuska

63. Conclusion Further investigation
What to do when you want to slow down over half.
Using the STFTM means there will be gaps between the sequences
21 September 2018
Veton Këpuska

64. Conclusion Further investigation
What to do when you want to slow down over half
Could replicate windowed segments
21 September 2018
Veton Këpuska

65. Conclusion Further investigation
Use the other methods to determine quality
Implement Sinusoidal Synthesis
Implement Linear Predictive Synthesis using linear prediction and homomorphic methods
Work on synchronizing pitch periods
Shift samples so that the peaks line up
Scott and Gerber—Synchronized Overlap and Add (SOLA)
Cross-correlation of two samples to find peak
Use the peaks to line up samples
Align the window at same relative location within a pitch period
21 September 2018
Veton Këpuska

66. Questions
Are there any questions?
21 September 2018
Veton Këpuska

67. References
Quatieri, Thomas E. Discrete-Time Speech Signal Processing. Prentice Hall, Upper Saddle River, NJ, 2002.
Rabiner, L.R. and Schafer, R.W. Digital Processing of Speech Signals. Prentice Hall, Upper Saddle River, NJ, 1978.
Oppenheim, A.V and Schafer, R.W. Digital Signal Processing. Prentice Hall, Englewood Cliffs, NJ, 1975.
Scott, R. and Gerber, S. “Pitch Synchronous Time-Compression of Speech,” Proc. Conf. Speech Communications Processing, p63-85, April 1972.
21 September 2018
Veton Këpuska

68. References
Fairbanks, G., Everitt, W.L., and Jaeger, R.P. “Method for Time or Frequency Compression-Expansion of Speech,” IEEE Transaction Audio and Electroacoustics, vol. AU-2 pp.7-12, Jan 1954.
21 September 2018
Veton Këpuska

69. Reference Material http://www.dspdimension.com/ of Stephan M. Bernsee
21 September 2018
Veton Këpuska