2. Communications & Multimedia Signal Processing Formant Based Synthesizer Qin Yan Communication & Multimedia Signal Processing Group Dept of Electronic & Computer Engineering, Brunel University 28 July, 2004

3. Communications & Multimedia Signal Processing Main Progress Kalman filter based formant tracking system in clean speech Speech Synthesis via formant tracks

4. Communications & Multimedia Signal Processing Formant Candidate Estimation LP Pole Analysis Kalman Filter Noisy Speech Restored Formant & Bandwidth tracks Formant Candidate Estimation Kalman Filter Vowel/ Consonant Classification Voiced? Yes No Noise Model LP-based Spectral Subtraction VAD Pos.& neg. Poles Reconstruction LP Spectrum Reconstruction Residual Real Pole Speech Reconstruction Enhanced Speech Formant Track Restoration Module Formant based Speech Enhancement System

5. Communications & Multimedia Signal Processing Confidence Score Calculation LP Pole Analysis Kalman Filter Clean Speech Formant & Bandwidth tracks Real Poles Speech Reconstruction Output Speech Residual Confidence Score Calculation Kalman Filter Positive Poles Vowel/ Consonant Classification Vowel? Yes No Formant Candidate Interpolation Formant Candidate Interpolation Speech Synthesis System Kalman Filter based Formant Tracker for Clean Speech Speech Synthesizer via Formant Tracks

6. Communications & Multimedia Signal Processing Vowel/Consonant Classification Discriminant feature used is the slope coefficient of a 1 st order polynomial of LP spectrum; Positive slope: Consonant; Negative slope: Vowel Confidence Scores of Formant Candidates The score quantifies how significant a pole is Score for Vowels: Mag(m) /BW(m) Score for Consonant: m*Mag(m) / BW(m) The candidate with highest score is interpolated with the closest formant candidate. The rest of formant candidates are sorted in ascending order. Interpolation function: Where W(m) is the weights Parallel Kalman Filters Two kalman filters: One for vowel segments, the other for consonant segments. Kalman Filter based Formant Track in Clean Speech

7. Communications & Multimedia Signal Processing Performance Red : Formant tracks from 2D-HMM; Green : Formant tracks from Kalman filter

8. Communications & Multimedia Signal Processing Speech Synthesis via Formant tracks Pos.& neg. Poles Reconstruction Noisy Speech Real Pole Speech Reconstruction Enhanced Speech Residual Restored Formant track LP Pole Analysis  Real poles are included to adjust the slope of LP spectrum  LP order = Number of formant tracks + 1 HMM based Formant tracks Kalman Filter based Formant Tracks

9. Communications & Multimedia Signal Processing The End

10. Communications & Multimedia Signal Processing Performance Evaluation

11. Communications & Multimedia Signal Processing Confidence Score Calculation LP Pole Analysis Kalman Filter Clean Speech Formant & Bandwidth tracks Real Poles Speech Reconstruction Output Speech Residual Confidence Score Calculation Kalman Filter Positive Poles Vowel/ Consonant Classification Vowel? Yes No Formant Candidate Interpolation Formant Candidate Interpolation Kalman Filter based Formant Tracker for Clean Speech Speech Synthesizer via Formant Tracks

12. Communications & Multimedia Signal Processing Significance Score Calculation LP Pole Analysis Kalman Filter Noisy Speech Formant & Bandwidth tracks Significance Score Calculation Kalman Filter Vowel/ Consonant Classification Voiced? Yes No Formant Candidate Interpolation Formant Candidate Interpolation Noise Model LP-based Spectral Subtraction VAD

13. Communications & Multimedia Signal Processing Source Speech Cepstral Feature Analysis LP Pole Analysis Speech HMMs Training Formant Features Extraction Speech Labelling & Segmentation Formant HMMs Training Formant candidates classification Formant Candidates Interpolation Formant Tracks State-dependent Kalman Filter R F i, BW i

14. Communications & Multimedia Signal Processing LP Pole Analysis Noisy Speech Restored Formant & Bandwidth tracks Formant Candidate Estimation Kalman Filter Vowel/ Consonant Classification LP Model Of Noise LP-Analysis and LP-Spectral Subtraction VAD Pos.& neg. Poles Reconstruction LP Spectrum Reconstruction Residual Speech Reconstruction Enhanced Speech Formant Track Restoration Module

15. Communications & Multimedia Signal Processing Formant Candidate Estimation LP Pole Analysis Kalman Filter Noisy Speech Restored Formant & Bandwidth tracks Formant Candidate Estimation Kalman Filter Vowel/ Consonant Classification Voiced? Yes No Noise Model LP-based Spectral Subtraction VAD