A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 1 Abstract Perception of speech under adverse listening conditions may be improved by processing it to incorporate properties of clear speech. It needs automated detection of stop land-marks and enhancement of bursts and transition segments. A technique for accurate detection of stop landmarks in continuous speech based on parameters derived from Gaussian mixture modeling (GMM) of log magnitude spectrum is presented. Applying the technique on sentences from the TIMIT database resulted in burst detection rates of 98, 97, 95, 90, and 73 % at temporal accuracies of 30, 20, 15, 10, and 5 ms respectively.

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 2 1. INTRODUCTION Acoustic Landmarks Regions with concentration of phonetic information, important for speech perception Stop Landmarks  Closure  Release burst  Onset of voicing Closure ▲ Release burst ▲ ▲ Onset of voicing /apa/

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 3 Problems in Stop Perception Perception of transient sounds with low intensity severely affected by noise / hearing impairment Clear Speech  Style adapted by speakers under noisy conditions (~17 % more intelligible than conversational speech)  Acoustic landmarks modified in duration & intensity ◄ Conversational ▼ Clear ‘the book tells a story’

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 4 Speech Intelligibility Enhancement Using Properties of Clear Speech  Automated detection of landmarks with  Good temporal accuracy  High detection rate and low false detections  Modification of speech characteristics around the stop landmarks

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 5 Some Earlier Landmark Detection Techniques  Liu (1996): Rate-of-rise measures of parameters from a set of fixed spectral bands. Detection rate: 84 % at ms, ~50 % at 5-10 ms.  Niyogi & Sondhi (2002): Optimal filtering approach with log energy, log energy in the band > 3 kHz & Wiener entropy. Detection rate  90 % at 20 ms.

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 6 Objective Detection of stop landmarks using Gaussian mixture modeling (GMM) of speech spectrum ▪ for improving the temporal accuracy of detection and reducing insertion errors ▪ with adaptation to speech variability ▪ for enhancing burst and transition segments to improve speech intelligibility under adverse listening conditions

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 7 2. GAUSSIAN MIXTURE MODELING OF SHORT- TIME SPEECH SPECTRUM Approximation of spectrum using a weighted sum of Gaussian functions  Means  Variances  Mixture weights  Good spectral approximation with 4 or 5 Gaussians (approximating the spectral resonances)  Adaptive to speech variability

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 8 Spectral Modeling  Short-time log magnitude spectrum of speech signal (S.R. = 10 kHz)  6 ms Hanning windowed frames (for suppressing the harmonic structure)  1 frame per ms (for tracking abrupt variations)  512-point DFT  Estimation of GMM parameters using Expectation Maximization (EM) algorithm

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 9 Estimation of GMM Parameters  Spectrum treated as histogram with rectangular bins placed at each frequency index  Iterative computation of parameters as maximum likelihood estimates  Initialization  Means: Average formant frequencies [600, 1200, 2400, 3600 Hz]  Variances: Extreme formant bandwidths [160, 200, 300, 400 Hz]  Mixture weights: Equal for all Gaussians  Number of iterations: ≤ 12

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 10 Example: Modeling for a segment of vowel / a / Modeling of a segment of vowel /a/: (a) windowed segment of 6 ms, (b) log magnitude spectrum (in dB), (c) smoothened spectrum (in dB), (d) GMM approximated spectrum with dotted lines indicating the individual Gaussian components. Ag(n)Ag(n) g(n)g(n) g(n)g(n)

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay DETECTION OF STOP LANDMARKS Detection based on  Rate of change (ROC) of GMM parameters  Voicing onset offset detector  Spectral flatness measure

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 12

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 13 GMM Rate of Change  A g,  g,  g smoothened by 30-point median filter  ROC: First difference (time step = 2 ms)  ROC Peak → Possible location of burst onset

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 14 Voicing Onset-Offset Detection [Liu, 1996]  Energy variations E(n) in 0:400 Hz band (6 ms Hanning windowed segments, every 1 ms)  Rate-of-rise r e (n) with 26 ms time-step  Voicing onset [+g]: r e ( n )  +9dB Voicing offset [-g]: r e (n)  -9dB Spectral Flatness Measure [ Skowronski & Harris, 2006] (20 ms Hanning windowed segments, every 1 ms)  Fricative segments: SFM  1  Voiced segments: SFM  0

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 15 Stop Landmark Detection  For a voicing onset [ +g ] or voicing offset [ -g ] at t, locate the preceding [ +g ] or [ -g ] If [ -g ] at t 0, select GMM ROC peak at t b during ( t 0 -50, t ms ), Else select GMM ROC peak at t b during ( t-50, t ms) as the burst candidate.  A burst is declared, if { SFM > 0.5 for 1 ms during ( t b -15, t b +15 ms)} and {each of the norm. ampl. A 2, A 3, A 4 < 0.5 for at least 10 ms during ( t 0, t b )}.  For burst at t b, closure is located at t 0, and voicing onset at t.

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 16 / apa /: Waveform (a), Gaussian parameter tracks (b: 1 st, c: 2 nd, d: 3 rd, e: 4 th ). (a) (b) (c) (d) (e) Ag(n)Ag(n) g(n)g(n) g(n)g(n)

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 17 / apa /: Waveform (a), Spectrogram (b), GMM spectrogram (c), Gaussian ROC (d) (a) (b) (c) (d)

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 18 -g +g (a) (b) (c) (d) (e) A2A2 A4A4 A3A3 / apa /: Waveform (a), -g, +g peaks (b), SFM (c), GMM ROC (d), Normalized Gaussian amplitudes for Gaussian 2, 3, 4 (e) tbtb t t0t0 ROC peak SFM

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay TEST RESULTS Comparison with manually labeled landmarks  VCV utterances ▪ Stops / b /, / d /, / g /, / p /, / t /, / k / & vowels / a /, / i /, / u / ▪ 10 speakers (5 F, 5 M)  TIMIT sentences ▪ 50 sentences ▪ 5 speakers (3 F, 2 M)

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay Det. Rates for VCV Utterances

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay 21 Det. Rates for TIMIT Sentences Insertions : 13 % ( Clicks, glottal stops : 8 %, Vowel-semivowel : 4 %, Stop to /l/, /r/ : 1 % )

A. R. Jayan, P. C. Pandey, EE Dept., IIT Bombay CONCLUSION Detection rate obtained using GMM based technique: comparable to other methods at ms temporal accuracy, better at ms.