Presentation is loading. Please wait.

Presentation is loading. Please wait.


Similar presentations


Richard Stern Department of Electrical and Computer Engineering and School of Computer Science Carnegie Mellon University Pittsburgh, Pennsylvania 15213 Telephone: (412) ; FAX: (412) Institute for Mathematics and its Applications University of Minnesota September 19, 2000

2 Introduction As speech recognition is transferred from the laboratory to the marketplace robust recognition is becoming increasingly important “Robustness” in 1985: Recognition in a quiet room using desktop microphones Robustness in 2000: Recognition over a cell phone in a car with the windows down and the radio playing at highway speeds

3 What I’ll talk about today ...
Why we use cepstral-like representations Some “classical” approaches to robustness Some “modern” approaches to robustness Some alternate representations Some remaining open issues

4 The source-filter model of speech
A useful model for representing the generation of speech sounds: Amplitude Pitch Pulse train source Noise source Vocal tract model p[n]

5 Implementation of MFCC processing
Compute magnitude-squared of Fourier transform Apply triangular frequency weights that represent the effects of peripheral auditory frequency resolution Take log of outputs Compute cepstra using discrete cosine transform Smooth by dropping higher-order coefficients

6 Implementation of PLP processing
Compute magnitude-squared of Fourier transform Apply triangular frequency weights that represent the effects of peripheral auditory frequency resolution Apply compressive nonlinearities Compute discrete cosine transform Smooth using autoregressive modeling Compute cepstra using linear recursion

7 Rationale for cepstral-like parameters
The cepstrum is the inverse transform of the log of the magnitude of the spectrum Useful for separating convolved signals (like the source and filter in the speech production model) “Homomorphic filtering” Alternatively, cepstral processing be thought of as the Fourier series expansion of the magnitude of the Fourier transform

8 An example

9 The vowel /uh/ in “one” after windowing

10 The raw spectrum

11 Signal representations in MFCC processing

12 Additional parameters typically used
Delta cepstra and delta-delta cepstra Power and delta power Comment: These features restore (some) temporal dependencies … more heroic approaches exist as well (e.g. Alwan, Hermansky)

13 Challenges in robust recognition
“Classical” problems: Additive noise Linear filtering “Modern” problems: Transient degradations Very low SNR “Difficult” problems: Highly spontaneous speech Speech masked by other speech

14 “Classical” robust recognition: A model of the environment
“Clean” speech x[m] h[m] Linear Filtering n[m] Additive Noise + z[m] Degraded speech

Close-talking microphone: Desktop microphone:

16 Representation of environmental effects in cepstral domain
Power spectra: Effect of noise and filtering on cepstral or log spectral features: or where is referred to as the “environment function” x[m] h[m] n[m] + z[m]

17 Another look at environmental distortions: Additive environmental compensation vectors
Environment functions for the PCC-160 cardiod desktop mic: Comment: Functions depend on SNR and phoneme identity

18 Highpass filtering of cepstral features
Examples: CMN (CMU et al., RASTA, J-RASTA (OGI/ICSI/IDIAP et al.), multi-level CMN (Microsoft, et al.) Comments: Application to cepstral features compensates for linear filtering; application to spectral features compensates for additive noise “Great value for the money” ^ z x Highpass filter

19 Two common cepstral highpass filters
CMN (Cepstral Mean Normalization): RASTA (Relative Spectral Processing, 1994 version):

20 “Frequency response” of CMN and RASTA filters
Comment: Both RASTA and CMN have zero DC response

21 Principles of model-based environmental compensation
Attempt to estimate parameters characterizing unknown filter and noise that when applied in inverse fashion will maximize the likelihood of the observations x[m] h[m] n[m] + z[m]

22 Model-based compensation for noise and filtering: The VTS algorithm
The VTS algorithm (Moreno, Raj, Stern, 1996): Approximate f(x,n,q) by the first several terms of its Taylor series expansion, assuming that n and q are known The effects of f(x,n,q) on the statistics of the speech features then can be obtained analytically The EM algorithm is used to find the values of n and q that maximize the likelihood of the observations The statistics of the incoming cepstral vectors are re-estimated using MMSE techniques

23 The good news: VTS improves recognition accuracy in “stationary” noise
(1990) Comment: More accurate modeling of VTS improves recognition accuracy at all SNRs compared to CDCN and CMN

24 But the bad news: Model-based compensation doesn’t work very well in transient noise
CDCN does not improve speech recognition errors in music very much

25 So what can we do about transient noises?
Two major approaches: Sub-band recognition (e.g. Bourlard, Morgan, Hermansky et al.) Missing-feature recognition (e.g. Cooke, Green, Lippmann et al.) At CMU we’ve been working on a variant of the missing-feature approach

Basic approach: Decompose speech into several adjacent frequency bands Train separate recognizers to process each band Recombine information (somehow) Comment: Motivated by observation of Fletcher (and Allen) that the auditory system processes speech in separate frequency bands Some implementation decisions: How many bands? At what level to do the splits and merges? How to recombine and weight separate contributions?

General approach: Determine which cells of a spectrogram-like display are unreliable (or “missing”) Ignore missing features or make best guess about their values based on data that are present


Some regions are affected far more than others

All regions with SNR less than 0 dB deemed missing (dark blue) Recognition performed based on colored regions alone

31 Filling in missing features at CMU (Raj)
We modify the incoming features rather than the internal models (which is what has been done at Sheffield) Why modify the incoming features? More flexible feature set (can use cepstral rather than log spectral features) Simpler processing No need to modify recognizer

32 Recognition accuracy using compensated cepstra, speech corrupted by white noise
Cluster Based Recon. Spectral Subtraction Temporal Correlations Accuracy (%) Baseline SNR (dB) Large improvements in recognition accuracy can be obtained by reconstruction of corrupted regions of noisy speech spectrograms Knowledge of locations of “missing” features needed

33 Recognition accuracy using compensated cepstra, speech corrupted by music
Cluster Based Recon. Spectral Subtraction Temporal Correlations Accuracy (%) Baseline SNR (dB) Recognition accuracy goes up from 7% to 69% at 0 dB with cluster based reconstruction

34 So how can we detect “missing” regions?
Current approach: Pitch detection to comb out harmonics in voiced segments Multivariate Bayesian classifiers using several features such as Ratio of power at harmonics relative to neighboring frequencies Extent of temporal synchrony to fundamental frequency How well we’re doing now with blind identification: About half way between baseline results and results using perfect knowledge of which data are missing About 25% of possible improvement for background music

35 Missing features versus multi-band recognition
Multi-band approaches are typically implemented with a relatively small number of channels while …. …. with missing feature approaches, every time-frequency point can be considered or ignored The full-combination method for multi-band recognition considers every possible combination of present or missing bands, eliminating the need for blind identification of optimal combination of inputs Nevertheless, missing-feature approaches may provide superior recognition accuracy because they enable a finer partitioning of the observation space if we could solve the identification problem

36 Some other types of representations
Physiologically-motivated representations (“ear models”) Seneff, Ghitza, Lyon/Slaney, Patterson, etc. Feature extraction using “smart” nonlinear transformations Hermansky et al.

37 Physiologically-motivated speech processing
In recent years signal processing motivated by knoweldge of human auditory perception has become more popular Abilities of human audition form a powerful existence proof

38 Some auditory principles that system developers consider
Structure of auditory periphery: Linear bandpass filtering Nonlinear rectification with saturation/gain control Further analysis Dependence of bandwidth of peripheral filters on center frequency Nonlinear phenomena: Saturation Lateral suppression Temporal response: Synchrony and phase locking at low frequencies

39 An example: The Seneff model

40 Timing information in the Seneff model
Seneff model includes the effects of synchrony at low frequencies Synchrony detector in Seneff model records extent to which response in a frequency band is phase-locked with the channel’s center frequency Local synchrony has been shown to represent vowels more robustly in the peripheral auditory system in the presence of additive noise (e.g. Young and Sachs) Related work by Ghitza, DeMori, and others shows improvements in recognition accuracy relative to features based on mean rate, but at the expense of much more computation

Number of multiplications per ms of speech: Comment: auditory computation is extremely expensive

42 Some other comments on auditory models
“Correlogram”-type representations (channel-by-channel running autocorrelation functions) being explored by some researchers (Slaney, Patterson, et al.) Much more information in display Auditory models have not yet realized their full potential because ... Feature set must be matched to classification system ….. features generally not Gaussian All aspects of available feature must be used Research groups need both auditory and ASR experts

43 “Smart” feature extraction using non-linear transformations (Hermansky group)
Complementary approaches using temporal slices (mostly): Temporal linear discriminant analysis (LDA) to obtain maximally-discriminable basis functions over a ~1-sec interval in each critical band Three vectors with greatest eigenvalues are used as RASTA-like filters in each of 15 critical bands Karhunen-Loeve transform used to reduce dimensionality down to 39 based on training data TRAP features Use MLP to provide nonlinear mapping from temporal trajectories to phoneme likelihoods Modulation-filtered spectrogram (MSG) Pass spectrogram features through two temporal modulation filters (0-8 Hz and 8-16 Hz)

44 Use of nonlinear feature transformations in Aurora evaluation
Multiple feature sets combined by averaging feature values after nonlinear mapping Best system combines transformed PLP features, transformed MSG features, plus TRAP features (63% improvement over baseline!) Aurora evaluation system used reduced temporal span and other shortcuts to meet delay, processing time, and memory specs of evaluation (40% net improvement over baseline) Comment: Procedure effectively moves some of the “training” to the level of the features …. generalization to larger tasks remains to be verified

45 Feature combination versus compensation combination: The CMU SPINE System

46 SPINE evaluation conditions

47 The CMU SPINE system (Singh)
Three feature sets considered: Mel cepstra PLP cepstra Mel cepstra of lowpass filtered speech Four compensation schemes: Codeword Dependent Codebook Normalization (CDCN) Vector Taylor Series (VTS) Singular Value Decomposition (SVD) Karhunen-Loeve Transform-based noise cancellation (KLT) Additional features from ICSI/OGI: PLP cepstra subjected to MLP and KL transform for orthogonalization

48 Summary of CMU and CMU-ICSI-OGI SPINE results
(MFCC) 4 Comp. 4 Feat. ICSI/OGI 3 Feat.

49 Comments Some techniques we haven’t discussed:
VTLN Microphone arrays Time-frequency representations (e.g. wavelets) Robustness to Lombard speech, speaking style, etc. Many others Some hard problems not addressed: Very low SNR ASR Highly spontaneous speech (!) A representation or pronunciation modeling issue?

50 Summary Despite many shortcomings, cepstral-based features are well motivated, typically augmented by cepstral highpass filtering “Classical” model-based robustness techniques work reasonably well in combating quasi-stationary degradations “Modern” multiband and missing-feature techniques show great promise in coping with transient interference , etc. Auditory models remain appealing, although their potential has not yet been realized “Smart” features can provide dramatic improvements, at least in small tasks Feature combination will be key component of future systems



Similar presentations

Ads by Google