Cepstral Vector Normalization based On Stereo Data for Robust Speech Recognition Presenter: Shih-Hsiang Lin Luis Buera, Eduardo Lleida, Antonio Miguel, Alfonso Ortega, and Óscar Saz Communication Technologies Group (GTC) Aragon Institute of Engineering Research (I3A) University of Zaragoza, Spain IEEE Transactions on Audio, Speech and Language Processing, Feb.,2007

2 Reference “Cepstral Vector Normalization based On Stereo Data for Robust Speech Recognition,” IEEE Trans on Audio, Speech and Language Processing, 2007 “Multi-environment models based linear normalization for robust speech recognition in car conditions,” in Proc. ICASSP 2004 “Multi-environment models based linear normalization for robust speech recognition,” in Proc. SPECOM 2004 “Robust speech recognition in cars using phoneme dependent multi- environment linear normalization,” in Proc. EuroSpeech 2005 “Recent advances in PD-MEMLIN for speech recognition in car conditions,” in Proc. ASUR 2005

3 Outline Introduction Approaches –Multi-Environment Model-based Linear Normalization (MEMLIN) –Polynomial MEMLIN (P-MEMLIN) –Multi-Environment Model-based Histogram Normalization (MEMHIN) –Phoneme-Dependent MEMLIN (PD-MEMLIN) –Blind PD-MEMLIN Experimental Results Conclusions

4 Introduction Robustness techniques have been developed along the following two main lines of research –Acoustic model adaptation method Require more data and computing time –MAP, MLLR, PMC –Feature vector adaptation/normalization method Map recognition space feature vectors to the training space –High-pass filtering »The results produced by those methods are limited individually »CMN, RASTA processing –Model-based techniques »VTS, CDCN –Empirical compensation »Entirely data-driven »Need a training phase where some transformations are estimated by computing the frame-by-frame differences between the stereo data »SPLICE, POF

5 Introduction (cont.) This paper focuses on empirical feature vector normalization base on stereo data and the MMSE estimator –Based on the joint modeling of clean and noisy space which splits noisy space into several basic environments and models each basic noisy and clean feature spaces using GMMs –To learn a transformation between clean and noisy feature vectors associated with each pair of clean and noisy model Gaussians Noisy basic environment space Clean space

6 Noise Effects Convolutional Noise: Shifts the mean of the coefficients Additive Noise: Modifies the PDF, reducing the variances of the coefficients Real Car Environment: Modifies the mean and variance, jointly

7 MMSE-based Feature Vector Normalization Methods Given the noisy feature vector, the estimated clean feature vector is obtained by using the MMSE criterion as –Method 1: CMN No assumption are made in estimating The clean feature vector is approximated as To estimated the bias vector transformation,, the mean square error,, is defined and minimized w.r.t.

8 MMSE-based Feature Vector Normalization Methods (cont.) In some cases, the mean of the clean feature vectors is removed before training acoustic model, so the bias vector transformation is computed as –Method 2: (RATZ) Multivariate Gaussian-based cepstral Normalization Modeling the clean space using a GMM Approximate the clean feature vector as The estimation of can produce a mismatch

9 MMSE-based Feature Vector Normalization Methods (cont.) –Method 3: SPLICE Modeling the noisy space instead of the clean one using GMM Approximate the clean feature vector as –Furthermore, several acoustic conditions have been developed Interpolated RATZ (IRATZ) SPLICE with environmental model selection

10 Multi-Environment Model-based Linear Normalization (MEMLIN) MEMLIN Approximation –Noisy space is divided into a combination of several basic environment, and the noisy feature vectors are modeled as a GMM for each basic environment –Clean feature vectors are modeled using a GMM –Clean feature vectors can be approximated as a linear function of the noisy feature vector, which depends on the basic environment and the clean and noisy model Gaussians bias vector transformation

11 MEMLIN Enhancement –Given the noisy feature vector, the estimated clean feature vector is obtained by using the MMSE criterion as –calculate is considered to be uniformly distributed over all the environments has to be close to 1 MEMLIN (Cont.) estimated in a training phase using stereo data

12 MEMLIN (Cont.) MEMLIN Training –Given a stereo data corpus for each environment The bias vector transformation,, is estimated by minimizing the defined mean weighted squared square error

13 –Calculate The cross probability is simplified by avoiding the time dependence given by the noisy feature vector The term can be estimated by either using a hard solution or using a soft decision –Hard decision (using relatively frequency) –Soft decision MEMLIN (Cont.)

14 Experiments Results Using Basic MMSE-based methods A set of experiments were performed using the Spanish SpeechDat Car database –Seven basic environments were defined E1: car stopped, motor running E2: town traffic, closed windows, and climatizer off (silent conditions) E3: town traffic and noisy conditions (windows open, and/or climatizer on) E4: low speed, rough road, and silent conditions E5: low speed, rough road, and noisy conditions E6: high speed, good road, and silent conditions E7: high speed, good road, and noisy conditions –Tow channels have been used (stereo data) Close talK channel (CLK) Hands-free channel (HF) The recognition task is isolated and continuous digits recognition

15 Experiments Results (cont.) The SPLICE MS method always produces better results than does IRATZ - because of the assumption of the a posterior probability MEMLIN performed better than IRATEZ and SPLICE MS

16 Improvement Over MEMLIN There are two important approximations in MEMLIN expressions that can affect the final performance of the method –The selection of the linear model for associated with a pair of Gaussians compensates for the mean shift, but not for the modification of variance –Treating all of the sound in the same way There is always a bias vector transformation which maps from a noisy model Gaussian to every clean model one –e.g. non silence noisy feature vectors are mapped towards the clean silence

17 Polynomial MEMLIN (P-MEMLIN) The transformation function for P-MEMLIN is –Given the noisy feature vector, the estimated clean feature vector is obtained by using the MMSE criterion as – and are computed in the training phase using stereo data where If the standard deviation terms are equal, the algorithm expressions are the same as those in MEMLIN

18 Multi-Environment Model-based Histogram Normalization (MEMHIN) Sometime noise can produce a more complex modification of clean and noisy feature pdfs associated with a pair of Gaussians –In that case, the linear approximation for of MEMLIN or P- MEMLIN is not the best option –Therefore, a nonlinear model based on histogram equalization is used The transformation function for MEMHIN is expressed as – band histograms associated with and for each component of the noisy and clean feature vectors are obtained in the training phase

19 Results from modifications P-MEMLIN and MEMHIN provide significant improvement over MEMLIN when few Gaussians are considered –33.87% of MIMP for MEMLIN –39.12% of MIMP for P-MEMLIN –37.82% of MIMP for MEMHIN –However, if the algorithms are evaluated using more than eight Gaussians per environment, the mean results are very similar among the three models Then, additive car noise was added to clean signals of the Spanish SpeechDat Car database (5dB noise) 4 Gaussians per environment

20 Phoneme-Dependent MEMLIN (PD-MEMLIN)

21 PD-MEMLIN (Cont.) PD-MEMLIN Approximation –Noisy space is split into several basic environment. The noisy feature vectors associated with the different phonemes of each basic environment are modeled as GMM –The clean feature vectors of each phoneme are modeled as a GMM –Clean feature vector can be approximated by a linear function that depend on the environment and the phoneme-dependent Gaussians of the clean and noisy model

22 PD-MEMLIN Enhancement –Given the noisy feature vector, the estimated clean feature vector is obtained by using the MMSE criterion as –calculate PD-MEMLIN Training PD-MEMLIN (Cont.)

23 –cross probability –bias vector transformation PD-MEMLIN (Cont.) For a more detailed description, please refer to the paper Hard solution Soft solution

24 25 Spanish phonemes and the silence –To make a fair comparisons between two methods, the results have been plotted as a function of the number of Transformations per basic Environment (TpE) Results from PD-MEMLIN The number of noisy Gaussians for phoneme The number of clean Gaussians for phoneme The number of phoneme (1 for MEMLIN) (2,4,8,16,32) The results show that PD-MEMLIN makes significant improvements relative to MEMLIN, specially when more than for Gaussians per phoneme are used

25 Results from PD-MEMLIN (cont.) To estimate the limit of the PD-MEMLIN –Each frame was normalized using only the bias vector transformation of the “correct” phoneme (KPD-MEMLIN) KPD-MEMLIN PD-MEMLIN MCP: mean correct phoneme

26 Blind PD-MEMLIN In many cases, stereo data are not available –An iterative “blind” training procedure is needed –Assume that the noisy training feature vectors and the phoneme-dependent clean and noisy GMMs are available –The problem is to estimate the cross probability and the bias vector transformation –It consists of an initialization and an iterative process

27 Initialization –The cross probability is estimated using a modified Kullback-Leibler distance Gives a similarity measure of and without considering the effects of the noise –Assume that the noise modifies mainly the mean vectors of the Gaussian model So, the similarity is computed in terms of the a priori probabilities and the diagonal covariance matrices of the corresponding Gaussians Since KL distance is not symmetric, and it is not proportional to the likelihood; therefore, a pseudo-likelihood is defined Blind PD-MEMLIN (cont.)

28 Blind PD-MEMLIN (cont.) Finally, is estimated as –On the other hand, is defined as –The mean improvement in WER over the seven basic environments and four Gaussians per phoneme-depend GMM was 20.2%

29 Blind PD-MEMLIN (cont.) Iterative process –Objective function –The value of is obtained by taking partial derivatives and setting it equal to zero

30 Blind PD-MEMLIN (cont.) –Update the cross probability and the bias vector transformation –The mean improvement in WER in this case was 41.03% if n=1 and 46.90% if n=10

31 Results from Blind PD-MEMLIN The results show that blind PD-MEMLIN is able to produce improvement that are very similar to MEMLIN ones for all the TpE It can be observed that PD-MEMLIN obtains the Best improvement with the smallest TpE