An Analysis of the Aurora Large Vocabulary Evaluation Authors: Naveen Parihar and Joseph Picone Inst. for Signal and Info. Processing Dept. Electrical and Computer Eng. Mississippi State University Contact Information: Box 9571 Mississippi State University Mississippi State, Mississippi Tel: Fax: EUROSPEECH 2003 URL: isip.msstate.edu/publications/conferences/eurospeech/2003/evaluation/ isip.msstate.edu/publications/conferences/eurospeech/2003/evaluation/

INTRODUCTION ABSTRACT In this paper, we analyze the results of the recent Aurora large vocabulary evaluations (ALV). Two consortia submitted proposals on speech recognition front ends for this evaluation: (1) Qualcomm, ICSI, and OGI (QIO), and (2) Motorola, France Telecom, and Alcatel (MFA). These front ends used a variety of noise reduction techniques including discriminative transforms, feature normalization, voice activity detection, and blind equalization. Participants used a common speech recognition engine to post-process their features. In this paper, we show that the results of this evaluation were not significantly impacted by suboptimal recognition system parameter settings. Without any front end specific tuning, the MFA front end outperforms the QIO front end by 9.6% relative. With tuning, the relative performance gap increases to 15.8%. Both the mismatched microphone and additive noise evaluation conditions resulted in a significant degradation in performance for both front ends.

INTRODUCTION MOTIVATION ALV goal was at least a 25% relative improvement over the baseline MFCC front end Two consortia participated:  QIO: QualComm, ICSI, OGI  MFA: Motorola, France Telecom, Alcatel Generic baseline LVCSR system with no front end specific tuning Would front end specific tuning change the rankings? MFCC: Overall WER – 50.3% 8 kHz – 49.6%16 kHz – 51.0% TS1TS2TS1TS2 58.1%41.0%62.2%39.8% QIO: Overall WER – 37.5% 8 kHz – 38.4%16 kHz – 36.5% TS1TS2TS1TS2 43.2%33.6%40.7%32.4% MFA: Overall WER – 34.5% 8 kHz – 34.5%16 kHz – 34.4% TS1TS2TS1TS2 37.5%31.4%37.2%31.5% ALV Evaluation Results

EVALUATION PARADIGM THE AURORA – 4 DATABASE Acoustic Training: Derived from 5000 word WSJ0 task TS1 (clean), and TS2 (multi-condition) Clean plus 6 noise conditions Randomly chosen SNR between 10 and 20 dB 2 microphone conditions (Sennheiser and secondary) 2 sample frequencies – 16 kHz and 8 kHz G.712 filtering at 8 kHz and P.341 filtering at 16 kHz Development and Evaluation Sets: Derived from WSJ0 Evaluation and Development sets 14 test sets for each 7 recorded on Sennheiser; 7 on secondary Clean plus 6 noise conditions Randomly chosen SNR between 5 and 15 dB G.712 filtering at 8 kHz and P.341 filtering at 16 kHz

EVALUATION PARADIGM BASELINE LVCSR SYSTEM Standard context-dependent cross- word HMM-based system: Acoustic models: state-tied 4-mixture cross-word triphones Language model: WSJ0 5K bigram Search: Viterbi one-best using lexical trees for N-gram cross- word decoding Lexicon: based on CMUlex Real-time: 4 xRT for training and 15 xRT for decoding on an 800 MHz Pentium Monophone Modeling State-Tying CD-Triphone Modeling CD-Triphone Modeling Mixture Modeling (2,4) Training Data

EVALUATION PARADIGM WI007 ETSI MFCC FRONT END The baseline HMM system used an ETSI standard MFCC-based front end: Zero-mean debiasing 10 ms frame duration 25 ms Hamming window Absolute energy 12 cepstral coefficients First and second derivatives Input Speech Fourier Transf. Analysis Cepstral Analysis Zero-mean and Pre-emphasis Energy /

FRONT END PROPOSALS QIO FRONT END 10 msec frame duration 25 msec analysis window 15 RASTA-like filtered cepstral coefficients MLP-based VAD Mean and variance normalization First and second derivatives Fourier Transform RASTA Mel-scale Filter Bank DCT Mean/Variance Normalization Input Speech / MLP-based VAD Qualcomm, ICSI, OGI (QIO) front end:

FRONT END PROPOSALS MFA FRONT END 10 msec frame duration 25 msec analysis window Mel-warped Wiener filter based noise reduction Energy-based VADNest Waveform processing to enhance SNR Weighted log-energy 12 cepstral coefficients Blind equalization (cepstral domain) VAD based on acceleration of various energy based measures First and second derivatives Input Speech Noise Reduction Cepstral Analysis Waveform Processing Blind Equalization Feature Processing VADNest VAD /

EXPERIMENTAL RESULTS FRONT END SPECIFIC TUNING Pruning beams (word, phone and state) were opened during the tuning process to eliminate search errors. Tuning parameters:  State-tying thresholds: solves the problem of sparsity of training data by sharing state distributions among phonetically similar states  Language model scale: controls influence of the language model relative to the acoustic models (more relevant for WSJ)  Word insertion penalty: balances insertions and deletions (always a concern in noisy environments)

EXPERIMENTAL RESULTS FRONT END SPECIFIC TUNING - QIO Parameter tuning  clean data recorded on Sennhieser mic. (corresponds to Training Set 1 and Devtest Set 1 of the Aurora-4 database)  8 kHz sampling frequency 7.5% relative improvement QIO FE # of Tied States State Tying ThresholdsLM Scale Word Ins. Pen. WER SplitMergeOccu. Base % Tuned %

EXPERIMENTAL RESULTS FRONT END SPECIFIC TUNING - MFA MFA FE # of Tied States State Tying ThresholdsLM Scale Word Ins. Pen. WER SplitMergeOccu. Base % Tuned % Parameter tuning  clean data recorded on Sennhieser mic. (corresponds to Training Set 1 and Devtest Set 1 of the Aurora-4 database)  8 kHz sampling frequency 9.4% relative improvement Ranking is still the same (14.9% vs. 12.5%) !

EXPERIMENTAL RESULTS COMPARISON OF TUNING Front End Train Set TuningAverage WER over 14 Test Sets QIO1No43.1% QIO2No38.1% Avg. WERNo38.4% QIO1Yes45.7% QIO2Yes35.3% Avg. WERYes40.5% MFA1No37.5% MFA2No31.8% Avg. WERNo34.7% MFA1Yes37.0% MFA2Yes31.1% Avg. WERYes34.1% Same Ranking: relative performance gap increased from 9.6% to 15.8% On TS1, MFA FE significantly better on all 14 test sets (MAPSSWE p=0.1%) On TS2, MFA FE significantly better only on test sets 5 and 14

EXPERIMENTAL RESULTS MICROPHONE VARIATION Train on Sennheiser mic.; evaluate on secondary mic. Matched conditions result in optimal performance Significant degradation for all front ends on mismatched conditions Both QIO and MFA provide improved robustness relative to MFCC baseline SennheiserSecondary ETSI QIO MFA

EXPERIMENTAL RESULTS ADDITIVE NOISE ETSI QIO MFA TS2TS3TS4TS5TS6TS7 Performance degrades on noise condition when systems are trained only on clean data Both QIO and MFA deliver improved performance TS2TS3TS4TS5TS6 TS7 Exposing systems to noise and microphone variations (TS2) improves performance

SUMMARY AND CONCLUSIONS WHAT HAVE WE LEARNED? Front end specific parameter tuning did not result in significant change in overall performance (MFA still outperforms QIO) Both QIO and MFA front ends handle convolution and additive noise better than ETSI baseline Both QIO and MFA front ends achieved ALV evaluation goal of improving performance by at least 25% relative over ETSI baseline WER is still high ( ~ 35%), further research on noise robust front end is needed

SUMMARY AND CONCLUSIONS AVAILABLE RESOURCES Speech Recognition Toolkits: compare front ends to standard approaches using a state of the art ASR toolkitSpeech Recognition Toolkits ETSI DSR Website: reports and front end standardsETSI DSR Website Aurora Project Website: recognition toolkit, multi-CPU scripts, database definitions, publications, and performance summary of the baseline MFCC front endAurora Project Website

SUMMARY AND CONCLUSIONS BRIEF BIBLIOGRAPHY N. Parihar, Performance Analysis of Advanced Front Ends, M.S. Dissertation, Mississippi State University, December 2003.Performance Analysis of Advanced Front Ends N. Parihar, J. Picone, D. Pearce, and H.G. Hirsch, “Performance Analysis of the Aurora Large Vocabulary Baseline System,” submitted to the Eurospeech 2003, Geneva, Switzerland, September 2003.Performance Analysis of the Aurora Large Vocabulary Baseline System N. Parihar and J. Picone, “DSR Front End LVCSR Evaluation - AU/384/02,” Aurora Working Group, European Telecommunications Standards Institute, December 06, 2002.DSR Front End LVCSR Evaluation - AU/384/02 D. Pearce, “Overview of Evaluation Criteria for Advanced Distributed Speech Recognition,” ETSI STQ-Aurora DSR Working Group, October 2001.Overview of Evaluation Criteria for Advanced Distributed Speech Recognition G. Hirsch, “Experimental Framework for the Performance Evaluation of Speech Recognition Front-ends in a Large Vocabulary Task,” ETSI STQ- Aurora DSR Working Group, December 2002.Experimental Framework for the Performance Evaluation of Speech Recognition Front-ends in a Large Vocabulary Task “ETSI ES v1.1.2 Distributed Speech Recognition; Front-end Feature Extraction Algorithm; Compression Algorithm,” ETSI, April 2000.ETSI ES v1.1.2 Distributed Speech Recognition; Front-end Feature Extraction Algorithm; Compression Algorithm

SUMMARY AND CONCLUSIONS BIOGRAPHY Naveen Parihar is a M.S. student in Electrical Engineering in the Department of Electrical and Computer Engineering at Mississippi State University. He currently leads the Core Speech Technology team developing a state-of-the-art public-domain speech recognition system. Mr. Parihar’s research interests lie in the development of discriminative algorithms for better acoustic modeling and feature extraction. Mr. Parihar is a student member of the IEEE. Joseph Picone is currently a Professor in the Department of Electrical and Computer Engineering at Mississippi State University, where he also directs the Institute for Signal and Information Processing. For the past 15 years he has been promoting open source speech technology. He has previously been employed by Texas Instruments and AT&T Bell Laboratories. Dr. Picone received his Ph.D. in Electrical Engineering from Illinois Institute of Technology in He is a Senior Member of the IEEE and a registered Professional Engineer.