1. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders

2. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 2 of 22 Evaluation Basics Purpose –Research –Vocoder Development –Vocoder Characterization –Selection –Validation Types of Conditions of Interest –Baseline –Acoustic Background Noise –Transmission Channel Impairments –Talker Variability –Signal Levels –System Tandems –Digital Circuit Multiplication Systems

3. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 3 of 22 Subjective Testing - Control of Variables Laboratory Factors –Listening Environment; Audio & Electronics Source/Processed Recording Factors –Speech Material Factors Linguistic and Phonetic Talker Factors Transducer Selection –Audio and Sampled Bandwidth Factors –Acoustic Noise Material and Speech + Noise Method Listener Factors Presentation Factors –Blocking, Order and Balance –Audio Level and Sidetone

4. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 4 of 22 Associated Issues User Population and Face Validity Context –Range of Candidate Systems –Reference and Calibration Systems Listen Only vs. Two-Way Methods –Delay –Asymmetric Transmission Channels –VoIP Speech material –Speech Sample length re impairment distribution –Uniqueness, Amount Available –Type

5. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 5 of 22 Associated Issues (cont.) Speech material (by increasing contextual content) –Types Scripted –Sounds –Words »rhyming, CVC, etc –Sentences »meaningful, nonsense, semantically anomalous, etc –Connected sentences –Scripts Scenario based –Representative of application? –Informational or Familiar –Information flow (balanced?, directional?) Task Based Open

6. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 6 of 22 Performance Characteristics & Test Methodology Quality –Diagnostic Acceptability Measure - DAM Voiers ICASSP77 –Category Rating Tests - ACR (MOS); DCR (DMOS) CCR (CMOS) ITU-T P.800: P.830 ITU HANDBOOK ON TELEPHONOMETRY IEEE Recommended Practices for Speech Quality Measures 1969 –Paired Comparison A/B Tests David, H.A, “The Method of Paired Comparison,” Oxford –Multi Stimulus Test with Hidden Reference and Anchor - MUSHRA ITU-R BS.1534-1 –Speech Communication Systems with Noise Suppression Algorithms ITU-T P.NSA

7. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 7 of 22 Performance Characteristics & Test Methodology Speaker Recognizability –NRL Speaker Recognition Test (speakers unknown) Schmidt-Nielsen SCW95, ICASSP96, JASA 1985 –TNO Speaker Recognition Test (speakers known) Steeneken & Leeuwen 1997 Language Dependency –SRT-LD Wijngaarden SCW02, EuroSpeech01, Ph.D. Dissertation 2003 Conservation of Stress State Characteristics

8. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 8 of 22 Performance Characteristics & Test Methodology Communicability –Conversation Opinion Tests ITU-T P.800 –Conversational & Third Party Listen Only Tests ITU-T P.832, P-581 (HATS) –Continuous Quality Evaluation Method - ECQ ITU-T P.PAC –Arcon Communicability Exercise - ACE Tardelli ICASSP96, NAS-NRC CHABA Symposium 1995 –TNO Communicability Test Wijngaarden EuroSpeech01

9. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 9 of 22 Performance Characteristics & Test Methodology Intelligibility –Modified Rhyme Test - MRT ANSI S3.2-1989; House 1965; Kruel 1968 –Diagnostic Rhyme Test - DRT ANSI S3.2-1989; Voiers 1973, 1987 –Consonant-Vowel-Consonant Test - CVC (AI Basis) Fletcher ATT 1920s, JASA 1950; Allen 1994, ICASSP02; Steeneken 1992 –Speech Reception Threshold - SRT Plomp & Mimpen 1979; Wijngaarden & Steeneken EuroSpeech99 –International Civil Aviation Org. Spelling Alphabet - ICAO Moser & Dreher 1955; Schmidt-Nielson NRL R9035 1987, R9174 1988 –INTELTRANS -(CVC, HATS) CELAR France MOD; J.C. Lafon 1958, 1964, 1968

10. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 10 of 22 Intelligibility Measures vs. Information Webster, 1979 ANSI S3.5-1969

11. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 11 of 22 Evaluation Decisions Purpose Types of Conditions Performance Characteristics of Importance Choice of Test Methodologies Development of Test Plan Selection Criteria if Selection Test

12. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 12 of 22 Vocoder Development Issues Application –Commercial –Strategic –Tactical Diagnostic Information –Intelligibility –Quality –Communicability

13. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 13 of 22 Low-Rate Vocoder for Tactical Use Harsh Acoustic Noise Environments Physical and Jamming Channel Issues LPI / LPD Intelligibility Talker Recognizability Conserve Stress State of Talker Audio Bandwidth Delay Size - Weight -Power

14. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 14 of 22 Narrowband Low-Rate Vocoder Intelligibility Intelligibility results for current low-rate military vocoders in acoustic background noise

15. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 15 of 22 Effects of Current Noise Preprocessors Intelligibility - DRT Quality - DAM

16. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 16 of 22 Road Map to Improved DRT Intelligibility Inherent Distinctive Features –Jacobson, Fant, and Halle 1952; Miller & Nicely, 1955 DRT Attributes –Voiers 1973, 1987 DRT Attributes : Distinctive Features :Acoustic Correlates –Voiers, Benchmark Papers in Acoustics, V11 1977 Diagnostic Capabilities of the DRT Cook Book

17. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 17 of 22 Inherent Distinctive Features (Jacobson, Fant, and Halle 1952) Fundamental Source Features VocalicNon-Vocalic ConsonantalNon-Consonantal Secondary Consonant Features –Envelope Features ContinuantInterrupted CheckedUnchecked StridentMellow –Supplementary Source VoicedVoiceless Resonant Features CompactDiffuse –Tonality Features GraveAcute FlatPlain SharpPlain TenseLax –Supplementary Resonator NasalOral

18. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 18 of 22 DRT Attributes

19. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 19 of 22 DRT Attributes : Distinctive Features : Acoustic Correlates DRT AttributesJFH Distinctive Features Voicing Voiced/Voiceless harmonic content, energy at concentration at LF, long duration, low peak power NasalityNasal/Oral nasal formants in regions of 200, 800 and 2400 Hz SustentionContinuant/Interrupted gradual onset > 130 msec, low level noise in MF to HF SibilationStrident/Mellow sustained HF noise of relatively high intensity CompactnessCompact/Diffuse LF spectral shape, low loci of 2nd and 3rd formants, dynamics of formant transitions GravenessGrave/Acute HF spectral shape, separation of 2nd and 3rd formants, dynamics of 2nd and 3rd formant

20. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 20 of 22 Diagnostic Capabilities of the DRT Talkers –Male : Female Attribute State –Present : Absent Attribute Bias Sub-Attribute Scores Characteristic Attribute Profile Empirical Studies

21. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 21 of 22 Cook Book for Improved Intelligibility

22. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 22 of 22 Pitfalls in Subjective Evaluation Measured Intelligibility vs. Real World Intelligibility –NAS-NRC CHABA 1989 Symposium Removal of Noise From Noise-Degraded Speech Signals –Vocoder Tuned to DRT Words –Vocoder based on “scripted word” characteristics that are not applicable to conversational speech. Danger of "self evaluation" by Vocoder Developers –Tardelli, ICASSP96, DAM vs MOS Study 1996

23. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 23 of 22 DAM vs. MOS Study A Systematic Investigation of the Mean Opinion Score (MOS) and the Diagnostic Acceptability Measure (DAM) for Use in the Selection of Digital Speech Compression Algorithms ARCON Corp. 1996 Available in DRAFT form at http://www.arcon.com/dld.html

24. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 24 of 22 P.NSA and WHY ETSI/3GPP AMR-NS 1999 Exp.. 3 MMOS w/ Multi-Dimensional Question You will hear speech samples reproduced in a telephone handset. Every sample consists of four short unconnected sentences in a noise environment. Your task is to indicate your opinion of the overall sound quality with respect to any unnatural sound in the sample. Please make your judgement of the sample considering unnatural sound during the complete sample. Resulted in Bimodal Decision P.NSA Subjective test methodology for evaluating speech communication systems that include noise suppression algorithm Summary This document proposes a methodology for evaluating the subjective quality of speech in noise and particularly appropriate for the evaluation of noise suppression algorithms. The proposed methodology uses separate rating scales to independently estimate the subjective quality of the Speech Signal alone, the Background Noise alone, and Overall Quality. ITU-T SG12/Q7 SQEG, Primarily Dynastat and FT

25. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 25 of 22 INTELTRANS Testbed

26. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 26 of 22 DRT Characteristic Attribute Profile

27. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 27 of 22 Empirical Study of DRT Attributes vs. SNR Band Limited Gaussian Noise Voiers, JASA 1973

28. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 28 of 22 Scripted Material - DRT Word Lists MOOTorBOOTVoicing SHEETorCHEATNasality JABorGABSustention POTorTOTSibilation GHOSTorBOASTGraveness DINTorTINTCompactness

29. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 29 of 22 Scripted Material - CVC Nonsense Words MIG(RAINE) COS(T) HAYM DIT TOUP(EE) BACH POD(IUM) SEM(I) LAL:PAL REAS(ON) REET:BEET SAYZ:DAYS BOD(Y) KOOM LEP(ER) PONE:BONE HIES DACK:BACK TEEG:LEAGUE MAHL

30. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 30 of 22 Problems with CVC Test Implementation CVC Corpus Balance –Talker by Word by Environment –Word by Distinctive Feature by Lexicon Regional Dialectic Differences –New England Spoken “COT” = “CAUGHT” Perception Midwest “CART” = “COT” Test Design –Uniqueness for Talker By Word by Environment by Process –Balance Across Distinctive Feature by Process –Balance Across Subject by Stimulus –Sufficient Subjects for Reasonable Resolution

31. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 31 of 22 Diagnostic Capabilities of INTELTRANS

32. ARCON Corporation J.D. Tardelli - jdt@arcon.com The Utilization of Subjective Evaluation in the Development of Vocoders 11/2003 Slide 32 of 22 Diagnostic Capabilities of INTELTRANS (cont.)