Objective and Subjective Degradations of Transcoded Voice for Heterogeneous Radio Networks Interoperability Ľubica Blašková 1, Jan Holub 1, Michael Street 2, Filip Szczucki 2 and Ondřej Tomíška 1 1 FEE CTU, Prague 2 NATO C3 Agency, The Hague

2 Presentation Outlines Voice Transcoding – Issue of Modern Heterogenous Networks Voice Transcoding – Issue of Modern Heterogenous Networks Speech Transmission Quality Measurements Speech Transmission Quality Measurements Experiments Performed Experiments Performed Results Results Conclusions Conclusions

3 Voice Transcoding I Effective voice communications will remains a key service for those operating in a tactical environment Effective voice communications will remains a key service for those operating in a tactical environment Multinational operations routinely require different tactical communication systems from different nations to connect together or for wired networks to connect to wireless sub-systems Multinational operations routinely require different tactical communication systems from different nations to connect together or for wired networks to connect to wireless sub-systems Different networks apply differing voice encoding methods to the voice signal Different networks apply differing voice encoding methods to the voice signal It is recognised that use of multiple voice coders in series degrades the quality and intelligibility of the resulting voice It is recognised that use of multiple voice coders in series degrades the quality and intelligibility of the resulting voice

4 Voice Transcoding II For complex telecommunication chains as appear today, multiple voice coding in one communication direction occurs, examples: For complex telecommunication chains as appear today, multiple voice coding in one communication direction occurs, examples:  GSM-to-GSM call: GSM (FR/EFR/HR) – G.711 – GSM (FR/EFR/HR)  GSM-to-DECT call: GSM-G.711-DECT  UMTS-to-PSTN: AMR-(G.729)-G.711  Skype-to-GSM (typ.): Skype-G.729-GSM  etc.

5 Voice Transcoding III For both ad-hoc and permanently interoperating (special) networks, even more coder types must be taken into account: TETRA (ACELP) STANAG 4591 (MELPe) Problem: the lower bit-rate, the higher risk the coder will not interoperate satisfactorily

6 Speech Transmission Quality Perceived connection quality is influenced by many transmission impairments (delay, echo, various kinds of noise, speech (de)coding distortions and artifacts, temporal and amplitude clipping,...), assessed and measured in MOS (Mean Opinion Score) Perceived connection quality is influenced by many transmission impairments (delay, echo, various kinds of noise, speech (de)coding distortions and artifacts, temporal and amplitude clipping,...), assessed and measured in MOS (Mean Opinion Score)  5 Excellent  4 Good  3 Fair  2 Poor  1 Bad Listening / Conversational Tests (ITU-T P.800) Listening / Conversational Tests (ITU-T P.800) Intrusive Algorithmic Methods (P.862 PESQ) Intrusive Algorithmic Methods (P.862 PESQ)  dedicated test call must be established Non-intrusive Algorithmic Methods (P.563 3SQM) Non-intrusive Algorithmic Methods (P.563 3SQM)  real voice samples acquired on real connections are processed Estimation Algorithmic Methods - jitter, delay, packet loss etc. mapped to MOS (P.564, E-model) Estimation Algorithmic Methods - jitter, delay, packet loss etc. mapped to MOS (P.564, E-model)

7 Experiments Performed Speech database recording No background noise, Hoth +10dB SNR No background noise, Hoth +10dB SNR 2 male, 2 female speakers 2 male, 2 female speakers ACELP, MELPe, G.729, GSM FR coding (different typical combinations) ACELP, MELPe, G.729, GSM FR coding (different typical combinations) 5 recordings per combination per speaker 5 recordings per combination per speaker

8 Subjective Testing ITU-T P.800 methodology ITU-T P.800 methodology 38 untrained listeners 38 untrained listeners listening chamber <190 ms, <10 dB SPL (A) listening chamber <190 ms, <10 dB SPL (A) Results shown for „no noise“ condition Results shown for „no noise“ condition TechnologyMOS-LQSnCI95% ACELP4,250,166 MELPe2,210,184 GSM3,640,198 G.7294,000,188 ACELP-MELPe1,300,121 MELPe-ACELP2,890,199 ACELP- GSM3,680,199 GSM-ACELP3,850,191 ACELP- G.7294,220,204 G.729-ACELP3,630,203 MELPe-G.7293,440,200 G.729-MELPe2,460,168 MELPe-GSM3,080,197 GSM-MELPe2,330,189 MELPe–G.729-ACELP3,130,203 ACELP-G.729-MELPe1,750,182

9 Objective Testing (PESQ-LQ, 3SQM)

10 Objective Testing II (PESQ-LQ, left: male voices, right: female voices)

11 Results PESQ: P P.862.1, regressed)3SQM: P.563 Correlation0,8360,370 Maximum pos. difference1,0603,288 Maximum neg. difference-1,550-2,722 RMSE0,5601,072

12 Conclusions: All tandem setups perform with decreased speech transmission quality Always both directions (coders “A-to-B” and “B-to-A”) must be tested as the results can differ significantly Neither PESQ-LQ neither 3SQM can be used reliably for objective voice QoS monitoring in case of multiple coder tandeming where at least one low bit-rate coder is used. However, PESQ-LQ after proper regression shows at least reasonable correlation with subjective data (0.84) In our experiment, both male and female transmitted voices were subjectively evaluated almost equally Objective methods underestimate MOS scores for female voice transmissions

Thank you for your attention !