1. D1 - 27/10/2015 The present document contains information that remains the property of France Telecom. The recipient’s acceptance of this document implies his or her acknowledgement of the confidential nature of its contents and his or her obligation not to reproduce, transmit to a third party, disclose or use for commercial purposes any of its contents whatsoever without France Telecom’s prior written agreement. Comparison of Narrowband and Wideband Speech codecs in noisy environnement Noel Chateau noel.chateau@francetelecom.com, Laetitia Gros laetitia.gros@francetelecom.com, noel.chateau@francetelecom.com laetitia.gros@francetelecom.com Catherine Quinquis catherine.quinquis@francetelecom.com, Jean Yves Monfort jeanyves.monfort@francetelecom.com catherine.quinquis@francetelecom.com jeanyves.monfort@francetelecom.com Workshop on Wideband Speech Quality in Terminals and Networks: Assessment and Prediction 8th and 9th June 2004 - Mainz, Germany

2. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D2 - 27/10/2015 Content 1. 3Gpp/VoIP context 2. Methodology 3. Description of Experiment 4. Results 5. Conclusion

3. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D3 - 27/10/2015 3GPP/VoIP Context  Rel 99 : March 2000  only narrow band speech and Circuit switched communications  Rel4: March 2001  Packet switched for download  Selection of wide band speech codec  Rel5: March 2002  Packet switched conversationnal communications  narrow band and wide band codecs  Rel6: September 2004  Characterisation of default codecs for Packet switched conversationnal communications

4. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D4 - 27/10/2015 3GPP/ Characterisation of default codecs for Packet switched conversationnal communications  1st phase  AMR codec in quiet and noisy environment  AMRWB codec in quiet and noisy environment  2nd phase  AMR and AMRWB versus legacy codecs in quiet environment  Further experiment  AMR and AMRWB versus legacy codecs in noisy environment : car noise

5. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D5 - 27/10/2015 Methodology  VoIP in UMTS  delay is one major issue –Interleaving due to radio environment –Buffering in decoder side to cope with jitter –IP transport –Packetisation in IP as well as on radio leg  Not only quality of codecs but acceptability of service  Conversation test  More accurate to assess acceptability of service

6. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D6 - 27/10/2015 Experiment: Description  Noisy environment : car noise @ 60dB Pa  Car noise is in one of the room, the other room is quiet  Codecs under test  AMR (6.7 and 12.2 modes)  AMRWB (12.65 and 15.85 modes)  G.723.1@ 6.3 kb/s  G.729@ 8 kb/s  G.711@ 64 kb/s  G.722 @ 64 kb/s  Environmental conditions  IP packet loss : 0% or 3%

7. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D7 - 27/10/2015 Experiment: Description PC 1 : VOIP Terminal Simulator Network Board A PC 2 : VOIP Terminal Simulator Network Board B PC 3 : Network Simulator Hub 1Hub 2

8. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D8 - 27/10/2015 Results on voice quality criterion  ANOVA  significant main effects: noise, PLP & codec  significant interactions: codec x PLP  Planned comparisons  Interaction band th x noise for PLP=0%: NS  Interaction Band th x PLP for Noise: NS  Tuckey tests show a superiority of some codecs on others for noisy and/or lossy conditions, but not systematic. Without noise and packet losses: no significant differences.  The superiority of WB Codecs is higher for noisy and packet-loss conditions, but there is no systematic benefit of WB in noise. FactorF noise *F(1,30) = 20.23 - p<0.001 packet loss percentage (PLP) *F(1,30) = 13.61 - p<0.001 codec *F(7,210) = 14.26 - p<0.001 noise x PLPF(1,30) = 0.68 - p=0.42 noise x codecF(7,210) = 1.25 - p=0.28 codec x PLP*F(7,210) = 2.3 - p<0.05 noise x codec x PLPF(7,210) = 0.76 - p=0.62 A N O V A

9. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D9 - 27/10/2015 Results on understanding criterion FactorF noise *F(1,30) = 68.11 - p<0.001 packet loss percentage (PLP) *F(1,30) = 11.07 - p<0.01 codec *F(7,210) = 20.3 - p<0.001 noise x PLPF(1,30) = 0.53 - p=0.47 noise x codecF(7,210) = 1.86 - p=0.08 codec x PLP*F(7,210) = 2.09 - p<0.05 noise x codec x PLPF(7,210) = 1.88 - p=0.07  ANOVA  significant main effects: noise, PLP & codec  significant interactions: codec x PLP  Planned comparisons  Interaction band th x noise for PLP = 0%: Signif.  Interaction Band th x PLP for Noise: NS  Tuckey tests show an almost systematic superiority of WB and G.711 codecs on NB codecs (except AMR NB 12.2 kbs) for noisy and/or lossy conditions. Without noise and packet losses: no significant differences.  Considered globally, WB and G.711 codecs better resist to noise for PLP=0%. A N O V A

10. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D10 - 27/10/2015 Results on interaction criterion FactorF noise *F(1,30) = 24.58 - p<0.001 packet loss percentage (PLP) *F(1,30) = 12.66 - p<0.001 codec *F(7,210) = 6.97 - p<0.001 noise x PLPF(1,30) = 0.3 - p=0.59 noise x codecF(7,210) = 2.67 - p<0.05 codec x PLP*F(7,210) = 0.63 - p=0.73 noise x codec x PLPF(7,210) = 0.96 - p=0.46  ANOVA  significant main effects: noise, PLP & codec  significant interactions: codec x noise  Planned comparisons  Interaction band th x noise for PLP = 0%: Signif.  Interaction Band th x PLP for Noise: NS  Tuckey tests show only a few significant differences for noisy conditions only.  As for the understandy criterion, considered globally, WB codecs better resist to noise for PLP=0%. A N O V A

11. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D11 - 27/10/2015 Results on default perception criterion FactorF noise *F(1,30) = 29.5 - p<0.001 packet loss percentage (PLP) *F(1,30) = 11.95 - p<0.01 codec *F(7,210) = 6.48 - p<0.001 noise x PLPF(1,30) = 0.21 - p=0.65 noise x codecF(7,210) = 1.67 - p=0.12 codec x PLP*F(7,210) = 2.47 - p<0.05 noise x codec x PLPF(7,210) = 0.82 - p=0.57 A N O V A  ANOVA  significant main effects: noise, PLP & codec  significant interactions: codec x PLP  Planned comparisons  Interaction band th x noise for PLP = 0%: NS  Interaction Band th x PLP for Noise: NS  Tuckey tests show no significant differences between codecs for any condition.  The default perception criterion does not seem relevant to detect a possible advantage of WB codecs in noise.

12. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D12 - 27/10/2015 Results on global quality criterion FactorF noise *F(1,30) = 57.37 - p<0.001 packet loss percentage (PLP) *F(1,30) = 14.51 - p<0.001 codec *F(7,210) = 13.54 - p<0.001 noise x PLPF(1,30) = 0.29 - p=0.59 noise x codecF(7,210) = 1.28 - p=0.26 codec x PLP*F(7,210) = 1.68 - p=0.11 noise x codec x PLPF(7,210) = 0.4 - p=0.9 A N O V A  ANOVA  significant main effects: noise, PLP & codec  significant interactions: none  Planned comparisons  Interaction band th x noise for PLP = 0%: NS  Interaction Band th x PLP for Noise: NS  Tuckey tests show an almost systematic superiority of WB and G.711 codecs on NB codecs (except AMR NB 12.2 kbs) for noisy and/or lossy conditions. Without noise and packet losses: no significant differences.  As for the voice-quality criterion, the superiority of WB Codecs is higher for noisy and packet-loss conditions, but there is no systematic benefit of WB in noise.

13. France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D13 - 27/10/2015 Summary of subjective tests results  ANOVA is not useful for determining if WB codecs are significantly better than NB codecs in noisy environment but show that noise was the strongest experimental factor that affected the subjective data.  Planned comparisons and Tuckey tests show that:  the MOS differences between WB and NB are significantly larger in noisy conditions than in silent conditions for no packet losses, for "Understanding" and "Interactivity" criteria.  the MOS differences between WB and NB are often but not systematically larger in noisy and/or 3% packet-loss conditions than in the silent and 0 % packet-loss condition, for the "Voice quality" and "Global quality" criteria.  the differences between WB and NB is independant on the presence of noise and packet loss for the "Default perception" criterion.  It can be concluded that without packet loss, in noise, the advantage of WB compared to NB is that it enhances the comprehension and the interaction with the other partner, but not voice and global quality, since these criteria are highly affected by the presence of noise.