1. ETSI STQ, Taiwan Workshop, February 13th, 2006 1 Recent improvements in transmission quality assessment : Background noise transmission Results of STF 273, STF 294 Jean-Yves Monfort, France Telecom ETSI TC STQ Chairman

2. ETSI STQ, Taiwan Workshop, February 13th, 2006 2 STFs  STF : Special Task Force  Founded by ETSI (STF 273) and e-Europe (STF 294)  Several Experts involved in a project  new inputs in standards  STF 273 Speech quality performance in the presence of background noise  A noise data base, freely available  EG 202 396-1  STF 294 Improving the quality of eEurope wideband speech applications by developing a standardised performance testing and evaluation methodology for background noise transmission  EG 202 396-2

3. ETSI STQ, Taiwan Workshop, February 13th, 2006 3 STF 273 Speech quality performance in the presence of background noise Objectives : a close-to-original recording and reproduction system The background noise recording technique should be – Easy to use – Easy to calibrate – Capable of wideband recording – Available at reasonable costs – Mostly compatible with existing standards and procedures used in telecommunications testing

4. ETSI STQ, Taiwan Workshop, February 13th, 2006 4 STF 273 Speech quality performance in the presence of background noise Objectives : a close-to-original recording and reproduction system The background noise simulation (reproduction) arrangement should be: – easy to set up – usable in typical office type room environments without special acoustical treatment – provide a mostly realistic background noise simulation for all typical background noises occurring in telecommunication applications – easy to calibrate – mostly insensitive to the positioning of (test) objects in the simulated sound field – applicable to all typical terminals used in telecommunication – available at reasonable costs

5. ETSI STQ, Taiwan Workshop, February 13th, 2006 5 STF 273 : The Recording Setup  Recordings for the setup in office rooms with  Artificial heads (ITU-T Rec. P.58) or  Binaural probe microphone (MIRE technique)

6. ETSI STQ, Taiwan Workshop, February 13th, 2006 6 STF 273 :The Background Noise Simulation Setup: 4.1 - Technique

7. ETSI STQ, Taiwan Workshop, February 13th, 2006 7 STF 273 : Comparison of Terminal Performance: Simulated vs. Real Background Noise Headset Mobile phone Desktop HFT Wideband conference system Green – original Red - simulated

8. ETSI STQ, Taiwan Workshop, February 13th, 2006 8 STF 273  Background noise Recording in cars  Instead of an artificial head, two methods for sound pick-up and equalization : the microphone of the terminal under test, a pair of cardioid microphones  Background noise Reproduction in cars

9. ETSI STQ, Taiwan Workshop, February 13th, 2006 9 STF 294 Improving the quality of eEurope wideband speech applications by developing a standardised performance testing and evaluation methodology for background noise transmission Objectives : to develop a quality model for wideband speech for new multimedia services (e-health, e- learning, e-bussiness,…) Program in two phases  Phase 1: Network simulation, Subjective tests (in french and csez langages) on realistic scenarios with background noise. Finalized in end 2006; EG 202 396-2 available  Phase 2 : Development and validation of an objective meyhodology for background noise transmission. Under Progress  August 2006

10. ETSI STQ, Taiwan Workshop, February 13th, 2006 10 Why Background noise is a problem in mostly all situations and conditions? Both networks and terminals involved  In mobile communication background noise is almost always present;  Problem emphasized due to the acoustical construction of mobile terminals.  Situation even worse for mobile hands-free terminals and mobile car hands-free terminals.  In multimedia conference and classroom systems hands-free type terminal with quite high distance between microphone und speaker.  Computer terminals (e.g. in VoIP scenarios) often used as hands-free terminals  Background noise may be produced by the computer/handheld device itself but may also be picked up from the environment such as an open plan office.  Effects of transmission network impairments in packet based communication networks, Wireless LAN,…

11. ETSI STQ, Taiwan Workshop, February 13th, 2006 11 Scope of the work STF 294  The aim of the work is to find methods for background noise transmission quality evaluation for real life network scenarios with speech present at the near end and at the far end as well as during periods where no speech is present but only background noise is transmitted.  Steps : Setup and verification of a transmission network simulation environment using realistic network scenarios for laboratory use; Setup of a database containing the relevant transmission network models and traffic patterns that would be used for subjective and objective evaluation; Subjective tests forming the basis for objective evaluation procedures, using the background noise database developed in the currently active ETSI STF 273; Evaluation and validation of background noise transmission evaluation procedures.

12. ETSI STQ, Taiwan Workshop, February 13th, 2006 12 Phase 1 Tasks – References for the development of the new objective Method  Phase 1 (Leader France Telecom)  Task 1: Transmission network simulation: Telefonica A transmission network simulation environment needs to be setup, to provide comparable network characteristics and traffic patterns as found in conditions of the real transmission networks.  Task 2: Database for transmission networks and conditions evaluation/simulation Telefonica A database containing the typical and realistic transmission network scenarios and traffic patterns needs to be created. There are no risks expected in this task, the procedures are known by the experts in this field.  Task 3: Subjective tests, France Telecom (leader), Mesaqin Subjective test results providing the speech quality parameters in the background noise situation by the individual parameters are to be conducted. This task must be conducted by experts in the field of subjective testing.

13. ETSI STQ, Taiwan Workshop, February 13th, 2006 13 STF 294

14. ETSI STQ, Taiwan Workshop, February 13th, 2006 14 Task 1 : Transmission network simulation  Evaluation of simulation methodologies and tools  Choice of the best simulation methodologies and tools  Setup of simulation environment in the laboratories of the STF experts  Validation of the simulation environment  The transmission network simulation tool chosen by the experts of STF 294 is an IP network simulator (namely : NIST Net). The network simulation is supposed to provide comparable network characteristics and traffic patterns as found in conditions of the real transmission networks (in terms of packet loss, delay and delay variation distributions).

15. ETSI STQ, Taiwan Workshop, February 13th, 2006 15 Task 2 : Database for transmission networks and conditions evaluation/simulation  Definition of the different network scenarios and conditions to be recorded  Preparation of the simulation recordings  Editing and preparing the data base  Description of the data base (STQ document)

16. ETSI STQ, Taiwan Workshop, February 13th, 2006 16 Steps in Recording Process Speech sample database (4320 samples) Database of noise type/wideband terminal/network impairment combinations Speech samples with noise and terminal conditions (720 samples) Speech samples with background noise conditions (80 samples) Noise-free speech samples (8 samples) Step 1 :Clean speech sequences recording Step 2 :Noisy conditions adding Step 3 : Wideband terminal simulation Step 4 : Network simulation

17. ETSI STQ, Taiwan Workshop, February 13th, 2006 17 Recorded speech samples  For the recordings performed by ETSI STF 294 the following conditions have been used:  48 kHz (16 bit) sampling rate;  wave format; and  the active speech level has been equalized to -26 dBoV (see ITU-T Recommendation P.56 [24], no filtering applied).  Also for STF 294, the number of samples to have a representative database large enough for subjective and objective evaluation has been decided:  4 speakers (2 male, 2 female), 8 sentences each;  two languages: Czech and French.  The length of recordings varies between 24 and 73 s. They contain neutral sentences of 2-3s length separated by pauses. Speech activity factor varies between 30 and 60 %. This arrangement allows for easy later editing to any required format.  Simulated terminals :  Mobile Handset in the street  Hands-Free with PC  Hands-Free in car

18. ETSI STQ, Taiwan Workshop, February 13th, 2006 18 Noisy Conditions / Noisy signal Processing  Noisy Conditions  5 background noise conditions selected: cafeteria noise; office room noise; road noise; crossroads; and car noise (car hands ‑ free at 130 km/h).  Each combination of speech file plus background noise is recorded at two microphone ‑ loudspeaker positions: typical handset microphone position (with loudness ratings adjusted to 7 dB); and hands ‑ free microphone position (with loudness ratings adjusted to 11 dB).  Noisy signal Processing  Three parameters With and without VAD (Voice Activity Detection) 2 different noise reduction filters 2 different noise reduction levels

19. ETSI STQ, Taiwan Workshop, February 13th, 2006 19 Task 3 : Subjective tests  Preparation of the test material  Conducting experts tests  Evaluation of the experts test results  Preparing test material for naïve listeners tests  Conducting naïve listeners tests (2 languages : French and Czech)  Evaluation of the results  Statistical analysis  the test results

20. ETSI STQ, Taiwan Workshop, February 13th, 2006 20 Listening tests Method used : according to ITU-T Recommendation P.835 three questions : - perception and judgment of noise level reduction, - perception and judgment of possible distortion on speech signal, - overall perception of quality. All these questions must be answered on a discrete 5 level scale. France Telecom R&D Test Laboratory Mesaquin Test Lab

21. ETSI STQ, Taiwan Workshop, February 13th, 2006 21 Exemples of results (1)  MOS distribution and associated 95% CI per noise condition for Czech and French data

22. ETSI STQ, Taiwan Workshop, February 13th, 2006 22 Exemples of results (2)  MOS distribution and associated 95% CI per network condition for Czech and French data (NETWORK I: delay jitter 0ms. packet loss 0%. NETWORK II: delay jitter 10ms. packet loss 1%. NETWORK III: delay jitter 20ms. packet loss 3%)

23. ETSI STQ, Taiwan Workshop, February 13th, 2006 23 Exemples of results (3)  MOS distribution and associated 95% CI per NSA condition

24. ETSI STQ, Taiwan Workshop, February 13th, 2006 24 What do we also find in EG 202 396-2  Performance parameter descriptions (IP Transmission)  Delay (from codec, packetization, network,…)  Jitter  Packet loss  Parameter interaction and dependences  Wideband speech codecs  Characteristics  Descriptions

25. ETSI STQ, Taiwan Workshop, February 13th, 2006 25 What do we also find in EG 202 396-2  Wideband speech codecs  Characteristics Speech coding algorithm DTX (discontinuous transmission) Packet Loss Concealment (PLC) …  Description of some wideband codecs ITU-T G.722 AMR WB …

26. ETSI STQ, Taiwan Workshop, February 13th, 2006 26 Next Phase – Development, Validation and sharing of he new objective Method  Task 4: Objective method An objective method for the quantification of background noise transmission performance needs to be developed. New methods shall be developed and/or existing methods have to be adapted to the problem.  Task 5: Validation of the new method The validation has to be performed independently from the development of the background noise evaluation method based on unknown data acquired in task 3.  Task 6: Final standardization of the new method The final standard(s) are drafted and adapted to the results of the previous tasks. An STQ workshop is planned and realized.  Workshop in Praha, November 13-14, 2006

27. ETSI STQ, Taiwan Workshop, February 13th, 2006 27 Complementary informations  Visit the STF 273 and STF 294 Web sites  http://portal.etsi.org/STFs/STQ/STF273.asp http://portal.etsi.org/STFs/STQ/STF273.asp  http://portal.etsi.org/STFs/STQ/STF294.asp http://portal.etsi.org/STFs/STQ/STF294.asp  You will find :  ToRs of the STF  More informations on the work programme  Coordinates of the experts  Database, results,…

28. ETSI STQ, Taiwan Workshop, February 13th, 2006 28 Questions ?