1. Buenos Aires - Argentina - 24 - 25 July 2014 1 Equipment for Testing Mobile QoS Joachim Pomy Consultant@joachimpomy.de OPTICOM, Germany ITU Workshop on “Monitoring and Benchmarking of QoS and QoE of Multimedia Services in Mobile Networks” (Buenos Aires, Argentina, 24-25 July 2014)

2. Buenos Aires - Argentina - 24 - 25 July 2014 2  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

3. Buenos Aires - Argentina - 24 - 25 July 2014 3 Measurement of QoS in Mobile  Measured from the perspective of an expert  Assessing the network to see if it delivers the quality that the network planner targeted (cf. Rec. ITU-T G.1000)  Tools and Methods for Operation and Maintenance  Protocol analysers are connected to BTSs, BSCs, and MSCs for a period of time to check for problems in the cellular network  Drive test (also walk tests) are used for monitoring operationally achieved QoS

4. Buenos Aires - Argentina - 24 - 25 July 2014 4 Measurement of QoS in Mobile  Drive tests allow the mobile network to be tested through the use of a team of people who take the role of users  Rate the QoS of the network  Not applicable to the entire network  Always a statistical sample  In Operation and Maintenance Centres (OMCs), counters are used for various events  Providing the network operator with vendor specific information on the state and quality of the network  Also, customer complaints are a vital source of feedback on the QoS, and must not be ignored

5. Buenos Aires - Argentina - 24 - 25 July 2014 5 End-to-end QoS Testing in Mobile  Minimum requirement for QoS measurement equipment to ensure that  for all QoS parameters  trigger points are well defined  values are determined following wqell defined procedures  Test-equipment fulfilling the specified minimum requirements  Will allow performing the proposed measurements  In a reliable and reproducible way

6. Buenos Aires - Argentina - 24 - 25 July 2014 6 Parameters are based on Field Measurements  Measurements from customers point of view  Full End-to-End perspective, taking into account needs of testing  Assumed end customer can handle his equipment and services to use  Operability is not evaluated at this time  For the purpose of measurement it is assumed that  service is not barred for any reason  routing is defined correctly without errors  target subscriber equipment is ready to process service request  speech and video quality values measured – Only to be applied to calls ended successfully – In order to receive proper statistical analysis

7. Buenos Aires - Argentina - 24 - 25 July 2014 7 Not yet any ITU Standards in Place  ETSI TS 102 250 part 4  Speech and multimedia Transmission Quality (STQ); QoS aspects for popular services in GSM and 3G networks; Part 4: Requirements for Quality of Service measurement equipment  globally recognized as "the requirement" in the area  extremely compehensive  detailed instructions

8. Buenos Aires - Argentina - 24 - 25 July 2014 8  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

9. Buenos Aires - Argentina - 24 - 25 July 2014 9 Test Scenarios  Test scenarios need to distinguish the following principal user cases  User-to-user services (typically telephony)  Store-and-forward services (e.g. SMS)  Information services (e.g. accessing the internet or FTP download)  Push services (sender initiated transfer)

10. Buenos Aires - Argentina - 24 - 25 July 2014 10  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

11. Buenos Aires - Argentina - 24 - 25 July 2014 11 Considerations on trigger points  It can be assumed that  any feasible test equipment will contain some kind of communication terminal (UE) which may be a special type (e.g. a Trace Phone)  or a standard UE  Also, it can be assumed that each such device will provide information from different communication layers  from Application Layer (close to the user interface)  down to lower layers, e.g. operating-system events, TCP/IP layer, or Layer 3 signalling information, which is used as trigger points for QOS PARAMETERS processing  Typically AT commands can be used in majority of cases  Action is typically triggered  by some emulated user action  causing some action on the air interface

12. Buenos Aires - Argentina - 24 - 25 July 2014 12 Considerations on trigger points  Process of event propagation is deterministic  allowing for kind of mapping between layers  associated with communication and processing delay in each stage.  Choice of the layer for trigger point information retrieval  determines the view expressed in a QOS PARAMETER  choosing lower-level events ( Layer 3) gives more network-centric view  events on higher levels to produce views more user-related  for same QoS PARAMETER source layer for events used as trigger points should be the same  In benchmarking, for all all networks under test  use same type of UE, and QOS PARAMETERS  use trigger points from the same layer.  Changing the source layer for a given trigger point  changes in QOS PARAMETERS should be expected  calibration measurements required  to assess influence on QOS PARAMETERS both quantitatively and qualitatively.

13. Buenos Aires - Argentina - 24 - 25 July 2014 13  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

14. Buenos Aires - Argentina - 24 - 25 July 2014 14 Mobile QoS Test-equipment (MQT)  Geographical data collection during a mobile measurement  Position, speed and heading  Taken from GPS whenever possible.  If no GPS available use other tools  e.g. navigation on a geo-referenced bitmap

15. Buenos Aires - Argentina - 24 - 25 July 2014 15  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

16. Buenos Aires - Argentina - 24 - 25 July 2014 16 Telephony Voice  Means to generate necessary calls or to answer automatically incoming calls  Received speech sample is measured  via the analogue output of the User Equipment  hardware setup to include necessary electrical adaptation  Hardware setup of such a system is essential for the correctness of the measured QoS Parameters  Includes a UE specific part for the electrical adaptation  audio output of handset to input of the soundcard

17. Buenos Aires - Argentina - 24 - 25 July 2014 17 Video Telephony  Means to generate necessary calls or to answer automatically incoming calls  Capability to provide the following measurement methods.  Video Telephony using the User Equipments application; or  Video Telephony using a Video Telephony Stack on the PC.

18. Buenos Aires - Argentina - 24 - 25 July 2014 18 MQT for FTP  FTP client to support the following functions:  Active / Passive Mode  Common Firewall support  Downloading from subdirectories.  Uploading to subdirectories

19. Buenos Aires - Argentina - 24 - 25 July 2014 19 Mobile based measurement equipment  Functionalities of the MQT can also be realised on a single mobile phone  Different types of QoS tests can run  Test results can be logged for post processing  Mobile based measurement equipment may be controlled  by an application, remotely or  may be operated by a human

20. Buenos Aires - Argentina - 24 - 25 July 2014 20  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

21. Buenos Aires - Argentina - 24 - 25 July 2014 21 Practical Hints  Which Mobile Phone to use ?  should be one offered in the shop ?  should be one used by new customers ?  should be one hipe to young people ?  should be one with good QoS ?  BUT  it MUST be a so-called Trace Mobile  ASK YOUR PHONE PROVIDER !!!  they are RARE !!!!

22. Buenos Aires - Argentina - 24 - 25 July 2014 22  Equipment for Testing Mobile QoS  Measurements  Test Scenarios  Trigger Points  Test Equipment  Testing for various Services  Practical Hints  POLQA - Voice Quality - Hot Topic ITRs: Setting the stage for a connected world [Content]

23. Buenos Aires - Argentina - 24 - 25 July 2014 23 Hot Topic: POLQA™ - Rec. P.863 (1)

24. Buenos Aires - Argentina - 24 - 25 July 2014 24 POLQA™ - Rec. P.863 (2)  New POLQA standard winner of rigorous technical competition  Carried out by ITU-T to define technology update for PESQ/P.862  PESQ was state-of-the-art in almost any telecom service globally  POLQA - “Perceptual Objective Listening Quality Assessment” - offers new level of benchmarking capability  to determine the voice quality of mobile network services

25. Buenos Aires - Argentina - 24 - 25 July 2014 25 POLQA™ - Rec. P.863 (3)  POLQA unified solution by OPTICOM, SwissQual and TNOOPTICOMSwissQual TNO  was selected as the winning approach  the underlying algorithms were subjected to extensive tests  over 45,000 speech samples from a broad range of targeted usage scenarios  wide array of tests covers the entire range of today’s telephony quality  over the last ten years test and measurement vendors have widely adopted ITU-T Recommendation P.862/PESQ (“Perceptual Evaluation of Speech Quality”) and there are now more than 20.000 installations worldwide  recently developed speech codecs for 3G and 4G, and new techniques for delivering an enhanced voice quality experience have generated strong demand for an update to this industry standard  POLQA is the answer the industry has been looking for

26. Buenos Aires - Argentina - 24 - 25 July 2014 26 POLQA™ - Rec. P.863 (4)  POLQA coalition formed by OPTICOM, SwissQual and TNO is the outcome of the competitive standardization process run by the ITU-T  An expert group within ITU-T spent more than four years evaluating various candidate algorithms to find a suitable successor to older standards  An enormous amount of effort has gone into ensuring POLQA works quickly and accurately  For example, more than 100,000 samples of mobile and VoIP phone calls were used to train and optimize the POLQA algorithm  POLQA is superior to existing standards, and clearly has overcome all known issues and limitations of PESQ

27. Buenos Aires - Argentina - 24 - 25 July 2014 27 POLQA™ - Rec. P.863 (5)  The limitations of existing standards that are now addressed by POLQA  CDMA  Chinese 3G TD-SCDMA  POLQA will also offer immediate, strong support for testing of new wideband 4G/LTE networks delivering HD-quality voice services  Tests carried out during the POLQA evaluation included future technologies such as  Unified Communications  Next Gen Networks  4G/LTE  HD Voice, i.e. "wide-band" and "super-wide-band"

28. Buenos Aires - Argentina - 24 - 25 July 2014 28 POLQA™ - Rec. P.863 (6)  More information on POLQA can be found in the technical white paper “POLQA: The Next Generation in Voice Quality Testing” available for download from http://www.polqa.info.

29. Buenos Aires - Argentina - 24 - 25 July 2014 29

30. Buenos Aires - Argentina - 24 - 25 July 2014 30 POLQA Introduction - (c) OPTICOM GmbH 201030 PESQPOLQA Acoustic measurements  Not easy Correct scoring with high background noise  AMR vs EVRC codec comparison  Representative scoring of reference signals  Effects of speech level in samples  Narrowband (300Hz -3400Hz) Wideband (100Hz-7000Hz) Use SWB Superwideband, SWB (50Hz – 14000Hz)  Linear Frequency distortion sensitivity 

31. Buenos Aires - Argentina - 24 - 25 July 2014 31 What we Perceive … POLQA Introduction - (c) OPTICOM GmbH 201031 In a subjective ACR experiment POLQA, PESQ and human beings perceive the following distortions: FactorHumanPOLQAPESQ Level too high or too lowXXno Strong linear filteringXXno Noise in the reference signalXXno High timbre in referenceXXno Level variationXXpoor SWB noise on NB/WB signalXXno

32. Buenos Aires - Argentina - 24 - 25 July 2014 32 POLQA Introduction - (c) OPTICOM GmbH 201032 Performance Validation The ITU has validated POLQA on: Languages included in the POLQA validation: German Swiss German Italian, Japanese, Swedish American English and British English Chinese (Mandarin), Czech, Dutch, French, 47000 file pairs across 64 subjective experiments

33. Buenos Aires - Argentina - 24 - 25 July 2014 33 Any questions ? Contact: Consultant@joachimpomy.de