1. TOWARDS QUALITY AND QOS ASSESSMENT FOR VOLTE IN ETSI TC INT chairman Giulio Maggiore - IMS World Forum 30th April 2014 © ETSI 2013. All rights reserved

2. Outline ETSI Technical Committee TC INT for Core Network and Interoperability Testing ETSI INT technical work specifications Quality and QOS Assessment for VoLTE in ETSI What’s next? 2 © ETSI 2013. All rights reserved

3. CORE NETWORK AND INTEROPERABILITY TESTING (INT) 3 © ETSI 2013. All rights reserved

4. TC INT (Core Network and Interoperability Testing) Member of EP E2NA In 2012 TC INT participated in the creation of the ETSI E2NA project contributing with its expertise in network testing as described in the E2NA ToRs approved by the Board#89. TC INT change of ToRs and name According to E2NA ToRs approved by Board#89 The modification of the ToRs in order to align its activities and contributions with EP E2NA. Change of name with Core Network and Interoperability Testing (INT) to better reflect the areas of activity. No change of acronym that will remain INT ETSI/BOARD(13)94_041 4

5. ETSI INT SPECIFICATIONS Interoperability, Performance, Security and Conformance 5 © ETSI 2013. All rights reserved

6. Production of test specifications Production of test specifications: (11) Interoperability (4) IMS/NGN Performance Benchmark (1) Security Production of IMS Conformance: (32) Test Suite Structure and Test Purposes (TSS&TP) (24) Abstract Test Suite (ATS) (24) Protocol Information Conformance Statement (PICS) (24) Protocol Implementation eXtra Information for Testing (PIXIT) INT full work programme 6 © ETSI 2013. All rights reserved

7. QUALITY AND QOS ASSESSMENT ETSI APPROACH 7 © ETSI 2013. All rights reserved

8. 8 Quality: the new approach Access Core APN Network CentricCustomer Centric Access Core APN ► Quality is the most important factor in client choice and churn. ► In order to offer the best user experience it’s necessary to shift in the quality monitoring: from a “Network Centric” point of view to “Customer Centric”

9. 9 Customer Centric monitoring requires new KPIs 9 LocalizedAccess layerOperative SystemDevice Network section Core Critical Indicators ► Moving from the network to the user centric point of view a new set of KPIs are required able to analyze the experience of the single user ► In order to monitor the e2e network chain the KPIs have to be technology/access independent Service KPI Immediate overview of the service Technical KPI bricks of service KPI monitor specific network elements

10. 2013 – VOLTE QOS ASSESSMENT TECHNOLOGY EVALUATION EVENT 10 © ETSI 2013. All rights reserved

11. VoLTE QoS Assessment event overview © ETSI 2013. All rights reserved 11 Scope: Technology evaluation Plugtests aiming at validating the different methodologies for end to end QoS Assessment for VoLTE described in ETSI TS 103 189 over a deployed LTE/IMS network. Dates: 18-20 November 2013 Venue: ETSI, Sophia Antipolis, FR Web: http://www.etsi.org/news-events/events/698-volte-qos-assessment-2013 http://www.etsi.org/news-events/events/698-volte-qos-assessment-2013 Participants: Supporter:

12. VoLTE QoS Assessment event scope © ETSI 2013. All rights reserved 12 End to end tests performed with several different UEs over a deployed LTE / IMS test network. 4 Test Setups Acoustic to acoustic Acoustic to electrical Electrical to acoustic Electrical to electrical Voice quality tests based on instrumental assessment of speech samples performed on different combinations of the acoustic and electrical interfaces which allows evaluating the different methodologies and setups. Parallel network performance evaluation

13. 2013 – VoLTE QoS Assessment Technology Evaluation event – test description TS 103 189 Sub-parameters delivered by the particular tests: End-to-end Tests based on Instrumental Assessment of Speech Samples (POLQA model deployed): Mean Opinion Score (MOS)-Listening Quality Objective Delay variation over time Difference in loudness level MOS stability End-to-end Tests based on Instrumental Assessment of Video Samples (PEVQ model deployed): DMOS (Difference Mean Opinion Score) Delay and Quality variation over time Jerkiness and Blockiness Blur, Chrominance and Luminance Temporal Distortion Frozen and Skipped Frames

14. 2013 – VoLTE QoS Assessment Technology Evaluation event – test description TS 103 189 Sub-parameters delivered by the particular tests: Other end-to-end tests in the voice channel: End-to-end frequency response Overall loudness rating End-to-end delay Quality of echo cancellation Double talk performance End-to-end Tests based on functional parameters: Telephony Call Setup Time Streaming Service Access Time Network Performance Parameters UNI-to-UNI delay over time UNI-to-UNI jitter UNI-to-UNI packet loss

15. 2013 – VoLTE QoS Assessment Technology Evaluation event – test description General description of tests defined in ETSI TS 103 189: QoS tests based on the ETSI INT specifications as described in the latest versions of: ETSI TS 186 011-1 (Test purposes for IMS NNI Interoperability) ETSI TS 186 011-2 (Test descriptions for IMS NNI Interoperability) ETSI TS 102 901 (IMS NNI interoperability test descriptions for RCS) ETSI TS 103 029 (IMS & EPC Interoperability). QoS tests divided into two parts: First one focusing on the instrumental assessment of speech and video quality by using real speech and video samples Second one dealing with functional parameters of voice and video over LTE connections and other parameters in the end-to-end voice channel On the top of it, selected network performance parameters listed in the TS 103 189 15 © ETSI 2013. All rights reserved

16. VoLTE QoS Assessment – Setup 1 Acoustic-to-Acoustic © ETSI 2013. All rights reserved 16

17. ITU-T Specifications for Quality measurements Recommendation ITU-T J.247: "Objective perceptual multimedia video quality measurement in the presence of a full reference". Recommendation ITU-T P.50: "Artificial voices". Recommendation ITU-T P.57: "Artificial ears". Recommendation ITU-T P.58: "Head and torso simulator for telephonometry". Recommendation ITU-T P.64: "Determination of sensitivity/frequency characteristics of local telephone systems". Recommendation ITU-T P.79: "Calculation of loudness ratings for telephone sets". Recommendation ITU-T P.340: "Transmission characteristics and speech quality parameters of hands-free terminals". Recommendation ITU-T P.380: "Electro-acoustic measurements on headsets". Recommendation ITU-T P.501: "Test signals for use in telephonometry". Recommendation ITU-T P.505: "One-view visualization of speech quality measurement results". Recommendation ITU-T P.502: "Objective test methods for speech communication systems using complex test signals". Recommendation ITU-T P.581: "Use of head and torso simulator (HATS) for hands-free terminal testing". Recommendation ITU-T P.800.1: "Mean Opinion Score (MOS) terminology".

18. VoLTE QoS Assessment – Setup 2 Acoustic to Electrical © ETSI 2013. All rights reserved 18

19. VoLTE QoS Assessment – Setup 3 Electrical to Acoustic © ETSI 2013. All rights reserved 19

20. VoLTE QoS Assessment – Setup 4 Electrical to Electrical © ETSI 2013. All rights reserved 20

21. MAIN OUTCOME OF THE VOLTE QOS ASSESSMENT AND NEXT STEPS… 21 © ETSI 2013. All rights reserved

22. Main event outcome Validation of 76% of the test cases in TS 103 189 A number of gaps and potential improvements in TS 103 189 have been identified A new work item has been open for the update of TS 103 189 with the new setup, acoustic-to-electrical identified during the event TS 103 189 has been published and is available on the ETSI portal 22 © ETSI 2013. All rights reserved

23. Maputo - Mozambique - 14 - 16 April 2014 23 Further steps with regard to ETSI TS 103 189 Verification of remaining voice quality test cases tbd Verification of video quality test cases (RTP-based video streaming): Test houses willing to do so are needed Extension of video quality test cases towards MPEG-DASH ITU-T SG12 Question 14 currently running a WI called P.NATS

24. 2013 – VoLTE QoS Assessment Technology Evaluation event – test results 24 © ETSI 2013. All rights reserved

25. 2013 – VoLTE QoS Assessment Technology Evaluation event – test results 25 © ETSI 2013. All rights reserved

26. Verification of video quality test cases (RTP-based video streaming) What we are looking for? What are the parameter we want to measure? What are the tools expected? © ETSI 2012. All rights reserved 26 Parameters Covered by TS 183.189 test tools focused on video quality Prototipization and validation

27. Customers perception KPIs © ETSI 2012. All rights reserved 27 Streaming (HTTP YouTube sessions included) Number of streaming sessions Average duration of streaming sessions Average Streaming Throughput Peak Streaming Throughput Setup Streaming Time Streaming Dropped Sessions Streaming Unavailability Re-Buffering Time during Streaming Streaming Synchronization Packet loss rate Latency Jitter MOS Audio/Video Quality VoLTE Non Accessibility Session Setup Failure Setup time Cut off Ratio Packet loss rate Latency Jitter MOS Audio/Video Quality Browsing Number of HTTP Sessions Average Duration of HTTP Sessions Average HTTP Throughput Peak HTTP Throughput HTTP Setup Time HTTP Dropped Connections HTTP Sessions Unavailability Retransmission Rate Packet loss rate Latency Jitter Loading Page Gaming Number of Gaming Sessions Gaming Sessions Average Duration Gaming Average Throughput (DL & UL) Gaming Overall responsiviness [ms] Retransmission Rate Packet loss rate Latency Jitter

28. Networks KPIs/KQIs © Zhilabs 2014, Confidential Information 8/6/2015 28

29. Next steps Get comments on TS 103 189 Extend TS 103 189 addiming more Customer Perception KPIs Get proposal from Video Test Houses Next event will be launched when VoLTE Native Handset and PEVQ tools will be available for testing 29

30. Contact Details: Giulio Maggiore Telecom Italia ETSI TC INT Chairman Giulio.Maggiore@telecomitalia.it 30 Thank you! © ETSI 2013. All rights reserved

31. TC INT TERMS OF REFERENCE 31 © ETSI 2013. All rights reserved

32. TC INT (Core Network and Interoperability Testing) Terms of Reference The scope of INT includes:INT Development of Core Network test specifications (interoperability, conformance, performance, security) based on 3GPP and E2NA specifications. To initiate and supervise interoperability events (such as Plugtests) related to Core Networks as well as other events (workshops and seminars) To coordinates interoperability efforts with other organisations GSMA, OMA, IMS/NGN Forum, IMTC, etc. Endorsement of test specifications from/to other SDOs e.g. ITU-T. 32 © ETSI 2013. All rights reserved

33. ETSI INT SPECIFICATIONS Interoperability, Performance, Security and Conformance 33 © ETSI 2013. All rights reserved

34. TC INT Interoperability Specification Work 2008 2009 2010 2011 2012 R6 R1 R2 R7Late R7 Late R2 R8 R9 R10 2007 TS 186 011 V 1.1.1 TS 186 011 V 2.1.1 TS 186 011 V 2.2.1 TS 186 011 V 3.1.1 TS 102 901 V 1.1.1 TS 186 011 V 4.1.1 TS 102 901 V 2.1.1 TS 103 129 TS 186 011 V 5.0.2 TS 102 901 V 5.0.6 TS 103 129 V2.0.4 2013 E2NA 34 ETSI/BOARD(13)94_041

35. Conformance Test Specifications IMS Basic Call Specifications and Supplementary Services; Diameter Conformance testing for Rx interface TS 101 580; Diameter Conformance testing for Gx interface TS 101 606; Diameter Conformance testing for S6a and S9 interface just started; ETSI/BOARD(13)94_041 35

36. Interoperability Test Specifications IMS NNI Interworking Test Specifications TS 186 011 RCS TS 102 901 IMS&EPC interoperability TS 103 129 ETSI/BOARD(13)94_041 36 IMS Interworking Test Specifications IMS Interconnection Test Specifications

37. INTEROPERABILITY EVENTS 37 © ETSI 2013. All rights reserved

38. Interoperability events 2007 – The first event examined interconnection aspects of 3GPP IMS Release 6, including basic call on the Mw interface. 2008 - The second event had a wider scope that included the testing of 3GPP IMS Release 7 interworking, roaming, border control, and integration of application servers executing selected Multimedia Telephony supplementary services. 2009 - The third ETSI IMS event assessed the interoperability as well as conformance of IMS core networks (composed of P/I/S-CSCF, IBCF, AS, DNS and HSS) which are implemented on the basis of ETSI TS 124 229/3GPP TS 24.229 (Version 7.14.0) (3GPP late release 7). 2012 – RCS&VoLTE interoperability event in cooperation with GSMA and MSF held in Kranj (Slovenia) and Beijng (China) ETSI/BOARD(13)94_041 38

39. Press Coverage: RCS VoLTE Test Event ETSI/BOARD(13)94_041 39

40. 2007 and 2008 IMS interoperability events The first event examined interconnection aspects of 3GPP IMS Release 6, including basic call on the Mw interface. The second event had a wider scope that included the testing of 3GPP IMS Release 7 interworking, roaming, border control, and integration of application servers executing selected Multimedia Telephony supplementary services. 40 © ETSI 2013. All rights reserved

41. 2009 - IMS interoperability events The third ETSI IMS event assessed the interoperability as well as conformance of IMS core networks (composed of P/I/S-CSCF, IBCF, AS, DNS and HSS) which are implemented on the basis of ETSI TS 124 229/3GPP TS 24.229 (Version 7.14.0) (3GPP late release 7). 41 © ETSI 2013. All rights reserved

42. 2012 – RCS & VoLTE interoperability event The forth event endorsed and tested a number of GSMA PRDs and validated the Testing Scenarios developed by MSF and ETSI to facilitate multi- vendor deployment of LTE./EPC/IMS technology for RCS and VoLTE services. http://www.etsi.org/news-events/past- events/642-2012-rcs-volte-interop-event 42 © ETSI 2013. All rights reserved