Benchmarked Key Performance Indicators ITU-T Workshop on Delivering Good Quality Telecommunication Service in a Safe Environment in Africa (Nairobi, Kenya, 26th July 2010) Benchmarked Key Performance Indicators Crisphine J. Ogongo, Engineer/Compliance Communications Commission of Kenya Nairobi - Kenya 26 - 27 July 2010

QoS Regulation And Comparison With Int’l Stds Conclusion Questions Content ITU/ETSI Standards QoS Regulation And Comparison With Int’l Stds Conclusion Questions Nairobi - Kenya 26 - 27 July 2010

ITU-T/ETSI STANDARDS Nairobi - Kenya 26 - 27 July 2010

Quality of service (QoS): Definition Quality of service (QoS): The collective effect of service performances, which determine the degree of satisfaction of a user of the service (ITU-T E.800) Nairobi - Kenya 26 - 27 July 2010

What QoS Really Is! Network Performance Parameters translated to; system design, configuration, operation and maintenance Quality of service parameter influenced by; Statistics e.g «Call Block Rate» User/customer requirements Individual experience e.g «inaccessibility»- User opinion/requirements feed back in to the network planning process to alter planned performance and/or practical operational standards Nairobi - Kenya 26 - 27 July 2010

ITU-T QOS Parameters with Targets Call Set Up Time/Post Dialing Delay Call Release Delay Answer Signal Delay End to End Blocking Handover Success Rate/Unsuccessful Handover Speech Quality Multimedia QoS Nairobi - Kenya 26 - 27 July 2010

Call Set up time Call type F-M M-F M-M Definition Authentication/ ciphering 0.0 2.5 Time interval between the end of dialing by the user and the reception by him of the appropriate tone or recorded announcement, or the abandon of the call without tone. Paging/alerting 4.0 Routing number transfer 2.0 Post-selection delay Local connection 3.0 Toll connection 5.0 International connection 8.0 TOTAL PLMN PSTN Normal Load High Load Mean 95% 9 5.5 11.5 3 s 6 s 4.5 s 9 s 11 7.5 13.5 5s 8 s 7.5 s 12 s 14 10.5 16.5 11 s 16.5 s Nairobi - Kenya 26 - 27 July 2010

Answer Signal Delay PLMN Connection Definition Call type F-M M-F M-M Time interval between the establishment of a connection between calling and called users, and the detection of an answer signal at the originating exchange. Local connection 1.0 1.25 Toll connection 1.75 2.0 International connection 2.75 3.0 PSTN Normal Load High Load Mean 95% 0.75 sec 1.5 sec 1.0 sec 2.0 sec 3.0 sec 4.0 sec 5.0 sec 3.3 sec 6.5 sec Nairobi - Kenya 26 - 27 July 2010

Call release delay (secs) Connection Type Call type F-M, M-F, M-M PSTN Calling, or called, party clears 1.0 sec 0.4 – 1sec Call release delay is defined as the time interval from the instant the first bit of the DISCONNECT message is passed by the user terminal which terminated the call to the access signaling system, until the last bit of the RELEASE message is received by the same terminal (indicating that the terminals can initiate/receive a new call). Definition Nairobi - Kenya 26 - 27 July 2010

End to End Blocking Definition: - The probability that any call attempt will be unsuccessful due to a lack of network resources 1990 Improvement PLMN On radio Channel 5-10% 1% PLMN to Fixed 0.5% Normal Load High Load PSTN Local 2% 3% Toll Connection 4.5% Int’l Connection 5% 7.5% Nairobi - Kenya 26 - 27 July 2010

Unsuccessful Handovers Is the probability that a handover attempt fails because of lack of radio resources in the target cell, or because of a lack of free resources for establishing the new network connection. The failure condition is based either on a specified time interval since the handover request was first issued or on a threshold on signal strength Call type F-M, M-F, M-M Probability of unsuccessful land cellular handover 0.5% Nairobi - Kenya 26 - 27 July 2010

Speech Quality PESQ: - Perceptual evaluation of speech quality ITU-T P.862 An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs PESQ compares an original signal X(t) with a degraded signal Y(t) that is the result of passing X(t) through a communications system. The output of PESQ is a prediction of the perceived quality that would be given to Y(t) by subjects in a subjective listening test(6). Provides Speech Quality raw values -0.5 to 4.5 Nairobi - Kenya 26 - 27 July 2010

perceived Speech quality PESQ: - Perceptual evaluation of speech quality ITU-T P.862 An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs Original Signal X (t) Communication Network Y(t) PESQ Algorithm perceived Speech quality Original Signal X(t) Subjects in Listening Test Perceived Speech quality Original Signal X(t) Nairobi - Kenya 26 - 27 July 2010

Speech Quality cont’d ITU-T P.862.1 - Mapping function for transforming P.862 raw result scores to MOS The mapping ensures a domain rescaling from –0.5 ... 4.5 to 1.02 ... 4.56 Nairobi - Kenya 26 - 27 July 2010

Speech quality depends on Transmission Rating Factor R Speech Quality Cont’d Speech quality depends on Transmission Rating Factor R The R‑value is a measure of a quality perception to be expected by the average user when communicating via the connection under consideration Nairobi - Kenya 26 - 27 July 2010

Transmission Rating Factor conversion to MOS equivalent Speech Quality Cont’d Transmission Rating Factor conversion to MOS equivalent R-value range Equivalent MOS Values Speech transmissio n quality category User satisfaction 90  R < 100 5 Excellent Very satisfied 80  R < 90 4 Good Satisfied 70  R < 80 3 Fair Some users dissatisfied 60  R < 70 2 Bad Many users dissatisfied 50  R < 60 1 Poor Nearly all users dissatisfied Conversion of R‑values into MOS MOS = MOS = 1+0.035R + R(R-60)(100-R)7.10⁻⁶ Nairobi - Kenya 26 - 27 July 2010

MOS User Satisfaction Levels Typical values used in R-Value MOS Value Comment Country R ≤ 0 1 Nearly all users dissatisfied R = 55 2.835 Many users dissatisfied Kenya R = 60 3.1 Some users dissatisfied Nigeria/Kenya in 3yrs R = 70 3.597 R = 80 4.024 Satisfied R ≥ 100 4.5 Very satisfied Satisfied MOS = MOS = 1+0.035R + R(R-60)(100-R)7.10⁻⁶ Nairobi - Kenya 26 - 27 July 2010

Data/Multimedia Parameters Critical to Users Delay variations Occurs at the transport layer in packetized data systems due to the inherent variable arrival times of individual packets Solved through buffering Delay the time taken to establish a particular service from the initial user request and the time to receive specific information once the service is established. Packet Loss Of packets or bits during transmission Includes coding degradation Nairobi - Kenya 26 - 27 July 2010

Multimedia (Audio and Video) QoS Parameters and targets KPI 1-way delay Delay Variation Information Loss (Packet Loss Ratio-PLR) Audio Conversational Voice <150 ms < 1 ms < 3% Audio Voice Messaging < 1 s playback High Quality Streaming Audio < 10 s << 1 ms < 1% Videophone (2-way) < 150 ms Videophone (1-way) Nairobi - Kenya 26 - 27 July 2010

Multimedia QoS Parameters KPI Symmetry T-Values Delay Web-browsing – HTML 1-way ~10 KB P< 2 s /page A < 4 s /page No delay variation or information Loss Key P-preferred A-Acceptable T-typical Bulk data transfer/retrieval 10 KB-10 MB P < 15 s A < 60 s Transaction services – high priority e.g. e‑commerce 2-way < 10 KB P < 2 s A< 4 s Command/control ~ 1 KB < 250 ms Still image < 100 KB Interactive games < 1 KB < 200 ms Telnet E-mail (server access) P< 2 s E-mail (server to server transfer) Can be several minutes Nairobi - Kenya 26 - 27 July 2010

QOS REGULATION IN AFRICA AND COMPARISON WITH INTERNATIONAL STANDARDS Case studies Kenya Nigeria Uganda Tanzania Nairobi - Kenya 26 - 27 July 2010

Status of Country QoS Regulation Nigeria Tanzania Uganda Kenya QoS regulated √ QoS Parameters and Targets Regulations x Guidelines X Licenses Objective parameters Subjective Parameters PSTN Network Cellular Mobile Network Internet Data (Leased Lines) Nairobi - Kenya 26 - 27 July 2010

Objective KPIs and Targets Benchmarks ITU Targets Kenyan Targets Nigeria Tanzania Uganda Completed Call (%) - 90 >99 99 Speech Quality (MOS) %age with Good SQ 4.0 ≤ MOS ≥ 4.5 95%>2.7 & 3.1 in 3 yrs 98 >2.0 MOS >95 95 Call Drop Rate (%) 2 <3 Call Block Rate (%) 1 10 Call Set Up Time (s) varied 13.5 < 10 Call Set Up Success Rate (%) Handover Success Rate (%) 99.5% 85 Call release delay 1s 2s Multimedia (IP) QoS Varied ITU Nairobi - Kenya 26 - 27 July 2010

Subjective QoS parameters Account complaint rate Account complaint resolution time Disconnection complaint rate Disconnection complaint resolution time  Miscellaneous complaint rate  Miscellaneous complaint resolution time  Fault report rate  Fault repair time Billing Accuracy  Service supply time  Call centre answer success ratio  Call centre answer time Complaint resolution Nairobi - Kenya 26 - 27 July 2010

Conclusion ITU-T recommendations have covered many QoS issues but very few targets have been specified Where ITU-T targets are specified, they should be held as minimum thresholds not negotiable downwards Many regulators monitor QoS parameters Targets adopted by regulators vary widely Nairobi - Kenya 26 - 27 July 2010

Questions How many regulators monitor QoS performance of providers? What are the KPI’s monitored? How comprehensive are KPIs in addressing customers requirements? How do the targets compare with international standards or best practices? Is there need for African Regulators to benchmark and adopt similar and adequate parameters and targets? Nairobi - Kenya 26 - 27 July 2010

Thank You Nairobi - Kenya 26 - 27 July 2010