Measuring Quality of Experience for Successful IPTV Deployments Dr. Stefan Winkler

Outline Challenges –Digital Video Quality Issues –Traditional Measurements (QoS) vs. Quality of Experience (QoE) Possible Solutions –QoE Measurement Approaches –End-to-end QoE Management Conclusions

Digital Video Challenges Demanding traffic profiles High bandwidth streams High traffic volumes Live, VOD Network effects Video impacted heavily with minor network impairments Multi-vendor network complicates diagnosis / troubleshooting High end-user expectations Defined with decades of history Grow rapidly with HD Low tolerance for poor quality New architectures Sensitive video processing devices create possibility for various impairment sources Ad-insertion, middleware Service quality degradations Difficult diagnosis, troubleshooting Rising management and OPEX costs Higher customer churn

What Drives End-Users Source: MRG 2007 IPTV Video Quality Survey, available at

Service Providers View Source: MRG 2007 IPTV Video Quality Survey, available at

Service Providers’ View 7 Source: MRG 2007 IPTV Video Quality Survey, available at

Sources of Video Issues Consider all elements for true end-to-end solution

Compression Artifacts OriginalMPEG-2H.264

PSNR vs. QoE Same amount of distortion (PSNR) – different perceived quality Understand & model human vision system

QoS vs. QoE Quality of Service –Network-centric –Delay, packet loss, jitter –Transmission quality –Content agnostic Quality of Experience –Content impairments –Blockiness, Jerkiness, … –End-user quality –Application driven QoS QoE

Same network impairments Packet Loss: 1% Delay: 10ms Jitter: 50us Bandwidth: 500kbps Different perceived quality! QoS vs. QoE

MDI vs. QoE Media Delivery Index (MDI) MDI consists of two metrics: –Delay Factor (DF) –Media Loss Rate (MLR) MDI limitations: –MDI assumes constant bit rate (CBR) traffic –MDI does not consider video payload or content –MDI values are not intuitive –MDI doesn’t correlate with video quality

MDI vs. QoE Media Loss MOS

QoS/QoE Cycle Desired QoE Perceived QoE Targeted QoS Delivered QoS End-user Service provider Alignment gap Perception gap Value gapExecution gap Adapted from ITU-T Rec. G.1000 and COM12–C185–E

Outline Challenges –Digital Video Quality Issues –Traditional Measurements (QoS) vs. Quality of Experience (QoE) Possible Solutions –QoE Measurement Approaches –End-to-end QoE Management Conclusions

Full-Reference Approach Comparison of individual video frames Offline analysis (capture is required) – lab applications High detail and accuracy Alignment procedure Compression/ Transmission System SenderReceiver Full reference information Video Full Ref. Quality Measurement

No-Reference Approach Non-intrusive, in-service measurement Real-time monitoring applications No alignment required Compression/ Transmission System SenderReceiver Video No-Ref. Quality Measurement

Monitoring applications Correlation of content and network impairments Encrypted environments Compression/ Transmission System SenderReceiver Video Reduced Ref. Measurement Feature Extraction Reduced-Reference Approach

Content & Network Metrics 20 "Vision is the most highly developed of the human senses, so people are even more sensitive to flaws in video images than, say, the sound of a telephone conversation.” Ken Wirt, Cisco Vice President Consumer Marketing, Jan 2008 (Correlation Engine)

Contrast perception –Visibility of different patterns –Frequency dependencies Masking effects –Interaction of content and impairments –Texture, edges, luminance –Spatial and temporal masking Color perception Spatial frequency [cpd]Temporal frequency [Hz] Sensitivity Vision Modeling Masker contrast Visibility threshold Target contrast Masking curve Threshold without masker

End-to-end QoE Deep Content Analysis (pixel by pixel) Source content and encoder / transcoder validation Human Vision System Model Video Quality Reports Content Impairments: Blockiness, blur Jerkiness Freeze/black frame Noise, Color Network Impairments: Loss Delay Jitter Bandwidth Content Stream Analysis: PES inspection PCR jitter etc. Deep Content Analysis (bitstream) Detect content impairments Deep inspection to associate content to timestamps (eg: TS1 = I-Frame) Network (header or stream) Analysis Detect QoS issues Content analysis where possible (unencrypted) Inspection of QoS to associate timestamps to impairments (eg: TS1 = Packet Loss) Q-Advisor Correlation Engine TS1 = I-Frame TS1 = Packet Loss Packet Loss -> I-Frame

IPTV QoE Management 25 IssuePossible Causes BlockinessEncoderTranscoderNetwork Loss BlurCamera (focus)EncoderTranscoderSTB (bad filtering) Freeze Frame, JerkinessEncoder (dropped frames) Network lossBad synchronization Black Screen, Blue Screen No Video Signal (source) Ads not insertedMajor network loss ColorEncoderCameraTranscoder Video Noise (analog noise) CameraSTB Noise (digital)EncoderTranscoder AudioMicrophoneEncoder (bad mono stereo encoding Encoder (lip sync) STB Understand the Service  Is there an issue?  Does it matter? 1. Understand the Service  Is there an issue?  Does it matter? 2. Understand the Problem  What does the customer see?  What is the exact cause? 2. Understand the Problem  What does the customer see?  What is the exact cause? 3. Understand the Solution  What is the impairment source? 3. Understand the Solution  What is the impairment source? Very Annoying Annoying Slightly Annoying Perceptible Imperceptible

Conclusions QoE is application-driven –Measure both network and content impairments QoE is user-oriented –Measure how end-user perceives service issues End-to-end quality measurement –Cover different impairment sources –Identify problem causes

Stefan Winkler Company: qoe.symmetricom.com qoe.symmetricom.com Further Reading: stefan.winkler.net/book.html stefan.winkler.net/book.html Contact Info