Download presentation
Presentation is loading. Please wait.
1
A Quest for an Internet Video Quality-of-Experience Metric
Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang
2
Internet Video is taking off
Improve Users’ Quality of Experience
3
Video Quality Metrics: The State of the Art
Objective Score (e.g., Peak Signal to Noise Ratio) Subjective Scores (e.g., Mean Opinion Score)
4
Problem 1: New Effects, New Metrics
PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate
5
Problem 1: New Effects, New Metrics
PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate
6
Problem 1: New Effects, New Metrics
PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate
7
Problem 1: New Effects, New Metrics
PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate
8
Problem 1: New Effects, New Metrics
PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate
9
Problem 1: New Effects, New Metrics
PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate
10
Problem 1: New Effects, New Metrics
PLAYER STATES EVENTS Joining Playing Buffering Buffer filled up empty Switch bitrate Join Time Buffering Ratio Rate of buffering Rate of switching Average bitrate
11
Problem 2: Opinion Scores Engagement
Opinion Scores - Not representative of “in the wild” experience - Combinatorial explosion of parameters Engagement as replacement for opinion score. (e.g., Play time, customer return rate)
12
Internet Video QoE Subjective Scores MOS Objective Scores PSNR
Subjective score replaced by eng. Objective Scores PSNR
13
(e.g., Fraction of video viewed)
Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) PSNR doesn’t take into account different effects Objective Scores PSNR
14
(e.g., Fraction of video viewed)
Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) Replace it with the metrics. But which one? Each one use only cover one aspect of the session. Objective Scores PSNR Join Time, Avg. bitrate, …?
15
(e.g., Fraction of video viewed) f(Join Time, Avg. bitrate, …)
Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) Objective Scores PSNR Join Time, Avg. bitrate, …? f(Join Time, Avg. bitrate, …)
16
(e.g., Fraction of video viewed) f(Join Time, Avg. bitrate, …)
Internet Video QoE Subjective Scores MOS Engagement (e.g., Fraction of video viewed) Objective Scores PSNR Join Time, Avg. bitrate, …? f(Join Time, Avg. bitrate, …)
17
Outline Need for a unified QoE What makes this hard?
Our proposed approach
18
Challenge: Complex Engagement-to-metric Relationships
Quality Metric First main challenge. Relationship between quality metric and eng – we had a simplistic view. But in the real world the relationships are more complex.
19
Challenge: Complex Engagement-to-metric Relationships
Non-monotonic Engagement Average bitrate Engagement Quality Metric Avg bitrate and engagement – non-monotonic [Dobrian et al. Sigcomm 2011]
20
Challenge: Complex Engagement-to-metric Relationships
Non-monotonic Engagement Average bitrate Engagement Quality Metric Engagement Rate of switching Threshold And rate of switching and engagement – threshold effect Measurement study by Dobrian et al. in Sigcomm 2011 show many of these relationships. [Dobrian et al. Sigcomm 2011]
21
Challenge: Complex Metric Interdependencies
Join Time Bitrate Rate of switching Rate of buffering Quality metrics, they are not really independent of each other. Buffering Ratio
22
Challenge: Complex Metric Interdependencies
Join Time Bitrate Rate of switching Rate of buffering Buffering Ratio
23
Challenge: Complex Metric Interdependencies
Join Time Bitrate Rate of switching Rate of buffering Buffering Ratio
24
Challenge: Complex Metric Interdependencies
Join Time Avg. bitrate Rate of switching Rate of buffering There might be several other dependencies. Buffering Ratio
25
Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies
26
Casting as a Learning Problem
Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies MACHINE LEARNING Engagement Quality Metrics QoE Model
27
Impact of the ML algorithm
Classify engagement into uniform classes Accuracy = # of accurate predictions/ # of cases ML algorithm must be expressive enough to handle the complex relationships and interdependencies
28
Challenge: Confounding Factors
Live and VOD sessions experience similar quality
29
Challenge: Confounding Factors
However, user viewing behavior is very different
30
Challenge: Confounding Factors
Devices User Interest Connectivity Need systematic approach to identify and handle confounding factors
31
Domain-specific Refinement
Engagement Quality Metrics MACHINE LEARNING QoE Model
32
Domain-specific Refinement
Engagement Confounding Factors Quality Metrics MACHINE LEARNING QoE Model
33
Improved prediction accuracy
Refined ML models can handle confounding factors
34
Concluding Remarks Internet Video needs unified quantitative QoE
What makes this hard? Complex engagement-to-metric relationships Complex metric-to-metric interdependencies Confounding factors (e.g., genre, device) Promising start Machine learning + domain-specific refinements Open Challenges Coverage over confounding factors System Design
Similar presentations
