Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang.

Similar presentations


Presentation on theme: "1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang."— Presentation transcript:

1 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang

2 Internet Video is taking off 2 Improve Users Quality of Experience

3 Video Quality Metrics: The State of the Art 3 Objective Score (e.g., Peak Signal to Noise Ratio) Subjective Scores (e.g., Mean Opinion Score)

4 Problem 1: New Effects, New Metrics 4 PLAYER STATES EVENTS JoiningPlayingBufferingPlaying Buffer filled up Buffer empty Buffer filled up Switch bitrate

5 Problem 1: New Effects, New Metrics 5 PLAYER STATES EVENTS Joining PlayingBufferingPlaying Buffer filled up Buffer empty Buffer filled up Switch bitrate

6 Problem 1: New Effects, New Metrics 6 PLAYER STATES EVENTS Joining Playing BufferingPlaying Buffer filled up Buffer empty Buffer filled up Switch bitrate

7 Problem 1: New Effects, New Metrics 7 PLAYER STATES EVENTS JoiningPlaying Buffering Playing Buffer filled up Buffer empty Buffer filled up Switch bitrate

8 Problem 1: New Effects, New Metrics 8 PLAYER STATES EVENTS JoiningPlayingBuffering Playing Buffer filled up Buffer empty Buffer filled up Switch bitrate

9 Problem 1: New Effects, New Metrics 9 PLAYER STATES EVENTS JoiningPlayingBuffering Playing Buffer filled up Buffer empty Buffer filled up Switch bitrate

10 Problem 1: New Effects, New Metrics 10 PLAYER STATES EVENTS JoiningPlayingBufferingPlaying Buffer filled up Buffer empty Buffer filled up Switch bitrate Join TimeBuffering Ratio Rate of buffering Rate of switching Average bitrate

11 Problem 2: Opinion Scores Engagement Opinion Scores - Not representative of in the wild experience - Combinatorial explosion of parameters Engagement as replacement for opinion score. (e.g., Play time, customer return rate) 11

12 Internet Video QoE 12 Objective Scores PSNR Subjective Scores MOS

13 Internet Video QoE 13 Objective Scores PSNR Subjective Scores MOS Engagement (e.g., Fraction of video viewed)

14 Internet Video QoE 14 Objective Scores PSNR Join Time, Avg. bitrate, …? Subjective Scores MOS Engagement (e.g., Fraction of video viewed)

15 Internet Video QoE 15 Objective Scores PSNR Join Time, Avg. bitrate, …? f(Join Time, Avg. bitrate, …) Subjective Scores MOS Engagement (e.g., Fraction of video viewed)

16 Internet Video QoE 16 Objective Scores PSNR Join Time, Avg. bitrate, …? f(Join Time, Avg. bitrate, …) Subjective Scores MOS Engagement (e.g., Fraction of video viewed)

17 Outline Need for a unified QoE What makes this hard? Our proposed approach 17

18 18 Challenge: Complex Engagement-to-metric Relationships Engagement Quality Metric

19 [Dobrian et al. Sigcomm 2011] 19 Challenge: Complex Engagement-to-metric Relationships Engagement Quality Metric Non-monotonic Engagement Average bitrate

20 [Dobrian et al. Sigcomm 2011] 20 Challenge: Complex Engagement-to-metric Relationships Engagement Quality Metric Non-monotonic Engagement Average bitrate Engagement Rate of switching Threshold

21 Challenge: Complex Metric Interdependencies 21 Join Time Bitrate Rate of buffering Rate of switching Buffering Ratio

22 Challenge: Complex Metric Interdependencies 22 Join Time Bitrate Rate of buffering Rate of switching Buffering Ratio

23 Challenge: Complex Metric Interdependencies 23 Join Time Rate of buffering Rate of switching Buffering Ratio Bitrate

24 Challenge: Complex Metric Interdependencies 24 Join Time Avg. bitrate Rate of buffering Rate of switching Buffering Ratio

25 25 Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies

26 Casting as a Learning Problem 26 MACHINE LEARNING EngagementQuality Metrics QoE Model Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies

27 Impact of the ML algorithm 27 Classify engagement into uniform classes Accuracy = # of accurate predictions/ # of cases ML algorithm must be expressive enough to handle the complex relationships and interdependencies

28 Challenge: Confounding Factors 28 Live and VOD sessions experience similar quality

29 Challenge: Confounding Factors 29 However, user viewing behavior is very different

30 Challenge: Confounding Factors 30 Devices User Interest Connectivity Need systematic approach to identify and handle confounding factors

31 Domain-specific Refinement 31 MACHINE LEARNING EngagementQuality Metrics QoE Model

32 Domain-specific Refinement 32 MACHINE LEARNING EngagementQuality Metrics QoE Model Confounding Factors

33 Improved prediction accuracy 33 Refined ML models can handle confounding factors

34 Concluding Remarks Internet Video needs unified quantitative QoE What makes this hard? – Complex engagement-to-metric relationships – Complex metric-to-metric interdependencies – Confounding factors (e.g., genre, device) Promising start – Machine learning + domain-specific refinements Open Challenges – Coverage over confounding factors – System Design 34


Download ppt "1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang."

Similar presentations


Ads by Google