Perceptual Quality Assessment of P2P Assisted Streaming Video for Chunk-level Playback Controller Design Tom Z.J. Fu, CUHK W. T. Leung, CUHK P. Y. Lam, CUHK Dah Ming Chiu, CUHK Zhibin Lei, ASTRI PV 2010, Hong Kong

Introduction and motivation Chunk-level impairment model Experiments with various Chunk Receiving Patterns (CRP) Heuristic on satisfaction function Future work and conclusion Outline

Internet streaming service becomes popular 1.C/S mechanism, 2.P2P mechanism, mostly implemented.  CDN,  single/multiple tree-based application layer multicast,  peer-to-peer streaming (live streaming / VoD). The evaluation for the above two mechanisms are quite different. Introduction and motivation

1. For the C/S mechanism,  The simple end-to-end link model is suitable for abstraction, link condition metrics:  Packet loss rate  End-to-end packet transmission delay  etc. Packet-level impairments are considered

Introduction and motivation 2.For the P2P assisted mechanism,  The simple end-to-end link model is not suitable:  The transmission pattern is dynamic and complicated. P2P mechanism forms overlay topology

Introduction and motivation 2. For the P2P assisted mechanism,  The simple end-to-end link model is not suitable: 1.The transmission pattern is dynamic and complicated. 2.The granularity of packet-level impairment is too fine, causing mismatch with system design: a)Important building blocks of P2P mechanism are based on chunks: chunk selection, peer selection, buffer map management  affecting the changing of the overlay topology b)System-wide performance metrics are chunk level:  Average playback (dis) continuity;  Buffer filling probability, etc c)In each client side, Playback decision is on top of chunk (How to make decisions with QoE considered?) Therefore, we need chunk-level impairment model !

Chunk-level impairment model

Video encoder – Different media codec, transmission rate could be chosen at the video encoder component Network transmission – chunk level impairment module  Chunk maker – responsible for organizing video stream packets into chunks.  Chunk-level distortion generator – three different ways are designed to implement chunk-level distortion generator  Chunk buffer manager – manages and keeps the received chunks in a local chunk-level buffer (serving other peers later)  Playback controller (client software) – make playback decision for each chunk. Video decoder – After being decoded by the video decoder component, the processed videos (PVS) are then displayed in the monitors to the users. Chunk-level impairment model

Various distortion sources: a) Peers’ dynamic behaviors b) Peers’ network condition (Uplink and Downlink) c) Chunk/peer selection strategy d) Scheduling algorithm, etc…… Consider these factors  too complicated What we can do  make abstraction Chunk receiving pattern (CRP) is the equivalent distortion effects of all the distortion sources considered together. Chunk-level impairment model

General model of chunk-receiving pattern a) r i (x) – download percentage of chunk, any non-decreasing function b) L i – chunk size (all equal to l in the example below) c) X i s – starting downloading time d) X i c – complete downloading time e) X i d – desired playback time, depend on previous chunk f) X i p – real play back time, playback decision Three conditions: r 1 (x): non-delayed chunk, playback normally. r 2 (x): delayed chunk, wait and playback after completion r 3 (x): delayed chunk, wait and playback before completion Chunk-level impairment model

Three ways to generate chunk-receiving patterns 1.Live experiments Collecting and recording the CRP for each chunk a real- life experiment (not repeatable) 2.Simulation results Collecting and record CRPs from the simulation trace in a large network with a large number of users. (Simulation is repeatable under same settings, CRPs are following certain distributions) 3.Artificial generating Manually generate CRPs (by implementing r i (x) with certain increasing curves and parameters, completely repeatable)  suitable for subjective testing studies Chunk-level impairment model The parameter space for subjective testing should not be too large!

In the previous work, we start from the simplest form of CRP: the step function In the previous experiment: Step function assumes that X i s = X i c for all chunks. Two types of chunks: non-delayed (X i c X i d ); Simple playback strategy: X i P = X i d + LWT Non-delayed, Normal playback Delayed, wait and normal playback Delayed, wait and skip Experiments under various CRPs: I

In the previous work, we start from the simplest form of CRP: the step function  Fixed LWT = 3 seconds  Two types of chunk delay D (= X i c – X i d ) distribution 1.Short delay: uniformly distributed in [0, 2] seconds; or 2.Long delay: all equals to 3 seconds, ( = LWT). LWT D2D2 D3D3

Experiments under various CRPs: I In the previous work, we start from the simplest form of CRP: the step function  Fixed LWT = 3 seconds  Two types of chunk delay D (= X i c – X i d ) distribution 1.Short delay: uniformly distributed in [0, 2] seconds; or 2.Long delay: all equals to 3 seconds, ( = LWT).  Number of delay chunks is determined by applying different values of average discontinuity (d = 1 - c): 0, 5%, 10%, 15%, 20%, 25%, 30%, 40%, 50%, 60% Experiment settings I: 1.50 source video clips (News, Music videos, Movie Trailers and Sports) with average length of 30 seconds; 2.30 subjects (16 males and 14 females), age range ( ); 3.Absolute Category Rating (ACR) with hidden reference as assessment

Experiments under various CRPs: I In the previous work, we start from the simplest form of CRP: the step function Subjective assessment results for each processed video sequence MOS value (left), DMOS value (right): The meaning for Mean Opinion Score (MOS) and DMOS:

Experiments under various CRPs: I In the previous work, we start from the simplest form of CRP: the step function Comparison between short and long chunk delay distribution Insights from the comparison: 1. PVSes with long delay distribution obtain higher MOSes than those with short delay distribution when average d is same. 2. Subjects care more about the number of screen freezing events than the duration of each freezing event.

In this work, we are trying more complicated form of CRPs: the piecewise linear function In this experiment, we assume: Two types of chunks: non-delayed (X i c X i d ); For delayed chunks, three piecewise linear patterns; Playback strategy with D: X i p = X i d + D (D <= LWT) R(D): Chunk completeness when playback (depend on D and Pattern) Experiments under various CRPs: II LWT R A (D) R B (D) R C (D) XidXid X i c =X i d + LWTX i p = X i d + D

In this work, we are trying more complicated form of CRPs: the piecewise linear function In this experiment:  Fixed probability of delayed chunks = 0.1; i.e., 3 out of 30 chunks are delayed;  Fixed LWT = 4 seconds;  Fixed X i c = X i d + LWT;  5 playback decisions: D = X i p – X i d = 0, 1, 2, 3, 4 seconds.  Real implementation of R(D): Experiments under various CRPs: II XidXid X i c =X i d + LWT LWT X i p = X i d + D R A (D) R B (D) R C (D)

In this work, we are trying more complicated form of CRPs: the piecewise linear function Experiments under various CRPs: II Experiment settings II: 1.56 source video clips (News, Music videos, Movie Trailers and Sports) with average length of 30 seconds; 2.42 subjects (19 males and 23 females), age range ( ); 3.Absolute Category Rating (ACR) with hidden reference as assessment

In this work, we are trying more complicated form of CRPs: the piecewise linear function Experiments under various CRPs: II Insights from the comparison among four video categories: 1. News earns the highest scores in all patterns, followed by MV. This is due to the nature of these two categories. 2. The overall difference among three patterns are not very big, probably because we fixed the number of delayed chunks. Patten A Patten B Patten C

We try to model the trade off between Waiting time D and Chunk completeness R. Heuristic Satisfaction function: S(R, D) = alog (R) + bD c, a >= 0, b 0 So we have: This is one possible choice, other forms may investigate in future. Heuristic on satisfaction function

After applying the three patterns to the heuristic satisfaction function with proper parameters: Experiments under various CRPs: II Insights from the comparison among S-function and experiment results: 1. The S-function does not match the results so well, more proper form of S-function needs to be studies and investigated. 2. However, the optimal D values for three patterns are similar as shown in both figures. It indicates that, usually, for pattern like A, small waiting time is better while patterns like C, longer waiting time is more preferred.

Future work  Carry out more experiments with different chunk receiving patterns and parameter settings, e.g. number of delayed chunks, LWT, etc.  Investigate various forms of S-functions to fit the results.  Consider layered coding in P2P streaming (another form of D and R) Conclusion  Chunk-level impairment model is proposed for P2P mechanism.  We introduce two sets of QoE Experiments by applying chunk-level impairment model.  The results are preliminary but still get some interesting insights. Future work and conclusion

The end Thanks! Q & A