Effects of Network Congestion (Packet Loss) on Video Streaming – A User Study Rahul Amin, France Jackson, Morris (Trey) Lee, Jim Martin, Juan Gilbert Last Updated: May 24, 2013

Objective The goal of the study was to test the following hypotheses: – Is it beneficial for video content providers to start the video at a lower quality so as to set low user expectations and eventually move to a little higher quality? – If the user expectation is set low by a low video quality in the beginning, are the users ok with a sub- par quality for the rest of the video? – Do users have different video quality expectations if they are told that the video they are watching is online content vs. TV Cable provider content?

Methodology – Pilot Study To determine the different levels of video quality, a pilot study with different packet loss settings was run – The goal of this study was to capture the packet loss settings where video quality degraded but no buffering screen message was received – At 12% packet loss, the video kept buffering every few seconds. So the the worst video quality loss setting had to be less than 12% packet loss. – After getting an opinion from several participants, the following video quality-packet loss mappings were determined Bad Quality – 9% Packet Loss OK Quality – 6% Packet Loss Good Quality – 0% Packet Loss

Methodology – Actual Study A 10-minute movie clip streamed using Netflix was recorded with 3 different network congestion settings – Setting 1 (Bad-to-Good): 9% Packet Loss for first 5 minutes, 0% Packet Loss for last 5 minutes – Setting 2 (Good-to-Bad): 0% Packet Loss for first 5 minutes, 9% Packet Loss for last 5 minutes – Setting 3 (Bad-to-OK): 9% Packet Loss for first 5 minutes, 6% Packet Loss for last 5 minutes

Demographic Breakdown

Mean Opinion Score Results – On Demand Study  Based on the results, bad-to- ok setting has the highest ‘overall’ MOS. So, starting the video quality low and then improving it a little satisfied most of the users.  Starting at low quality and bumping up the quality to maximum level (bad-to-good) does not help with the ‘overall’ MOS because the participant sees the actual difference on what he/she should have obtained (end compared to beginning)  Good-to-Bad setting also frustrates the viewer and results in about same ‘overall’ rating as Bad-to-Good setting

Mean Opinion Score Results – Online Study  Based on the results, bad-to- ok setting again has the highest ‘overall’ MOS. So again, starting the video quality low and then improving it a little satisfied most of the users.  The overall MOS for each setting is lower than that of the ‘On-Demand’ study as users have a pre-conceived notion of ‘online’ quality generally being not too good. This is more apparent in the ‘willingness to pay’ result presented in the next slide.

Mean Opinion Score Results – Willingness to Pay for the Content  ‘On-demand’ or ‘online’ content makes a huge difference in participant’s willingness to pay. If the content is online, the participants generally do not want to pay too much for it; whereas the participants are willing to pay a little more.  This clearly shows that viewer’s have different expectations for on-demand vs online content. This will have an effect on how the participants answered some of the questions.

Mean Opinion Score Results  There isn’t much variation in MOS for all the settings for this question. Perhaps once users see some degradation (beginning/end) which is the case for all settings, they are always going to be a little likely to change service providers.

Mean Opinion Score Results  This question was asked at the very end after asking questions about ‘video’ as well as ‘audio’ quality.  The bad-to-ok on-demand setting again has the highest MOS; however, bad-to-good on-demand/online settings have the lowest MOS!!

Oral Question Responses  Participants were asked an opinion question at the very end: would they prefer the quality of video be a little bad at the beginning or end? Or would any degradation be unacceptable to them?  Most of the participants said they would not tolerate any degradation (45% - None).  Majority of Participants who would tolerate a little bit of degradation would prefer the degradation in the beginning (33%) as opposed to the end (22%).  Remark: Some participants stated different responses based on if they were paying for the content. If they were paying for it, they would not accept any degradation; but if it was free, then some degradation in the beginning or end was acceptable (which is what is plotted in the pie chart)

General Conclusions  If the user expectation is set low by a low video quality in the beginning, bumping up the quality a little (bad-to-ok) setting results in a higher MOS than the case where the quality is bumped up all the way (bad-to-good)  Overall MOS results for ‘On-Demand’ are a little higher than ‘Online’ studies; perhaps due to the viewer expectation being low for online quality video