1. Streaming Media Performance
January 2995
doc.: IEEE /1531r0
September 2005
Date:
Streaming Media Performance
Authors:
Name
Company
Address
Phone
Fahd Pirzada
Dell
One Dell Way
Round Rock, TX 78682
Pratik Mehta
Notice: This document has been prepared to assist IEEE It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein.
Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE
Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures < ieee802.org/guides/bylaws/sb-bylaws.pdf>, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE Working Group. If you have questions, contact the IEEE Patent Committee Administrator at
Fahd Pirzada - Dell
Bruce Kraemer, Conexant

2. September 2005
Goals
Determine the recommended practices for a wireless benchmark that reflects the end-user’s streaming media experience
Throughput and Range are sufficient to characterize data-oriented usages (browsing, , file transfer, etc)
Need help in breaking down the complexities to a manageable set of measurements for streaming media usages
Introduce the various components involved in network media streaming
Define performance measurements for streaming media usage over networks
Discuss ways to correlate the performance measurements to end-user experience
Fahd Pirzada - Dell

3. Contents of a Benchmark
September 2005
Contents of a Benchmark
A defined Workload
A defined streaming stack
Performance measurements at various layers of the streaming stack
Correlation to end-user experience
Complications
Large variety of encoding/decoding mechanisms in defining the workload
Was the video quality good enough to start with?
Large variety of network streaming protocols and services
Did all the information make it over the link?
Large variety of media players or content renderers
Was there any effect (scaling, color conversion) introduced by the display device?
Need a simplified subset of the above for a successful benchmark
Fahd Pirzada - Dell

4. September 2005
Workload
The unique requirements of the steaming media usage case differentiate streaming traffic (workload) from data-oriented traffic
Workload should meet the following criteria:
The workload should represent the usage by end-users
The workload should be compatible with widely-used networking stacks and protocols
The workload should be intensive enough to exercise and stress the network stack
The same workload should be used for all measurements
Describe how to create such a workload
Transscale
Transrate
Transcode
Transcrypt
Storage
Delivery
Fahd Pirzada - Dell

5. September 2005
Streaming Stack
Different streaming stacks may result in different end-to-end performance over the same network
Represent common usage by end-users by describing the requirements for the streaming stack at the different levels
Commonly used codecs for streaming applications
Real-time streaming/playback support with optional buffering
Streaming service that enables proper hosting of workload
Standard protocol parameters that are suited for real-time applications
TCP, UDP, HTTP or relevant delivery mechanism represented
IP networking support for relay over networks
Standard MAC/PHY baseline/modifier parameters
Codec
Player
Service
Protocol
Transport
Network
MAC/PHY
Fahd Pirzada - Dell

6. Performance Measurements
September 2005
Performance Measurements
The end-to-end performance of streaming media measured at various layers of the stack
The correlation between measurements at various layers provide a measure of user perception of performance
User perception: acceptable, annoying
Codec
Codec
Player
Player
Player Statistics: blockiness, frame rate, blur, buffering vs. real-time
Service
Service
Video encapsulation to network packets
Protocol
Protocol
Protocol indicators: buffers, bandwidth, losses
Transport
Transport
Min. UDP Throughput, Lost Packets, TCP re-tx
Network
Network
MAC Throughput, Loss Rate
MAC/PHY
MAC/PHY
Interference, Background traffic characteristics
Contending traffic over the air, QoS
Fahd Pirzada - Dell

7. Performance Correlation
September 2005
Performance Correlation
How are the performance measurements linked to each other?
Perceived Performance
Is the video quality unacceptable to the user?
What is the upper bound of streaming protocol performance needed for acceptable video quality?
Protocol Performance
Hardware Performance
How resilient is the network hardware to losses over the network?
Can we overcome the impacts of inherently unreliable channel (wireless) that has bi-directional information flow?
Network Performance
What level of network congestion affects the streaming performance?
Fahd Pirzada - Dell

8. Perceived Performance
September 2005
Perceived Performance
User Perceived Media Quality Guide:
5 – Perfect video quality, Perfect audio synchronization
4 – Slight visible blur/blockiness, Perfect audio synchronization
3 – Jerkiness or blockiness, Perfect audio synchronization
2 – Greater jerkiness, Perfect audio synchronization
1 – Greater jerkiness or out of sync audio
0 – Unable to display video
Unacceptable
Acceptable
Current Tool: Human eye
Future Tool: Visual Quality Metrics
(currently only works for uncompressed video)
VQM provides an efficient mechanism to analyze video performance
VQM measures the extent of blockiness, blur and jerkiness but it does not quantize the effect of these artifacts on end-user experience
VQM does not provide a measure of audio-video synchronization. AV synchronization has a significant impact on end-user experience
VQM can be used to separate the end-user performance into two buckets: acceptable and unacceptable. The unacceptable bucket will require further performance analysis.
Fahd Pirzada - Dell

9. Video Quality Measurement
September 2005
Video Quality Measurement
Comparison
Workload
Media Quality
Comparison
Comparison
Workload
Change Format
Change Format
Uncompressed
Workload
Uncompressed
Workload
Uncompress
Uncompress
Encoded
Workload Source
802.11
Encoded
Workload Sink
Encode
Original Workload
Fahd Pirzada - Dell

10. Protocol Performance September 2005
Streaming Server
Streaming Client
For this RTP streaming scenario, video streams under 25Mbps may be supported
Fahd Pirzada - Dell

11. Hardware Performance September 2005
Starting from PHY performance, we can model the overhead caused by the network stack
802.11a net throughput with partitioning of raw bit rate into various kinds of overhead
802.11a PHY Throughput across various QoS access categories (10 clients with target bit rate of 2.4Mbps)
Fahd Pirzada - Dell

12. Network Performance September 2005 Background Average Throughput
Unacceptable video performance
Background Average Throughput
Video Server
Access Point
Network Switch
Chariot Server
Chariot Client
Video Client 1
Video Client 2
Video Client 3
Methodology: Measure background traffic to Chariot Client under the following scenario:
Single 6Mbps video stream to video client
Two 6Mbps video streams to different video clients
Three 6Mbps video streams to different video clients
Fahd Pirzada - Dell

13. Performance Correlation
September 2005
Performance Correlation
Measure protocol performance (throughput, range, etc.) for a specific source-sink pair
Measure perceived media quality for the same source-sink pair
Correlate media quality measurements to network performance at various ranges
Use a single source and multiple sinks to determine network performance
With background traffic
Without background traffic
Similar curves for Latency, Packet Loss and other secondary metrics
Fahd Pirzada - Dell

14. September 2005
Summary
We can provide a good measure of acceptable or unacceptable end-user perception using standard tools like VQM
We can analyze video artifact and evaluate their impact on end-user perception
We have to show the correlation between perceived, protocol, hardware and network performance
This correlation can help us quantify the impact of protocol, hardware and network performance measurements to streaming media performance
This correlation will enable the users of the TGT recommended practice to integrate perceived performance measurements with various secondary metrics that are defined in TGT
Fahd Pirzada - Dell