1. Video Performance Authors: July 2006 Month Year
doc.: IEEE yy/xxxxr0
July 2006
Video Performance
Authors:
Notice: This document has been prepared to assist IEEE It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein.
Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE
Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures < ieee802.org/guides/bylaws/sb-bylaws.pdf>, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE Working Group. If you have questions, contact the IEEE Patent Committee Administrator at
Royce Fernald - Intel Corporation
John Doe, Some Company

2. Month Year
doc.: IEEE yy/xxxxr0
July 2006
Abstract
This presentation defines the primary metrics for video performance as well as key secondary metrics
This presentation corresponds to the draft text submission /0964r0
Royce Fernald - Intel Corporation
John Doe, Some Company

3. Agenda July 2006 Month Year doc.: IEEE 802.11-yy/xxxxr0 Background
Proposal Framework
Test Output
Video Delivery and Video Quality
Measuring Video Delivery and Video Quality
The Media Delivery Index
Video Delivery, Video Quality and Wireless Performance
References
Test Description (doc.: /0964r0)
Introduction and Purpose
Resource Requirements
Test Setup
Video Content Preparation and Analysis
Calibration
Baseline Configuration
Modifiers
Measurement Procedures
Reported Results
Sample Data
Motion
Royce Fernald - Intel Corporation
John Doe, Some Company

4. Proposal Framework Usage case: Video streaming applications
July 2006
Proposal Framework
Usage case: Video streaming applications
Test Environments:
Conducted Environment
OTA Environments
Primary Metrics
Video Delivery
Video Delivery Error Rate (VDER)
Video Quality
Perceived Video Quality Mean Opinion Score (MOS)
Secondary Metrics
Media Delivery Index (MDI)
Media Loss Rate (MLR)
Delay Factor (DF)
Royce Fernald - Intel Corporation

5. July 2006
Test Output
Video Delivery and Video Quality metrics are the primary output of the test
MDI quantifies the impact of the network on the test outcome
Helps isolate root causes of poor performance
Helps establish the performance envelope required for wireless devices to deliver an expected video experience
Royce Fernald - Intel Corporation

6. Video Delivery and Video Quality
July 2006
Video Delivery and Video Quality
There are two complementary ways to characterize video performance – video delivery and video quality
Video delivery metrics quantify the impact of dropped, repeated or out-of-sequence video frames
Video delivery errors are caused by large-scale packet losses or delay on the link
Video quality metrics quantify the impact of video artifacts on end user perception
Video artifacts are caused by packet loss
Different codecs / content types translate levels of packet loss to different levels of video quality
Royce Fernald - Intel Corporation

7. Measuring Video Delivery and Video Quality
July 2006
Measuring Video Delivery and Video Quality
Video Delivery Measurements
Video delivery measurements are made with video playback analysis applications (VPAA)
VPAAs provide quantitative metrics that reflect the performance of the underlying transport medium
A correlation can be established between video delivery measurements and MOS of end user perception
Video Quality Measurements
Video quality analysis applications (VQAA) output mean opinion scores (MOS) that represent the expected end user perception of video quality
VQAAs are based on psychovisual models (such as VQM) which are designed to characterize the quality of a perceived video clip relative to a source file
Royce Fernald - Intel Corporation

8. The Media Delivery Index
July 2006
The Media Delivery Index
MDI defines media-weighted secondary metrics that quantify the impact of wireless link performance on observed video quality
The media loss rate (MLR) reflects the portion of the video stream that was undelivered due to packet loss
The MLR is the packet loss rate expressed as a percentage of the video stream
High MLR corresponds to video delivery errors (client buffer starvation) or noticeable display artifacts from undelivered packets
The delay factor (DF) quantifies the amount of buffering required to maintain an uninterrupted media stream
The DF is the required client buffer depth divided by the bitrate of the video stream
High DF corresponds to non-fluid playback and loss of interactivity for real-time applications such as video conferencing
Royce Fernald - Intel Corporation

9. July 2006
Video Delivery, Video Quality and Wireless Performance – Best Effort Traffic (i.e. UDP/RTP)
Royce Fernald - Intel Corporation

10. July 2006
Video Delivery, Video Quality and Wireless Performance – Guaranteed Traffic (i.e. TCP)
Royce Fernald - Intel Corporation

11. References IEEE 802.11-06/0964r0 “Video Performance”
July 2006
References
IEEE /0964r0 “Video Performance”
IEEE /0651r1 “Video Delivery versus Attenuation in a Conducted Environment”
IEEE /0321r0 “Video Gross Error Detector Video over Wireless Methodology”
IEEE /0144r1 “Video over Wireless Methodology”
IEEE /1194r0, “Video Testing Methodology”
IEEE /0887r0, “Video Testing Strategy”
Royce Fernald - Intel Corporation

12. Agenda July 2006 Month Year doc.: IEEE 802.11-yy/xxxxr0 Background
Proposal Framework
Test Output
Video Delivery and Video Quality
Measuring Video Delivery and Video Quality
The Media Delivery Index
Video Delivery, Video Quality and Wireless Performance
References
Test Description (doc.: /0964r0)
Introduction and Purpose
Resource Requirements
Test Setup
Video Content Preparation and Analysis
Calibration
Baseline Configuration
Modifiers
Measurement Procedures
Reported Results
Sample Data
Motion
Royce Fernald - Intel Corporation
John Doe, Some Company

13. Introduction and Purpose
July 2006
Introduction and Purpose
This test measures the video streaming performance of the link between a DUT and a WLCP
Video traffic is streamed to the DUT over the RF interface
The DUT display data is captured and analyzed for video delivery errors and video quality defects
Traffic measured by the traffic analyzer is used to calculate the MDI (media loss rate and delay factor)
Royce Fernald - Intel Corporation

14. Hardware Resource Requirements
July 2006
Hardware Resource Requirements
The basic test configuration is defined in section 5
In addition, the following items are required:
A traffic generator capable of generating video traffic
The video traffic generator can be a general purpose PC or dedicated device that streams video content to the DUT
Video capture device, a PC or dedicated device capable of connecting to the DUT display port and saving video data to non-volatile storage
The DUT may also act as the video capture device if it has the ability to save or process media streams in real-time (not progressive download)
Royce Fernald - Intel Corporation

15. Software Resource Requirements
July 2006
Software Resource Requirements
A video playback analysis application capable of instrumenting video content for the detection of dropped, repeated and out-of-sequence video frames
A video quality analysis application capable of analyzing video frames for image defects and rendering a quality score based on a perceptual model of the video format under test
A video encoder application for encoding test media in the required format
A video capture application for capturing display data from the DUT
Royce Fernald - Intel Corporation

16. July 2006
Test Setup
Royce Fernald - Intel Corporation

17. Video Content Preparation
July 2006
Video Content Preparation
Video content must be instrumented by the video playback analysis application before it can be used for this test
Instrumentation ensures content-independent VDER results
Uncompressed source content is first instrumented and then encoded in the desired test format before being loaded on the video traffic generator
Preparation can be performed in real-time if supported by the video traffic generator
Royce Fernald - Intel Corporation

18. Video Content Preparation Procedure
July 2006
Video Content Preparation Procedure
Royce Fernald - Intel Corporation

19. Video Content Analysis
July 2006
Video Content Analysis
After passing through the RF data connection, video content is captured from the DUT display and saved to a file (typically uncompressed)
The instrumented video capture file is examined for video delivery error and video quality defects by the video playback analysis application and the video quality analysis application
Analysis can be performed in real-time if supported by the video capture device
Royce Fernald - Intel Corporation

20. Video Content Analysis Procedure
July 2006
Video Content Analysis Procedure
Royce Fernald - Intel Corporation

21. July 2006
Calibration
Video performance tests require that the DUT support the video format, bitrate, network transport and streaming protocol used by the video traffic generator
Video format compatibility must be verified before the test begins
Video performance measurements can be affected by the capture device, so capture device performance must be verified with local video playback from the DUT
If there are too many errors detected during calibration, the capture device should not be used for video performance tests
Royce Fernald - Intel Corporation

22. Baseline Configuration
July 2006
Baseline Configuration
Maximum transmit power setting
QoS enabled with video traffic at high priority (no background traffic)
No security
No power management
Video traffic configured to model desired application (i.e. 8 Mbps MPEG2 for DVD-quality video)
Default DUT buffer size
No content encryption
Royce Fernald - Intel Corporation

23. Modifiers QoS disabled
July 2006
Modifiers
QoS disabled
QoS enabled with background traffic at low priority
MAC security
Power management enabled, different profiles
Alternate video bitrates and/or network transports
DUT buffer size
Video content encryption
Royce Fernald - Intel Corporation

24. Measurement Procedure
July 2006
Measurement Procedure
Load video content on the video traffic generator
Associate DUT with WLCP
Repeat the following steps for each attenuation or range increment:
Start the video capture operation on the capture device and then start the video stream to the DUT
After the specified test duration, stop the capture operation and then stop the video stream
Analyze the resulting video capture file with the video playback analysis application and the video quality analysis application to measure the video delivery error rate and the video quality score
Calculate MDI using metrics from the traffic analyzer
Royce Fernald - Intel Corporation

25. July 2006
Reported Results
The reported results should include the following items
Calibrated VDER, including the capture device baseline
Video Quality MOS and DUT MOS upper bound
MDI MLR and DF
Detailed information about the test configuration including the video format, network transport and video capture device
Sample Report:
Attenuation (dB) or Range (m)
Video Delivery Error Rate (Errors/minute)
Video Quality (MOS)
MDI – MLR
MDI – DF
Royce Fernald - Intel Corporation

26. July 2006
Sample Data
Compares video streaming performance of wireless adapters “A” and “B”
Conducted environment
Standard definition video (720x480 pixels)
MPEG2 transport stream
12 Mbps MPEG2 video
Constant bitrate
No audio
UDP video transport
Royce Fernald - Intel Corporation

27. July 2006
Sample Data – Adapter A
Royce Fernald - Intel Corporation

28. July 2006
Sample Data – Adapter B
Royce Fernald - Intel Corporation

29. Agenda July 2006 Month Year doc.: IEEE 802.11-yy/xxxxr0 Background
Proposal Framework
Test Output
Video Delivery and Video Quality
Measuring Video Delivery and Video Quality
The Media Delivery Index
Video Delivery, Video Quality and Wireless Performance
References
Test Description (doc.: /0964r0)
Introduction and Purpose
Resource Requirements
Test Setup
Video Content Preparation and Analysis
Calibration
Baseline Configuration
Modifiers
Measurement Procedures
Reported Results
Sample Data
Motion
Royce Fernald - Intel Corporation
John Doe, Some Company

30. Month Year
doc.: IEEE yy/xxxxr0
July 2006
Motion
Move to instruct the editor to incorporate the contents of document /0964r0 into the P draft.
Technical (75%) Y: N: A:
Royce Fernald - Intel Corporation
John Doe, Some Company