1. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 1 Video Gross Error Detector Video over Wireless Methodology Notice: This document has been prepared to assist IEEE 802.11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein. Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE 802.11. Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE 802.11 Working Group. If you have questions, contact the IEEE Patent Committee Administrator at.http:// ieee802.org/guides/bylaws/sb-bylaws.pdfstuart.kerry@philips.compatcom@ieee.org Authors:

2. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 2 Abstract Goals Video Gross Error Detector Overview Base Test Configuration Conducted Test Configuration System Requirements Calibration Test Procedure Modifiers Sample Report Caveats

3. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 3 Goals Tie GED subjective assessment results into objective performance metrics – measure acceptable user experience threshold for video streaming over devices under test Use conducted configuration to demonstrate video over wireless testing with the GED

4. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 4 Video GED Overview The Video Gross Error Detector is a high-level video performance analysis application Provides objective data on dropped, repeated or out-of- sequence frames Enables automated, quantitative, repeatable measurements of playback smoothness and frame rate stability Other video quality analysis tools (such as VQM) are used in addition to the GED to measure image quality

5. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 5 Platform Independence The GED supports any video playback solution that can be connected to a hardware capture device Since the GED operates directly on capture files, it can measure and compare the video performance of various network transports, operating systems, streaming applications, media players and compression formats The GED allows direct comparison between PC platforms and consumer electronics devices

6. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 6 GED Methodology Source clips are encoded with GED frame identifiers, a sequence of color blocks Video clips are played through the system under test and captured The GED reads the color blocks to locate dropped or repeated frames GED Encode

7. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 7 GED Processing 0. Source Material1. Marked Source Material 2. Compressed and Marked 4. GED Analysis GED Encode Video Encoder System Under Test 3. Capture Results GED Decode

8. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 8 Base Equipment Requirements Test controller Media server Ethernet switch Wireless AP Client system – desktop, laptop or CE device with video over wireless support Video capture system Display device

9. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 9 Base Test Configuration

10. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 10 Conducted Equipment Requirements Base configuration Variable attenuator RF shield boxes

11. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 11 Conducted Test Configuration

12. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 12 Media Server Requirements Typical desktop PC, laptop or capable PDA Video streaming software – TCP, UDP or RTP transports

13. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 13 Video Capture System Requirements Video capture adapter that supports the format of the display device (s-video, HDTV component, DVI, VGA, etc.) High-performance disk array for real-time capture of uncompressed video

14. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 14 Client System Requirements Desktop, laptop, PDA or CE device with video over wireless support Media player application that supports the streaming format of the media server

15. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 15 Media Preparation Encode each source clip with GED identifiers If testing a sequence of short clips, concatenate them with a video editor after the GED encode – this method is recommended because it allows testing with a wide range of video types Once encoded and assembled, compress the video clip in the desired format, such as MPEG-2 Copy the clips to the media server and the system under test (for local playback calibration)

16. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 16 Test Calibration Video capture hardware is not perfect and will sometimes introduce additional dropped or repeated video frames To measure the error introduced by the capture process, play the test video sequences locally on the DUT while capturing the results Analyze the capture file with the GED to determine the baseline error level Several hours of calibration should be performed for an accurate assessment of capture device errors

17. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 17 Test Parameters Video clip duration – 5 minutes GED pass/fail score – 30 errors per clip (Mean Opinion Score > 4.8)

18. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 18 Test Procedure Starting at the optimal value, increase attenuation in increments of 5 dB At each attenuation level, start video playback from the client device Once the client finishes initial buffering, start capturing video After the clip has finished streaming, stop the capture operation and analyze the capture output with the GED If the clip passes, increase attenuation by 5 dB If the clip fails, return to the last passing attenuation and increase in increments of 1 dB (record at least ten 1 dB steps above the breaking point) Conduct throughput tests for the ten attenuator settings

19. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 19 Test Modifiers Video resolution Compression format Bitrate Number and type of video and audio streams Network transport type and settings Server and client software AP configuration Client device configuration Wireless channel

20. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 20 Conducted Test Report The following fields should be included in the test report Attenuator setting Average throughput GED score Pass/fail results (pass = fewer than baseline plus capture system calibration)

21. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 21 Sample Report SD Video (480i59.94, 720x480) MPEG-2 10 Mbps CBR AC-3 audio 256 Kbps Media Server X, TCP streaming AP X Attenuation (dB)Throughput (Mbps)GED ScoreResult 8615.12Pass 8714.33Pass 8814.42Pass 8912.05Pass 9011.95Pass 9111.77Pass 9210.512Pass 939.9212Fail Client platform X, Wireless card Y, Media player Z g-only mode, channel 6 Capture device calibration: 2 errors Pass/Fail criterion: 32 errors

22. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 22 Caveats OTA video tests are less deterministic than conducted measurements Statistical methods must be used to characterize expected video performance in practical OTA environments

23. doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 23 References IEEE 802.11-06/0144r1 “Video over Wireless Methodology"