doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 1 Video Gross Error Detector Video over Wireless Methodology Notice: This document has been prepared to assist IEEE It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein. Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE Working Group. If you have questions, contact the IEEE Patent Committee Administrator at. Authors:

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 2 Abstract Goals Video Gross Error Detector Overview Base Test Configuration Conducted Test Configuration System Requirements Calibration Test Procedure Modifiers Sample Report Caveats

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 3 Goals Tie GED subjective assessment results into objective performance metrics – measure acceptable user experience threshold for video streaming over devices under test Use conducted configuration to demonstrate video over wireless testing with the GED

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 4 Video GED Overview The Video Gross Error Detector is a high-level video performance analysis application Provides objective data on dropped, repeated or out-of- sequence frames Enables automated, quantitative, repeatable measurements of playback smoothness and frame rate stability Other video quality analysis tools (such as VQM) are used in addition to the GED to measure image quality

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 5 Platform Independence The GED supports any video playback solution that can be connected to a hardware capture device Since the GED operates directly on capture files, it can measure and compare the video performance of various network transports, operating systems, streaming applications, media players and compression formats The GED allows direct comparison between PC platforms and consumer electronics devices

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 6 GED Methodology Source clips are encoded with GED frame identifiers, a sequence of color blocks Video clips are played through the system under test and captured The GED reads the color blocks to locate dropped or repeated frames GED Encode

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 7 GED Processing 0. Source Material1. Marked Source Material 2. Compressed and Marked 4. GED Analysis GED Encode Video Encoder System Under Test 3. Capture Results GED Decode

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 8 Base Equipment Requirements Test controller Media server Ethernet switch Wireless AP Client system – desktop, laptop or CE device with video over wireless support Video capture system Display device

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 9 Base Test Configuration

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 10 Conducted Equipment Requirements Base configuration Variable attenuator RF shield boxes

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 11 Conducted Test Configuration

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 12 Media Server Requirements Typical desktop PC, laptop or capable PDA Video streaming software – TCP, UDP or RTP transports

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 13 Video Capture System Requirements Video capture adapter that supports the format of the display device (s-video, HDTV component, DVI, VGA, etc.) High-performance disk array for real-time capture of uncompressed video

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 14 Client System Requirements Desktop, laptop, PDA or CE device with video over wireless support Media player application that supports the streaming format of the media server

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 15 Media Preparation Encode each source clip with GED identifiers If testing a sequence of short clips, concatenate them with a video editor after the GED encode – this method is recommended because it allows testing with a wide range of video types Once encoded and assembled, compress the video clip in the desired format, such as MPEG-2 Copy the clips to the media server and the system under test (for local playback calibration)

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 16 Test Calibration Video capture hardware is not perfect and will sometimes introduce additional dropped or repeated video frames To measure the error introduced by the capture process, play the test video sequences locally on the DUT while capturing the results Analyze the capture file with the GED to determine the baseline error level Several hours of calibration should be performed for an accurate assessment of capture device errors

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 17 Test Parameters Video clip duration – 5 minutes GED pass/fail score – 30 errors per clip (Mean Opinion Score > 4.8)

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 18 Test Procedure Starting at the optimal value, increase attenuation in increments of 5 dB At each attenuation level, start video playback from the client device Once the client finishes initial buffering, start capturing video After the clip has finished streaming, stop the capture operation and analyze the capture output with the GED If the clip passes, increase attenuation by 5 dB If the clip fails, return to the last passing attenuation and increase in increments of 1 dB (record at least ten 1 dB steps above the breaking point) Conduct throughput tests for the ten attenuator settings

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 19 Test Modifiers Video resolution Compression format Bitrate Number and type of video and audio streams Network transport type and settings Server and client software AP configuration Client device configuration Wireless channel

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 20 Conducted Test Report The following fields should be included in the test report Attenuator setting Average throughput GED score Pass/fail results (pass = fewer than baseline plus capture system calibration)

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 21 Sample Report SD Video (480i59.94, 720x480) MPEG-2 10 Mbps CBR AC-3 audio 256 Kbps Media Server X, TCP streaming AP X Attenuation (dB)Throughput (Mbps)GED ScoreResult Pass Pass Pass Pass Pass Pass Pass Fail Client platform X, Wireless card Y, Media player Z g-only mode, channel 6 Capture device calibration: 2 errors Pass/Fail criterion: 32 errors

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 22 Caveats OTA video tests are less deterministic than conducted measurements Statistical methods must be used to characterize expected video performance in practical OTA environments

doc.: IEEE /0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 23 References IEEE /0144r1 “Video over Wireless Methodology"