doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 1 Dell IEEE TGT Output Notice: This document has been prepared to assist IEEE It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein. Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE Working Group. If you have questions, contact the IEEE Patent Committee Administrator at. Date: Authors:

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 2 Agenda Introduction and Target Audience Dell’s vision of IEEE TGT output Comparison with Existing Benchmarks Conclusion

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 3 Introduction Scope and Purpose of TGT (in PAR): –“The scope of the project is to provide a set of performance metrics, measurement methodologies, and test conditions to enable measuring and predicting the performance of WLAN devices and networks at the component and application level.” –“The purpose of the project is to enable testing, comparison, and deployment planning of WLAN devices based on a common and accepted set of performance metrics, measurement methodologies and test conditions” –“The target audience for the Recommended Practice includes developers of chipsets, components, equipment and software that uses or must interact with wireless equipment, as well as users of equipment, including system installers, IT managers and test laboratories.” Key Requirements for defining a proposal – Test Environments/Conditions – Performance Metrics – Measurement Methodology – Correlation with end-user experience Usage + Environment Metrics Primary/Secondary Methodology Test Environment Test Procedure Test Template Analyze/Validate /Correlate

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 4 TGT Stakeholders, Target Audience TGT Output Secondary Metrics (examples) TGT Output Primary Metrics (examples) System Integrators, Hardware resellers, Installers, IT managers, Test Labs End users, System Installers, Magazine Reviewers, Test Laboratories, IT Managers, etc. Receiver Sensitivity, Radiated Power, Forwarding Rate, SNR, Loss Rate, ARP, etc. Throughput, Range, Dropped Frame Rate, Image quality, Voice Quality, Bandwidth utilization (call capacity), etc. TGT defined Metrics, Methodology & Test Conditions Silicon Providers, Software Providers, Test Systems, ODMs System Integrators, Hardware Resellers, OEMs

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 5 Dell’s Vision of TGT Output How Dell expects the IEEE TGT work to be used –IEEE TGT Output = Recommended Practices for a Wireless Benchmark that reflects the end-user’s wireless experience –“Somebody else” will REQUIRE the use of TGT recommended practice as a wireless benchmark –“Somebody else” will define what are acceptable levels when this benchmark is run

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 6 Dell’s Vision of TGT Effort Test Environments –TGT output will recommend that measurements be made in several test environments (conducted, chamber, indoor LOS/NLOS, outdoor, etc.), with correlation between the environments Performance Metrics –TGT output will include primary as well as secondary metrics that affect end-user experience for Data, Streaming, Latency-sensitive usage cases Measurement Methodology –TGT output will describe the measurement steps for benchmarking primary metrics (user, application level) as well as secondary metrics (MAC, PHY, application level). The benchmark results will be reported in a standard format. The results will include details about the equipment and setup used for running the benchmark. Correlation between primary/secondary metrics

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 7 Primary and Secondary Metrics Primary metrics –They are not limited to MAC/PHY level metrics -- user/application level primary metrics are also included in the purview –They are not limited to metrics that can only be directly measured -- primary metrics that are inferred as a result of correlations are also included in the purview Secondary metrics –They are not limited to MAC/PHY level metrics –They can be directly measured – usually not inferred Primary and Secondary metrics are both important for wireless benchmarking –Designation of primary or secondary does not imply importance of the metric –Primary metrics are those that are directly perceivable by users regardless of “level/layer” –Secondary metrics are those that affect primary metrics –Secondary metrics also provide insights that aid product development, sensitivity analysis for performance, debugging, etc.

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 8 TGT Correlation Needs 1. Analysis & Validation Correlation of measurements between Test Environments Chamber OTA Test Environment Outdoor OTA Test Environment Conducted Test Environment Indoor (N)LOS OTA Test Environment Models Analysis Models Receiver Sensitivity, Radiated Power, Forwarding Rate, SNR, Loss Rate, ARP etc. Throughput, Range, Dropped Frame Rate, Image quality, Voice Quality, Bandwidth utilization (call capacity) etc. TGT Output Secondary Metrics (examples) TGT Output Primary Metrics (examples) Correlation between Primary & Secondary Metrics 2.

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 9 Dell’s Vision of TGT Correlation Correlation is not “an exercise left to the reader” – it is a key value of TGT output without which the applicability of TGT output will be curtailed Pull in whatever models that we can leverage and use them in developing the wireless benchmark recommendations output of TGT The correlation effort is used for two distinct sets of measurements: 1.The exact same metrics measured in different test environments –This correlation will establish the relative degree of variability of measurements in different test environments –The correlation effort may use models to validate/analyze measurements 2.Primary metrics and secondary metrics measured in the same test environment –Primary metrics that are not directly measured need correlation with measured values of secondary metrics This correlation may use models to calculate the values of primary metrics These models may use measurements of secondary metrics as inputs Validation is still needed to ensure that the output using models reflects accurately –Most secondary metrics will need the correlation to primary metrics This correlation effort will help reflect the end-user’s wireless experience in term of secondary metrics measurements

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 10 Use of Models in TGT Existing models that are appropriate for wireless and the three usage cases should be identified and used by TGT TGT output must show how those models are used for the purpose of the wireless benchmark it recommends –Show how the models result in good correlation between environments –Show how the models result in good correlation/inferences for primary metrics TGT does not preclude any of its customers from using alternate models of their choice

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 11 Analogy: Battery Life Benchmark Purpose: Measure battery life using BAPCO Mobilemark 2004 Primary Performance Metrics: Secondary Performance Metrics Test Conditions: Fix Screen brightness (nits), screen resolution, etc. Set power profile: laptop/portable, suspend, screen savers, etc. Disable background/start-up applications … etc. Methodology: Run various applications with wait periods and measure system performance, response time Log battery life while these applications are running Generate standard report Correlation: This benchmark is able to directly measure Primary (user/application level) Metrics that encapsulate the effects of Secondary Metrics without actually measuring the Secondary Metrics, so no correlation is necessary Battery Life (minutes)Actual Battery Charge Capacity etc. System PerformanceProcessor/Memory Utilization etc. Response Time (seconds)Hard drive speed, Cache Latency etc.

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 12 Wireless Benchmark for Data Usage Purpose: Throughput and Range performance Primary Performance Metrics: Secondary Performance Metrics Test Condition: Background Noise (dB), environmental characteristics etc. Specify variability Turntable, Orientation of DUT… etc.. Methodology: Run applications to measure throughput performance Repeat at different range to determine range performance Generate standard report Correlation: This benchmark is able to directly measure Primary Metrics (Throughput) that encapsulate the effects of Secondary Metrics without actually measuring the Secondary Metrics (Forwarding Rate, Loss Rate, etc.), no correlation is necessary Throughput (Mbps)MAC Forwarding Rate, Packet Loss Rate Range (meters)Receiver Sensitivity, ARP, Transmit Power

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 13 Wireless Benchmark for Streaming Usage Purpose: Measure wireless performance of streaming media content Primary Performance Metrics: Secondary Performance Metrics Test Condition: Background Noise (dB), environmental characteristics etc. Specify variability Turntable, Orientation of DUT… etc.. Methodology: Generate a data stream of “x” Mbps (e.g Mbps for ATSC-compliant HDTV MPEG-2 files) Measure the throughput & note the number of times the throughput falls below a certain threshold Repeat at different range to determine range performance Generate standard report Correlation: In this case Secondary Metrics (Throughput, etc.) are being measured, correlation to Primary Metrics (Video quality, etc.) is required Video Quality (Jerkiness, Blockiness)Throughput, Packet Drop/Error Rate, etc. Range (meters)Receiver Sensitivity, ARP, Transmit Power

doc.: IEEE /0177r0 Submission March 2005 Pratik Mehta - DellSlide 14 Conclusions TGT needs to provide output that can be used by other organizations to create wireless benchmarks Correlation, as described, is essential to enable this Identifying Primary and Secondary metrics at user/application levels is essential Choices in approach –Measure Primary metrics directly only –Measure Secondary metrics only and show the correlation to Primary metrics clearly –Do a combination of both the above