Framework, Usages, Metrics Proposal for TGt Pratik Mehta, Fahd Pirzada – Dell Paul Canaan – Intel Amer Hassan, Don Berry – Microsoft September 2004

TGt Doc: wpp 2 Excerpt from Nov 2003 Presentation – Schedule Plan Work Plan / Timeline July 2003Initial presentation to IEEE WNG committee. August 2003Approved as study project by IEEE EMC Society SDCom Aug.-Nov. 2003Conference calls surveying state-of-the-art & preparing for 2 nd WNG presentation. Nov. 2003Committee passes 40 participants, 27 orgs Nov nd Presentation at WNG and request for approval as IEEE study project The milestones set out in Nov 2003 are on track … WPP/TGt have been meeting expectations … Presenter: Stephen Berger, Reference doc.: wng 

TGt Doc: wpp 3 Excerpt from Nov 2003 Presentation – Schedule Plan Work Plan / Timeline Nov.-Dec. 2003Analysis of state-of-the-art Initial definition of tasks and deliverables Jan st direct meeting Feb.-June 2004Development of study project report. Drafting of PAR & 5CRIT July 2004Submission of PAR & 5CRIT Presentation of report March st Letter Ballot Presenter: Stephen Berger, Reference doc.: wng The milestones set out in Nov 2003 are on track … WPP/TGt have been meeting expectations …    

TGt Doc: wpp 4 Excerpt from Nov 2003 Presentation – Scope ISIS NOT Predicting coverage, throughput, directivity performance, latency, errors Measurements of radio component performance Asynchronous and isochronous channels Regulatory issues Measurement correlation, precision & repeatability Purely theoretical analysis Correlation to differing deployment scenarios Purely conducted tests Identify and comprehend the impact of variables like Antenna gain and pattern, radio specs, environmental and system noise, band crowding Designation of acceptable performance Proposed Scope WPP/TGt continues to focus on the scope set out in Nov 2003 … Presenter: Pratik Mehta, Reference doc.: wng

TGt Doc: wpp 5 Excerpt from Nov 2003 Presentation – Metrics WPP/TGt continues to discuss metrics highlighted in Nov 2003 by Broadcom… Presenter: Jason Trachewsky et al, Reference doc.: wng Deciding what parameters are to be considered is the challenge. How do we transform user perception of performance into a set of repeatably- measurable quantities? – Throughput and Range – Visibility of APs – Delays in association – Host CPU utilization – Ability to roam without loss of connections – Etc. Slides of that presentation delve further into metrics that are still relevant … … TGt can decide what makes sense …

TGt Doc: wpp 6 Excerpts from Presentations – Metrics WPP/TGt continues to discuss usages metrics… Presenter: Bob Mandeville, Reference doc.: wng Roaming is important to consider Roaming brings in many variables and causes that impact performance More recent presentation on metrics by Bob … – Maximum Forwarding Rate – FWMOL (Fwd rate at max offered load) – Frame Loss, Frame Loss Rate – Latency – Jitter – Association Capacity, Association Rate – Rate versus Range Presenter: Bob Mandeville, Reference doc.: wpp

TGt Doc: wpp 7 Excerpt from Recent WPP/TGt Presentation – Framework WPP/TGt has discussed a framework approach … below is as highlighted by Intel… Presenter: Paul C. Canaan, Reference doc.: wpp Framework for WPP Test Environment – Come to consensus on environment types (NLOS, LOS, Conductive, etc) – Develop a mechanism to characterize the environment – Propose recommended practices for the environmental set up Metrics – Define metrics relevant to characterize the environment – Identify metrics that impact WLAN performance as well as corresponding methodology to obtain these metrics – Devise a mechanism to predict the metric performance Test Case Template – Come to consensus on the necessary metrics to characterize the environment that should be reported in a test template – Define test case scenarios for each metric – Determine recommended practice for reporting/analyzing of the results Usage – Come to consensus on usage scenarios that represent the WLAN traffic pattern (file transfers, streaming media, connectivity, voice over IP, etc) – Categorize which metrics have the greatest impact on the performance of those usage scenarios

TGt Doc: wpp 8 Interim Summary TGt is focusing on the right approaches TGt is also meeting deliveries that were highlighted 9 months ago TGt has the right constituencies and experts represented … In short, TGt is a healthy group, and can move forward and deliver There is a call for urgency in the short-term to address some key industry needs – OEM needs during product development – Reviewer/Publisher needs for product reviews – Guidelines for system integrators The following slides provide a proposal to move forward

TGt Doc: wpp 9 Restatement of TGt Framework Usage + Environment Metrics Sub-metrics Methodology Test Environment Test Procedure Test Template Usage + Environment  Metrics  Methodology  Test Template …  Prediction Use Measurements To Predict

TGt Doc: wpp 10 Definition of Terms in TGt Framework Usage + Environment Metrics Sub-metrics Methodology Test Environment Test Procedure Test Template Usage + Environment  Metrics  Methodology  Test Template …  Prediction Use Measurements To Predict Usage and environment of end-user Metrics are the key performance indicators as perceived by users – for the usages they employ in their environments (above) Sub-metrics are drivers of those metrics Sub-metrics help understand what affects the metrics Methodology is the way to measure metrics and sub-metrics Test Environment is a model of the user’s environment Test Procedure is the set of steps to perform measurements Test Template is the content and format used to report the measurements Output from Methodology and Measurements are used to predict performance

TGt Doc: wpp 11 Proposal – Focus on three Cases Attributes common to all three usages to consider: No. of Clients; No. of APs; Stationary/Roaming; Encryption/Authentication Usage + Environment  Metrics  Methodology  … 2. Streaming Media Apps + Home, Corporate Environment Metrics: All of 1, Frames dropped, Image quality Sub-Metrics: TCP Re-transmissions, Packet errors, Latency, AV synchronization, Control Channel (Gaming, Multicast), Image quality (Jitter, Blockiness, Blur), Bandwidth stability, Buffering (pre-roll buffer, jitter buffer) Methodology: Unicast, Multicast, Real-time streaming, Stored content, Decoder/Encoder characteristics, Network protocols (TCP, UDP), Buffering Number of streams, Apps: Bobsled, VideoLAN, Real server. 1. Data Oriented Applications + Home, Corporate Environment Metrics: Throughput, Range, Directionality Sub-metrics: Path loss, EVM measurements, Interference, Multipath, Noise, SNR, Receive sensitivity, Transmit power Methodology: Chariot scripts – Chamber, Indoor Web-downloads – Indoor File-Transfers, File-Sharing – Indoor Rotating tables 3. Latency Sensitive Apps + Home, Corporate Environment Metrics: All of 1, Latency, Jitter Sub-Metrics: MOS/PESQ, PSQM, One-way delay, Packet loss Methodology: Decoder/Encoder characteristics, Network protocols (H.323, P2P, SIP) Apps: Netmeeting, Skype, Vonage Each of these Usage cases will need to be partitioned … (next slide shows example for #1)

TGt Doc: wpp 12 Partitioning Approach for Usage Case #1 1. Data Oriented Applications + Home, Corporate Environment Chariot Throughput + Indoor, Chamber Environment Metrics: Throughput, Range, Directionality Sub-Metrics Path loss, EVM measurements, Interference, Multipath, Noise, SNR, Receive sensitivity, Transmit power Methodology: ? Web Downloads + Indoor Environment Metrics: Latency, Throughput, Range Sub-Metrics Path loss, EVM measurements, Interference, Multipath, Noise, SNR, Receive sensitivity, Transmit power Methodology: ? Usage + Environment  Metrics  Methodology  Test Template … File transfer, File sharing + Indoor Environment Metrics: Throughput, Range Sub-Metrics Path loss, EVM measurements, Interference, Multipath, Noise, SNR, Receive sensitivity, Transmit power Methodology: ?

TGt Doc: wpp 13 Similar Partitioning is needed for #2 and #3 To be filled out

TGt Doc: wpp 14 Proposed Timeline for TGt Framework Adoption Pre-Berlin 9 th Sep’04 Usage Case #1 Nov’04 Letter Ballot Mar’05 Framework Adoption Report In Berlin Sep’04 Usage Case #2 Usage Case #3 Jan’05 Metrics/Sub-metrics, Methodology development

TGt Doc: wpp 15 Next Steps Get agreement on the framework and approach Get agreement that the three usage cases presented are the one we can focus on Align in Berlin meeting with wider audience in TG Develop the three usage cases further – Sanitize and update the metrics and sub-metrics – Hone in on the Methodology components, test environment, procedure, etc. – Finalizing the test case templates for reporting and analysis Usable outputs for Usages #1, #2, #3 (see timeline) Draft recommended practices specification – Include definitions and terminology

Backup

TGt Doc: wpp 17 Parking Lot for Metrics and Sub-metrics ApplicationTransportData LinkPhysical Layer Performance metrics by OSI model layers Proposing a “parking-lot” list of parameters – need to determine whether they are metrics or sub- metrics, or part of the methodology, and which of the usage cases they apply to …