1. Link Layer Metrics Proposal
July 2005
Link Layer Metrics Proposal
Date:
Authors:
Notice: This document has been prepared to assist IEEE It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein.
Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE
Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures < ieee802.org/guides/bylaws/sb-bylaws.pdf>, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE Working Group. If you have questions, contact the IEEE Patent Committee Administrator at
Tom Alexander, VeriWave Inc.

2. July 2005
Introduction
Link layer (Layer 2, or MAC) metrics are secondary metrics
They do not directly affect the user experience, but contribute very strongly to primary metrics that do
Example: for VoIP, the R-value metric indicates voice quality, but it is calculated from secondary link layer metrics such as loss, latency, jitter
Also, link layer metrics provide strong indicators as to what is going wrong when measurements of primary metrics show problems
Both client and AP metrics are covered
IBSS clients as well as regular clients
Metrics for this clause are intended to be used in conjunction with test environments and methodologies defined in Clause 4
Example: the throughput test is defined so that it can be performed in a conducted environment (cabled), a fully shielded environment (screen room), or a Calibrated Over The Air environment (COAT)
Tom Alexander, VeriWave Inc.

3. Functional Model Simple functional model, consisting of DUT and tester
July 2005
Functional Model
Simple functional model, consisting of DUT and tester
Implementation details left up to test implementer AP
DUT
Client
DUT
Test
Software on
DUT
Wireless Media Interface
(Under Test)
Wireless Media Interface
(Under Test)
Secondary Wired or Wireless Interface
Wireless Media Interface
Wired Media Interface
Client
DUT
(NIC and
stack only)
Tester
Tester
Tester
Client DUT Testing (alternative)
AP/switch DUT Testing
Client DUT Testing (preferred)
Tom Alexander, VeriWave Inc.

4. Metrics Covered Metrics are divided into three categories
July 2005
Metrics Covered
Metrics are divided into three categories
Throughput, latency/timing, capacity
Throughput tests deal with rates
Unicast intra-BSS and ESS throughput, forwarding rate, loss
Multicast forwarding rate
Authentication and association rate
Power management mode throughput, forwarding rate, loss
Latency/timing tests deal with times
Unicast ESS latency
Failover time
Beacon timing
Capacity tests deal with amounts
Packet burst capacity
Association database capacity
Power-save buffer capacity
Tom Alexander, VeriWave Inc.

5. Unicast Throughput Metrics
July 2005
Unicast Throughput Metrics
Throughput is the most basic and fundamental performance metric
Unicast intra-BSS (air-to-air) throughput, forwarding rate, loss
Applicable to infrastructure clients, IBSS clients, APs
Measures the basic data transfer performance of the client or AP
Unicast ESS (air-to/from-DS) throughput, forwarding rate, loss
Applicable only to APs
Measures data transfer performance of AP when forwarding frames between wireless and wired media
Both metrics follow a simple, common methodology
Set up selected environment per Clause 4
Set up configuration parameters and test conditions
Run the specified traffic
Measure frames forwarded and frames lost over trial duration
Calculate and report throughput / forwarding rate / loss rate
Tom Alexander, VeriWave Inc.

6. Multicast Forwarding Rate
July 2005
Multicast Forwarding Rate
Multicast (DS-to-air) packet forwarding rate
Applicable only to APs
Quantifies ability of AP to cope with broadcast and multicast data streams (e.g., multicast video)
Methodology similar to that of unicast ESS throughput
Set up selected environment per Clause 4
Set up configuration parameters and test conditions
Run the specified multicast traffic to one or more test clients
Count the number of multicast frames that were successfully received by at least one client over the trial duration
Multicast frames are unacknowledged and not retried; hence cannot guarantee that all clients can receive all multicast frames
Calculate and report multicast forwarding rate
Tom Alexander, VeriWave Inc.

7. Authentication and Association Rate
July 2005
Authentication and Association Rate
Rate at which AP can carry out authentication and association functions
Significant for applications such as VoIP (can impact call setup times)
See /r0 for an example
Particularly significant when i Authentication and Key Management is being used
Methodology uses a number of test clients
Set up selected environment per Clause 4, and set up configuration parameters and test conditions
Cause test clients to authenticate and associate with DUT at a given rate
Measure the total time required for all clients to authenticate and associate
If one or more clients fail to connect, the rate is reduced
Successful associations are verified with data transfer
Calculate and report authentication/association rate
Tom Alexander, VeriWave Inc.

8. Power Management Mode Throughput
July 2005
Power Management Mode Throughput
Measures client data throughput performance while in power-save mode
Applicable to both IBSS and infrastructure BSS clients (but not APs)
Indicates ability of mobile or handheld clients to maintain performance while still conserving battery power
Can be used as a secondary metric for battery life metrics
Particularly useful for quantifying e QoS enhancements
802.11e introduces advanced power-save features
Measures throughput maintenance when these power-save features are used
Methodology similar to unicast throughput metrics
Set up selected environment per Clause 4, and set up configuration parameters and test conditions
Place the DUT in power-save mode
Run the specified traffic
Measure frames forwarded and frames lost over trial duration
Calculate and report throughput / forwarding rate / loss rate
Tom Alexander, VeriWave Inc.

9. Unicast Latency and Latency Variation
July 2005
Unicast Latency and Latency Variation
Measures latency and latency variation (per RFC 1242)
Applicable to APs
Key secondary metric for all delay-sensitive traffic parameters
Latency and latency variation measured under different offered loads
Methodology
Set up selected environment per Clause 4, and set up configuration parameters and test conditions
Generate and run time-stamped traffic (time-stamped at transmitter and receiver)
Measure instantaneous delay through DUT by subtracting timestamp of transmitted packet from timestamp of received packet
Calculate and report average latency and maximum latency variation
Minimum and maximum latency could also be reported
Tom Alexander, VeriWave Inc.

10. July 2005
Failover Time
Measures time required for client to transition between APs
Applicable to APs and clients
Assumes an abrupt transition – not roaming time!
Enables the calculation of availability and downtime for enterprise applications
Measures time required for clients to transition to a co-located backup AP when a primary AP fails or is powered off
Methodology
Set up selected environment per Clause 4, and set up configuration parameters and test conditions
For AP DUTs, use test clients; for client DUTs, use two test APs
Generate and run data traffic between client(s), and force them to abruptly transition from AP to AP
Measure time for which test data traffic is interrupted during the transition period
Calculate and report failover time
Tom Alexander, VeriWave Inc.

11. July 2005
Beacon Timing
Measures accuracy of beacon timing for an AP or IBSS client
Significant for infrastructure or IBSS clients in power-save mode
Power-save clients need to wake up and listen for beacons at the Target Beacon Transmission Time, to determine if traffic is available
Irregular beacon times can lead to lost traffic, delayed traffic or wasted battery power
Secondary metric for user-level metrics such as battery life and power-save mode responsiveness
Methodology: requires only a test traffic analyzer
Set up selected environment per Clause 4, and set up configuration parameters and test conditions
Configure the DUT to generate beacons
Measure the time between beacons received by the analyzer, and also capture the Beacon Interval from the beacons
Calculate and report measured average beacon interval, beacon interval variation, and beacon interval accuracy with respect to the advertised Beacon Interval parameter
Tom Alexander, VeriWave Inc.

12. July 2005
Packet Burst Capacity
Measures ability of AP to deal with bursts of back-to-back packets
Useful for quantifying throughput and frame loss of AP in heavily loaded networks with many clients
A burst of packets received on the wired side of an AP may have to be buffered, and transmitted on the wireless side during quiet periods
Buffer capacity in the AP thus determines higher-layer issues such as TCP retransmissions, TCP window synchronization, etc.
Methodology similar to that for throughput tests
Set up selected environment per Clause 4, and set up configuration parameters and test conditions
Generate traffic consisting of bursts of frames separated by an Inter-Burst Gap (IBG)
Run the specified traffic at different offered loads
Measure maximum length of burst for zero loss at a given offered load
Calculate and report burst capacity
Tom Alexander, VeriWave Inc.

13. Association Database Capacity
July 2005
Association Database Capacity
Maximum number of client connections that can be supported by AP
Measures ability of AP/switch to support large numbers of low-bandwidth clients
For example: hotspots, conference rooms, outdoor areas
Also useful for determining upper limit on number of test clients to be used in measurements of other metrics (e.g., association rate)
Methodology similar to association rate test, but focuses on number of clients rather than a rate
Set up selected environment per Clause 4, and set up configuration parameters and test conditions
Cause test clients to authenticate and associate with DUT at a low rate
Measures the maximum number of clients that can successfully associate and remain connected
Successful associations are verified with data transfer
Calculate and report association database capacity
Tom Alexander, VeriWave Inc.

14. Power Save Buffer Capacity
July 2005
Power Save Buffer Capacity
Measures ability of AP to support large numbers of clients in power-save mode
Indicates power-management mode buffer capacity within AP
A large power-management buffer is essential for assuring high throughput, low packet loss and high responsiveness when dealing with sleeping clients
Particularly significant for applications such as phones and PDAs that must conserve battery power by sleeping as often as possible
Can be used as a secondary metric for mobility tests
Methodology
Set up selected environment per Clause 4, and set up configuration parameters and test conditions
Place the test clients in power-save (sleep) mode, so AP cannot forward packets to them
Send the specified traffic to the test clients
Wake up test clients and cause them to receive frames (via PS-Poll)
Measure number of frames forwarded; if frames were lost, decrease the number of frames sent to sleeping clients until zero frame loss occurs
Calculate and report power-save buffer capacity
Tom Alexander, VeriWave Inc.

15. July 2005
Conclusions
A set of 11 link layer metrics have been proposed, together with draft text
These metrics are intended to be used in conjunction with test environment setup and calibration methodologies in Clause 4
The proposal / draft-text contains:
Objective of test and metric, and expected correlation to user experience
Relevant configuration parameters, test conditions and modifiers
Test procedure
Results reporting
Tom Alexander, VeriWave Inc.

16. July 2005
Motion
Move to adopt the contents of document /r0 into the P draft.
Technical (75%) Y: N: A:
Tom Alexander, VeriWave Inc.