1. Obtaining In-Context Measurements of Cellular Network Performance Aaron Gember, Aditya Akella University of Wisconsin-Madison Jeffrey Pang, Alexander Varshavsky, Ramon Caceres AT&T Labs 1

2. Performance During User Activity 2 How does it work when you’re using it?

3. In-Context Measurements 3 InteractionTime, place, & speed when the network is used Limit to specific contexts Device model & OS version Want to accurately reflect the range of performance experienced by users Representative distribution of contexts

4. Use Cases 4 Compare cellular network providers Evaluate effect of network changes Find cause of poor network performance

5. 5 How do we capture in-context measurements of cellular network performance?

6. Existing Approaches Field Testing Network-based Passive Analysis Self-initiated Reporting 6 1)Difficult to determine or control context 2)Difficult to eliminate confounding factors 1)Requires manual user intervention 2)Most users only report problems 1)Limited range of contexts 2)May not accurately reflect usage patterns

7. Crowdsource active measurements Deploy to 12 volunteers Our Contributions Empirical StudyMeasurement System Measurements depict performance while user is active 7 Network data from 20,000 subscribers 100s of controlled experiments

8. active idle active idle Active vs. Idle Devices LatencyLoss 16ms lower when idle 6% less when idle Measurements on idle devices may overestimate performance 8 active idle active idle

9. active idle What causes the performance differences? – Time of day – Coarse geo-location – Signal strength – Other low-level factors Active vs. Idle Devices Signal Strength No correlation 9 active idle

10. Impact of Low-Level Factors Many low-level factors may affect performance – Difficult to account for – Determined by device’s physical context 2)What aspects of a device’s physical context contributes to the observed differences? – Environment – Device position 10

11. Impact of Physical Context iPerf and ping from devices we control – Vary environment (in/out, location, speed) and position relative to user – ≥ 5 measurements in each position (round-robin) and environment 11

12. Impact of Environment Location – Three offices in the same building Stationary vs. moving – Walking outdoors: 950Kbps – Stationary outdoors: 1540Kbps 12 LocationThroughputLatency Indoors 1a1491 Kbps416 ms Indoors 1b 98 Kbps475 ms Indoors 1c1842 Kbps412 ms Confirm prior results: environment changes may cause performance differences

13. Impact of Device Position 13 > 350Kbps difference in some locations Latency > 15ms difference in some locations Devices in different positions may experience difference performance Throughput

14. What causes the performance differences? – Cell sector – Signal strength – Small scale fading Impact of Device Position 14 Signal stengthThroughput Loc 1a Indoors Hand Pocket

15. Measurement System Crowdsource in-context active measurements – Android-based prototype run by 12 volunteers Throughput measurements gathered – Ground Truth: screen on; no network activity – In-Context – Random: every 2-4 hours 15

16. Measurement Accuracy Do in-context measurements gathered by our system accurately quantify experienced performance? 16 In-Context = Ground Truth for 18 hours Accurately quantify performance experienced by users interacting with device

17. Measurement Accuracy Do random measurements quantify experienced performance? 17 Random differs by > 1Mbps Analyses which ignore context will not accurately quantify experienced performance

18. Conclusion Quantify performance experienced when users are interacting with their device in specific contexts 18 Empirical Study Idle devices: 6% less loss; 16ms lower latency Physical context change: > 350Kbps difference; > 15ms difference Measurement System Android-based prototype deployed to 12 volunteers Measurements depict performance experienced while user is active

19. Related Work Cellular measurement tools – Mark the Spot, MobiPerf, 3G Test, WiScape Automated active measurement systems – NIMI, Scriptroute, DipZoom, ATEM, CEM Cellular network performance studies – Latency, TCP performance, fairness, etc. 19

20. Impact of Context Which contextual factors are most predictive of cellular network performance? 20 Cell sector Phone model Location area Hour of day Month Connection type Indoors/outdoors Movement speed Signal strength Most InfluentialLeast Influential

21. Measurement Opportunities 21

22. Measurement Service Decision Process 22

23. Measurement Service Benchmarks Device position change detection Energy overhead 23 EventCorrect False Negatives False Positives Desk → Hand70- Web browsing5-2 Hand → Pocket70- In pocket7-0 Pocket → Hand70- Hand → Desk61- FunctionalityEnergy Consumed in 1 Min Idle0 Joules Active Monitoring0.44 Joules Environment Monitoring (with GPS)16.85 Joules Environment Monitoring (no GPS)0.15 Joules