1. 1 SurroundSense: Mobile Phone Localization via Ambience Fingerprinting Ionut Constandache Co-authors: Martin Azizyan and Romit Roy Choudhury

2. 2 Context Pervasive wireless connectivity + Localization technology = Location-based applications

3. 3 Location-Based Applications (LBAs) For Example:  GeoLife shows grocery list when near Walmart  MicroBlog queries users at a museum  Location-based ad: Phone gets coupon at Starbucks iPhone AppStore: 3000 LBAs, Android: 500 LBAs

4. 4 Location-Based Applications (LBAs) For Example:  GeoLife shows grocery list when near Walmart  MicroBlog queries users at a museum  Location-based ad: Phone gets coupon at Starbucks iPhone AppStore: 3000 LBAs, Android: 500 LBAs Location expresses context of user  Facilitates content delivery

5. 5 Location is an IP address As if for content delivery

6. 6 Thinking about Localization from an application perspective…

7. 7 Emerging location based apps need place of user, not physical location Starbucks, RadioShack, Museum, Library Latitude, Longitude

8. 8 Emerging location based apps need place of user, not physical location Starbucks, RadioShack, Museum, Library Latitude, Longitude We call this Logical Localization …

9. 9 Can we convert from Physical to Logical Localization?

10. 10 Can we convert from Physical to Logical Localization? State of the Art in Physical Localization: 1.GPS Accuracy: 10m 2.GSMAccuracy: 100m 3.Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m State of the Art in Physical Localization: 1.GPS Accuracy: 10m 2.GSMAccuracy: 100m 3.Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m

11. 11 Widely-deployable localization technologies have errors in the range of several meters Widely-deployable localization technologies have errors in the range of several meters Can we convert from Physical to Logical Localization? State of the Art in Physical Localization: 1.GPS Accuracy: 10m 2.GSMAccuracy: 100m 3.Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m State of the Art in Physical Localization: 1.GPS Accuracy: 10m 2.GSMAccuracy: 100m 3.Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m

12. 12 Several meters of error is inadequate to logically localize a phone Physical Location Error

13. 13 Several meters of error is inadequate to logically localize a phone RadioShackStarbucks Physical Location Error The dividing-wall problem

14. 14 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

15. 15 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

16. 16 It is possible to localize phones by sensing the ambience Hypothesis such as sound, light, color, movement, WiFi …

17. 17 Sensing over multiple dimensions extracts more information from the ambience Each dimension may not be unique, but put together, they may provide a unique fingerprint

18. 18 SurroundSense Multi-dimensional fingerprint  Based on ambient sound/light/color/movement/WiFi Starbucks Wall RadioShack

19. 19 B A C D E Should Ambiences be Unique Worldwide? F G H J I L M N O P Q Q R K

20. 20 Should Ambiences be Unique Worldwide? B A C D E F G H J I K L M N O P Q Q R GSM provides macro location (strip mall) SurroundSense refines to Starbucks

21. 21 + + Ambience Fingerprinting Test Fingerprint Sound Acc. Color/Light WiFi Logical Location Matching Fingerprint Database = = Candidate Fingerprints GSM Macro Location SurroundSense Architecture

22. 22 Fingerprints Sound: (via phone microphone) Color: (via phone camera) Amplitude Values -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Normalized Count 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 Acoustic fingerprint (amplitude distribution) Color and light fingerprints on HSL space Lightnes s 1 0.5 0 Hue 0 0.5 1 0 0.2 0.4 0.6 0.8 1 Saturation

23. 23 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Moving

24. 24 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Moving Queuing

25. 25 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Queuing Seated Moving

26. 26 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Pause for product browsing Moving

27. 27 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Pause for product browsing Short walks between product browsing Moving

28. 28 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Walk more Moving

29. 29 Fingerprints Movement: (via phone accelerometer) CafeteriaClothes Store Grocery Store Static Walk more Quicker stops Moving

30. 30 Fingerprints Movement: (via phone accelerometer) WiFi: (via phone wireless card) CafeteriaClothes Store Grocery Store Static ƒ (overheard WiFi APs) Moving

31. 31 Discussion Time varying ambience  Collect ambience fingerprints over different time windows What if phones are in pockets?  Use sound/WiFi/movement  Opportunistically take pictures Fingerprint Database  War-sensing

32. 32 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

33. 33 Evaluation Methodology 51 business locations  46 in Durham, NC  5 in India Data collected by 4 people  12 tests per location Mimicked customer behavior

34. 34 Evaluation: Per-Cluster Accuracy Cluster No. of Shops 12345678910 4737455655 Accuracy (%) Cluster Localization accuracy per cluster

35. 35 Evaluation: Per-Cluster Accuracy Cluster No. of Shops 12345678910 4737455655 Accuracy (%) Cluster Localization accuracy per cluster Multidimensional sensing

36. 36 Evaluation: Per-Cluster Accuracy Cluster No. of Shops 12345678910 4737455655 Fault tolerance Accuracy (%) Cluster Localization accuracy per cluster

37. 37 Evaluation: Per-Cluster Accuracy Cluster No. of Shops 12345678910 4737455655 Accuracy (%) Cluster Localization accuracy per cluster Sparse WiFi APs

38. 38 Evaluation: Per-Cluster Accuracy Cluster No. of Shops 12345678910 4737455655 No WiFi APs Accuracy (%) Cluster Localization accuracy per cluster

39. 39 Evaluation: Per-Scheme Accuracy ModeWiFiSnd-Acc-WiFiSnd-Acc-Lt-ClrSS Accuracy70%74%76%87%

40. 40 Evaluation: User Experience Random Person Accuracy Average Accuracy (%) 0 10 20 30 40 50 60 70 80 90 100 1 0.9 0.8 0.7 0.6 0.5 CDF 0.4 0.3 0.2 0.1 0 WiFI Snd-Acc-WiFi Snd-Acc-Clr-Lt SurroundSense

41. 41 Economics forces nearby businesses to be different Not profitable to have 3 coffee shops with same lighting, music, color, layout, etc. SurroundSense exploits this ambience diversity Why does it work? The Intuition:

42. 42 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

43. 43 Limitations and Future Work Energy-Efficiency Localization in Real Time Non-business locations

44. 44 Limitations and Future Work Energy-Efficiency  Continuous sensing likely to have a large energy draw Localization in Real Time Non-business locations

45. 45 Limitations and Future Work Energy-Efficiency  Continuous sensing likely to have a large energy draw Localization in Real Time  User’s movement requires time to converge Non-business locations

46. 46 Limitations and Future Work Energy-Efficiency  Continuous sensing likely to have a large energy draw Localization in Real Time  User’s movement requires time to converge Non-business locations  Ambiences may be less diverse

47. 47 Contents SurroundSense Evaluation Limitations and Future Work Conclusion

48. 48 SurroundSense Today’s technologies cannot provide logical localization Ambience contains information for logical localization Mobile Phones can harness the ambience through sensors Evaluation results:  51 business locations,  87% accuracy SurroundSense can scale to any part of the world

49. 49 Questions? Thank You! Visit the SyNRG research group @ http://synrg.ee.duke.edu/