1. Location-Aware Computing John Krumm Microsoft Research Redmond, Washington, USA

2. Thank You Toshio Hori Takeo Kanade Chie Nakamura Digital Human Research Center

3. Location

4. Why Bother to Sense Location? Find conference room near me Who is in this meeting with me? Where are the people who are supposed to be here? If I’m in conference room, don’t allow cell phone, alerts, or IM How long will it take to get from here to next appointment? Route planning in strange buildings

5. No Really, Why Bother? Remind me when I’m near a certain customer Where are my kids, buddies, colleagues? Index my documents (email, photos) by location Change default printer and network settings based on location Electronic graffiti, e.g. “There’s a better Thai restaurant one block north.”

6. Why Not Use GPS? Does not work indoors Needs view of satellites

7. Location Sensing Hazas, Scott, Krumm, “Location-Aware Computing”, IEEE Computer Magazine, February 2004. Rosum, promising but where is it going? Indoor/outdoor, coverage spreading Still a hard problem, can give much more than location (good & bad)

8. Beyond Location sensors → location → context Patterson, Liao, Fox, Kautz, “Inferring High-Level Behavior from Low-Level Sensors”, 2003 Use GPS tracking to infer user’s mode of transportation as (car, bus, walk) Sparacino, “Sto(ry)chastics: A Bayesian Network Architecture for User Modeling and Computational Storytelling for Interactive Spaces”, 2003 Use indoor location sensing in MIT museum to classify visitor into (greedy, busy, selective)

9. Outline Introduction Video Tracking Active badge – SmartMoveX Coarse Wi-Fi location – “Here I Am” Fine Wi-Fi location – L OCADIO SPOT Wristwatch Location – RightSPOT Wi-Fi proximity – NearMe “Longhorn” Location Service – Sensor Fusion

10. Video Tracking EasyLiving Project Steve Shafer Barry Brumitt Steve Harris Brian Meyers Greg Smith Mike Hale John Krumm

11. Video Tracking Is Very … … accurate (centimeters) … easy for user (no devices to carry) … hard to set up (camera calibration) … hard to get right (live demos still rare) … CPU intensive (one PC per camera) … intrusive Research can solve

12. Outline Introduction Video Tracking Active badge – SmartMoveX Coarse Wi-Fi location – “Here I Am” Fine Wi-Fi location – L OCADIO SPOT Wristwatch Location – RightSPOT Wi-Fi proximity – NearMe “Longhorn” Location Service – Sensor Fusion

13. SmartMoveX Transmitter Receiver Microsoft Research’s entry into the active badge space (along with Xerox PARC, UW, Intel, AT&T Cambridge, MIT, etc.) Hardware: Lyndsay Williams (Microsoft Research Cambridge UK) Software: John Krumm & Greg Smith (Microsoft Research Redmond) Multiple receivers for position triangulation from signal strengths

14. Receiver Network DB PC RX PC RX

15. Graph Algorithm Compute path instead of single locations Constrain path to allowable routes Process with Hidden Markov Model (HMM – same as used for speech recognition) Average error 3.05 meters Constraints make things easier

16. SmartMoveX Evaluation Good Cheap hardware Uses existing network infrastructure Graph algorithm imposes natural constraints on paths Needs Improvement Privacy – depends on central server Convenience –Extra device to wear –No display on device Infrastructure – requires special receivers

17. Outline Introduction Video Tracking Active badge – SmartMoveX Coarse Wi-Fi location – “Here I Am” Fine Wi-Fi location – L OCADIO SPOT Wristwatch Location – RightSPOT Wi-Fi proximity – NearMe “Longhorn” Location Service – Sensor Fusion Active Badge

18. “Here I Am” – Coarse Location With Steve Shafer (Microsoft Research)

19. “Here I Am” MAC AddressBuildingFloorRoom 0:40:96:29:e:71822075 0:40:96:29:f:781811407 0:40:96:29:1a:6c1822229 0:40:96:29:24:c71811055 0:40:96:29:3f:c31811212 0:40:96:29:48:371822115 0:40:96:29:4a:401822039 0:40:96:29:55:81811081 0:40:96:29:57:a01822498 0:40:96:29:58:2f1811265 0:40:96:29:59:161811426 0:40:96:29:5b:1d1811137 0:40:96:29:63:1a1833367 802.11 Access Point Data Room Numbers Hand-Entered from Maps “Here I Am” returns position of strongest access point

20. Outline Introduction Video Tracking Active badge – SmartMoveX Coarse Wi-Fi location – “Here I Am” Fine Wi-Fi location – L OCADIO SPOT Wristwatch Location – RightSPOT Wi-Fi proximity – NearMe “Longhorn” Location Service – Sensor Fusion

21. Location from 802.11 with L OCADIO * Mobile device measures signal strengths from Wi-Fi access points Computes its own location Wi-Fi (802.11) access point * Location from Radio John Krumm & Eric Horvitz

22. L OCADIO - Fine Location Radio survey to get signal strength as a function of position

23. L OCADIO Setup 1.Map – for radio survey and feasible paths 2.Radio Survey – signal strengths taken at various locations 3.Feasible Paths – Constrain paths where a person could walk Signal strengths from these positionsFeasible walking paths No new access points installed Access point positions unnecessary Access points need no network connection

24. Hidden Markov Model (HMM) Location 11 Location 3 Location 8 Location 2 Location 19 0.560.230.180.80 s1s1 s2s2 s3s3 s4s4 s5s5 states are discrete locations transition probability between each pair of states observation likelihood for each state (802.11 signal strengths)

25. Radio Survey Walk around with Wi-Fi device and record signal strengths at known locations Above calibration had one point for every 10 square meters 1 minute at each point, spinning around Saw ~ 25 access points in all Average 3.6 access points seen from any given location 63 locations

26. Radio Survey Results Probability of seeing access point “i” from location “j” Probability of signal strength at point “j” if access point seen from location “i” Distributions used later to infer location, but first some constraints …

27. Path Constraints Draw feasible paths on mapFill in at 1 meter spacing Prevents computed paths from penetrating walls Serves as graph of nodes for Hidden Markov Model (HMM) Need transition probabilities between all nodes … Constraints make things easier

28. Internode Distances d ij = shortest path distance between “i” and “j” Computed with Dijkstra’s shortest path algorithm

29. Internode Speeds - Walking John J. Fruin, Pedestrian Planning and Design, 1971, New York: Metropolitan Association of Urban Designers and Environmental Planners. “… distribution of free-flow walking speeds obtained in surveys of about 1000 non-baggage-carrying pedestrians inside the Port Authority Bus Terminal and Pennsylvania Station in New York City.” p(walking speed | moving)

30. Internode Speeds - Other p(speed | other) speed 10.22 meters/second American Tim Montgomery, world record 100 meters in 9.78 seconds, September 14, 2002 in Paris (average 10.22 meters/second) p(speed | moving) = p(speed | walking) p(walking | moving) + p(speed | other) (1- p(walking | moving)) Guess p(walking | moving) = 0.9

31. Internode Still vs. Moving What is p(moving)? Variance of still vs. moving rssiRssi seems noisier when device is moving Bayes:

32. Smooth Still vs. Moving Transition probabilities for still vs. moving: s = number of seconds (28,800 in 8 hours) m = number of moves (10) r = rssi sampling rate (3.7 Hz) a SM = a MS = mr/s = 0.00132 Markov Model 87.4% correct classification 24 transitions 84.5% correct classification 172 transitions 14 transitions

33. Location Transition Probabilities a ij = probability of transition between nodes i and j Distance: d ij = shortest path distance between nodes i and j Time: Δt = rssi sampling interval Speed: s ij = d ij /Δt p(s) = p(s | moving) p(moving) + p(s | still) (1 - p(moving)) = probability of speed δ(0) a ij ~ p(s ij ) Transition probabilities account for: Building floorplan (walls & doors) Expected pedestrian speeds Still vs. moving inference

34. HMM for Location 1.Location nodes, i = 1 … N (N = 317) 2.Initial state probabilities: π i = 1/N (could be anywhere) 3.Transition probabilities: a ij (from previous 7 slides) 4.Observation probabilities: from radio survey, but … Location nodes (317)Radio survey (63) RBFs to interpolate from survey nodes to location nodes

35. Observation Probabilities Observation Data I t = (true, false, true, true, true) which access points were seen s t = (-58, Ø, -57, -78, -79) signal strengths from each visible access point Likelihood of Observation Given Location accounts for seen and unseen access points assumes independence among access points (likely true)

36. What Happens N location nodes t0t0 I 0, s 0 a ij(0) t1t1 I 1, s 1 t2t2 I 2, s 2 t3t3 I 3, s 3 a ij(1) a ij(2) Viterbi algorithm efficiently computes optimal path after every new rssi scan

37. Results Test Data 10 short walks over 700 m 2 calibrated area Each walk 2 minutes, 25 seconds on average Ground truth by interpolating between turns Total 4586 test points Results Median error 1.53 meters

38. L OCADIO - Constraints No passing through wallsNo speedingWe know when you move Make the client as smart as possible to reduce calibration effort

39. L OCADIO - Results Hidden Markov model gives median error of 1.53 meters

40. Outline Introduction Video Tracking Active badge – SmartMoveX Coarse Wi-Fi location – “Here I Am” Fine Wi-Fi location – L OCADIO SPOT Wristwatch Location – RightSPOT Wi-Fi proximity – NearMe “Longhorn” Location Service – Sensor Fusion

41. SPOT Watch weather traffic diningmovies Commercial FM: transmit new data every ~2 minutes Filter on watch to take what it wants Watch displays “personalized” data

42. Location-Sensitive Features Nice to have Local traffic Nearby movie times Nearby restaurants Need to know location of device …

43. Use FM Radio Signal Strengths Scan signal strengths of 32 FM radio stations at 1 Hz

44. Clustering Approach KPLU KMTT RedmondBellevueSeattleWoodinvilleSammamish But Each watch scales signal strengths differently Impractical to calibrate every watch Input Power Measured RSSI ABC

45. Ranking Approach Redmond:KPLU < KMTT < KMPS Bellevue:KMTT < KPLU < KMPS Issaquah:KMTT < KMPS < KPLU … n radio stations  n! possible rankings f = (f 1, f 2, f 3, …, f n ) = scanned frequencies s = (s 1, s 2, s 3, …, s n ) = signal strengths r = (r 1, r 2, r 3, …, r n ) = ranks of signal strengths e.g.s = (12, 40, 38, 10) r = (2, 4, 3, 1) R(r) = permutation hash code = [0, 1, 2, … n!-1] Any monotonically increasing function of signal strength preserves ranking

46. Test Six suburbs and six radio stations 81.7% correct from 8 radio stations

47. Avoid Manual Training Seattle KMPS 94.1 MHz KSER 90.7 MHz

48. Classify Into Grid Cell Find location in grid Use predicted signal strengths to avoid manual training ≈ 8 kilometers average error Summer intern Adel Youssef, U. Maryland

49. Outline Introduction Video Tracking Active badge – SmartMoveX Coarse Wi-Fi location – “Here I Am” Fine Wi-Fi location – L OCADIO SPOT Wristwatch Location – RightSPOT Wi-Fi proximity – NearMe “Longhorn” Location Service – Sensor Fusion

50. “NearMe” conference rooms printers bathroom reception desk people Find people and things nearby

51. NearMe Basic Idea 802.11 Access Points NearMe Server

52. NearMe ≠ Location Hightower, Fox, Borriello, “The Location Stack”, 2003 Why compute absolute locations when you only need relative locations? Tomasi, Kanade, “Shape and Motion from Image Streams: a Factorization Method”, 1991

53. Short Circuit Get signal strengths Get location Compare

54. NearMe Screen Shots 1. Register with server 3. Nearby people 2. Report Wi-Fi signals 4. Nearby printer(s) NearMe Server SQL Server.NET Web Service

55. NearMe Distance Estimate Distance = f(n ∩,ρ s ) n ∩ = number of access points seen in common ρ s = Spearman rank correlation of signal strengths Estimate distance between two clients by comparing “Wi- Fi signatures” 15 meters rms error

56. NearMe Applications Look up URLs of nearby people/things Send email to people nearby Device association (with Ken Hinckley, Microsoft Research)

57. Outline Introduction Video Tracking Active badge – SmartMoveX Coarse Wi-Fi location – “Here I Am” Fine Wi-Fi location – L OCADIO SPOT Wristwatch Location – RightSPOT Wi-Fi proximity – NearMe “Longhorn” Location Service – Sensor Fusion

58. “Longhorn” Location Service Location Service AN/PLR-3 Helmet-Mounted Radar (I am not making this up.) GPS cell phone Wi-Fi Bluetooth other unanticipated other location resolvers your location “Longhorn” PC knows its location

59. Longhorn Location Service MapPoint Db / Serv. AD Mgmt App (shell, netxp, OEM) WinFS Wireless (802.11) Zero Configuration Service BT Configuration Service OEM Service Plugin Manager Master Resolver Fuser LocProv API LocMgmt API 802.11 Provider Blue Tooth Provider OEM Provider User Resolver AD Resolver Map Point Resolver User Pref. Db. Location API Notification Service App (Shell, OEM) Cache Tracey Yao, PM Florin Teodorescu, Dev Vivek Bhanu, Dev Jim Seifert, Test Madhurima Pawar, Test

60. Fusion Metric Measurements Building40 (0.9)40 (0.4)41 (0.4) Floor3 (0.8)3 (0.2)1 (0.3) Room3019 (0.2) 1502 (0.2) Hierarchies Kalman Filter Weighted Hierarchical Voting Proper fusion of measurements depends on knowledge of uncertainty

61. The End