1. Feasibility of Interactive Localization and Navigation of People with Visual Impairments Ilias Apostolopoulos, Navid Fallah, Eelke Folmer and Kostas E. Bekris Computer Science and Engineering 1 September 2010, IAS

2. Introduction Problem Navigation in an indoor environment for individuals with visual impairment(VI) Solution Create a navigation system using: Minimal sensors (i.e. compass, pedometer) available in smart phones Interaction with the user Techniques from robotics

3. Wall Hallway intersection Door

4. Motivation Outdoors Outdoor navigation systems typically use GPS for localization Indoors Indoor navigation systems use RFID tags, cameras, laser scanners Individuals with VI navigate with compensatory senses (e.g., touch) -Results in reduced mobility A need for navigation assistance rises Easy Challenging

5. Motivation Indoor navigation should be accurate and efficient Solution proposed here requires only minimal sensors that can be found in a smart phone. Easier to create virtual infrastructure instead of a physical one

6. Characteristics of the Approach User interacts with the phone through an audio/speech interface Phone provides directions using landmarks that are recognizable by individuals with VI(i.e. doors, hallway intersections) User confirms landmarks through touch Directions Audio Feedback Speech Landmark Confirmation Desired Destination

7. Challenges and Premises Challenges Is there enough computational power on widely available portable devices? Is the sensing information sufficient? Can people perform well as sensors in this setup? Premises Individuals with VI can recognize landmarks through touch Indoor spaces, while complex, are highly constrained While pedometers and compasses are highly erroneous: They can provide a sufficiently good estimate for the user’s motion When integrated with state of the art methods for localization

8. Objectives of Feasibility Study Two research questions 1.Is it possible for a human user to be successfully guided with the overall approach? How do the type of directions provided affect the probability of success? 2.Is it possible to localize the user with an accuracy that will help us adapt directions on the fly?

9. Background

10. Localization Techniques 1.Dead-Reckoning Integrate measurements of the human motion Accelerometers and radars have been used -P. Van Dorp et al., Human walking estimation with radar, 2003 Error grows unbounded 2.Beacon-based Use identifiers in the physical space Beacons can be detected by cameras, infrared or ultrasound identifiers Popular solutions use RFID tags -V. Kulyukin et al., Rfid in robot-assisted indoor navigation for the visually impaired, 2004 Locating beacons might be hard and inaccurate Significant time and cost spent installing and calibrating beacons

11. Localization Techniques 3.Sensor-based Use cameras to detect pre-existing features (e.g., walls, doors) -Cameras need good lighting and have prohibitive computational cost -O. Koch and S. Teller, Wide-area egomotion estimation from known 3d structure, 2007 Use 2D laser scanners -Expensive and heavy -J. A. Hesch et al., An indoor localization aid for the visually impaired, 2007 The system proposed is a sensor-based system Pedometer and compass for localization User as a sensor for landmark confirmation Use of probabilistic tools from robotics

12. Bayesian Methods from Robotics Candidate methods: 1.Extended Kalman Filter Assumes normal distribution +Returns optimum estimate under certain assumptions -Not a good model for multimodal distribution R. E. Kalman, A new approach to linear filtering and prediction problems, 1960 2.Particle Filter Employs a population of discrete estimates +Can represent a multimodal distribution -High computational cost N. J. Gordon et al., “Novel approach to nonlinear/non-gaussian bayesian state estimation”, 1993 Particle Filter is chosen due to better accuracy about the location of the user

13. Methodology Part 1: Directions

14. High-level Operation 1.A user specifies a start and destination room number to travel to. 2.The system computes the shortest path using A* and finds landmarks along the path. 3.Directions are provided iteratively upon completion through the phone’s built-in speaker. The user presses a button on the phone after successfully executing each direction.

15. Types of Directions The type of directions affects the efficiency and reliability of navigation High reliability when user confirms all landmarks, but low efficiency High efficiency when relying more on odometry, but low reliability Two types of direction provision were tested Landmark based Metric based Three granularities for each method No Max Threshold 15 Meters Threshold 9 Meters Threshold Metric based Landmark based Intersection Door 15 steps 20 steps

16. Types of Directions Metric based Intersection Door 15 steps 20 steps Landmark based Directions Metric based Landmark based No Max Threshold 9 Meters Threshold 15 Meters Threshold No Max Threshold 9 Meters Threshold 15 Meters Threshold

17. Direction Provision Landmark No Max Threshold "Exit the room then turn right" "Move forward until you reach a hallway on your left“ "Turn left to the hallway“ "Move forward until you reach a water cooler on your left“ "Move forward until you reach a hallway on your left“ "Turn left to the hallway" "Follow the wall on your left until you reach the third door“ "You have reached your destination"

18. Direction Provision Landmark 15 Meters Threshold "Exit the room then turn right“ "Move forward until you reach a hallway on your left“ "Turn left to the hallway“ "Follow the wall on your left until you reach the third door“ "Move forward until you reach a water cooler on your left" "Move forward until you reach a hallway on your left" "Turn left to the hallway" "Follow the wall on your right until you reach the 5th door" "Follow the wall on your left until you reach the first door" "You have reached your destination" 1 2 3 1 234 5

19. Direction Provision Landmark 9 Meters Threshold "Exit the room then turn right" "Follow the wall on your right until you reach the first door" "Move forward until you reach a hallway on your left“ "Turn left to the hallway" "Follow the wall on your left until you reach the second door“ "Move forward until you reach a water cooler on your left“ "Move forward until you reach a hallway on your left" "Turn left to the hallway" "Follow the wall on your right until you reach the third door“ "Follow the wall on your left until you reach the first door" "You have reached your destination"

20. Direction Provision Metric No Max Threshold "Exit the room then turn right" "Walk x steps until you reach a hallway on your left“ "Turn left to the hallway“ "Walk x steps until you reach a water cooler on your left“ "Walk x steps until you reach a hallway on your left“ "Turn left to the hallway“ "Walk x steps until you reach a door on your left“ "You have reached your destination"

21. Direction Provision Metric 15 Meters Threshold "Exit the room then turn right“ "Walk x steps until you reach a hallway on your left" "Turn left to the hallway“ "Walk x steps until you reach a door on your left“ "Walk x steps until you reach a water cooler on your left“ "Walk x steps until you reach a hallway on your left“ "Turn left to the hallway“ "Walk x steps until you reach a door on your right“ "Walk x steps until you reach a door on your left" "You have reached your destination"

22. Direction Provision Metric 9 Meters Threshold "Exit the room then turn right"; "Walk x steps until you reach a door on your right“ "Walk x steps until you reach a hallway on your left“ "Turn left to the hallway“ "Walk x steps until you reach a door on your left“ "Walk x steps until you reach a water cooler on your left“ "Walk x steps until you reach a hallway on your left“ "Turn left to the hallway"; "Walk x steps until you reach a door on your right“ "Walk x steps until you reach a door on your left“ "You have reached your destination"

23. Methodology Part 2: Localization

24. Localization Objective is an accurate location of the user Previous location and sensor data are used as input The new location is the output of the system Model n static landmarks ξ = (x, y, θ) state of the system m map that stores information about the world Landmarks belong to k different types (e.g, doors, hallway intersections)  Landmarks in the same class are indistinguishable Data d T = (o(0:T), u(0:T-1))  u(0:T-1) transitions  o(0:T) observations

25. Transitions and Observations Transitions A transition corresponds to a motion where the agent acquires the orientation and moves forward This transition determines the kinematic transition model of the system Observations Observation of a landmark from state implies: (x t+1,y t+1,θ t+1 ) utfutf utθutθ (x t,y t,θ t ) (x t, y t ) R obs lili (x i, y i )

26. Transitions and Observations Transitions A transition corresponds to a motion where the agent acquires the orientation and moves forward Observations Observation of a landmark from state implies: (x t+1,y t+1,θ t+1 ) utfutf utθutθ (x t,y t,θ t ) (x t, y t ) R obs lili (x i, y i )

27. Filtering The objective is to be able to incrementally estimate the user’s state at time T. The general Bayes filter computes a belief distribution at time T over given data d T. It requires:  An initialization B 0  A transition model  The observation model Given the normalization factor η the belief distribution can be updated as follows: Previous Belief Transition model Observation model Applying Transition model filter with observation

28. Filtering The objective is to be able to incrementally estimate the user’s state at time T. Given the normalization factor η the belief distribution can be updated as follows: Previous Belief Transition model Observation model Applying Transition model filter with observation

29. Particle Filter It is possible to represent B T through a set P of particles Each particle stores a state estimate together with a weight At each time step T, the following steps are: A.For each particle i.Employ the transition model to acquire ii.Employ the observation model to compute the new weight B.Sample a new population of P particles given the weights

30. Implementation of Transition Model Collects orientation from compass and step counts from pedometer Multiple readings from compass are averaged for a time step Pedometer returns 1 if the user moved a step, else 0 Step length is calculated for each user through some train paths Noise is added with a normal distribution to introduce uncertainty

31. Implementation of Observation Model Two cases: No landmark detected by user All particles get weight 1 Landmark detected Particles within observation range get weight inversely proportional to distance Particles out of range get weight 0

32. Sampling Algorithm samples with higher probability particles with higher weight When all particles get a weight of 0: Algorithm finds the closest visible landmark of the type observed by the user for each particle Particle is sampled near the landmark

33. Experiments

34. Setup System was implemented in Java for Google’s Android platform Map of a buildings floor in campus was created in Keyhole Markup Language(KML) Map contains: 3 water coolers 1 floor transition marked by a metal strip 3 hallway intersections & 2 hallway turns 72 doors 5 paths were defined two alternatives for directions with three granularities for each

35. Participants 10 volunteers Users held the phone in one hand and a cane in the other 1 volunteer was legally blind and assisted with the setup of the experiments 9 more sighted volunteers were blindfolded Some users had prior knowledge of the building while others didn’t Each user executed ten traversals Two traversals per path using different directions

36. Ground Truth An observer was recording the user’s motion Markers were placed on the floor every two meters Every time the user was crossing a marker the observer was recording the time in a second smart phone Assume that user moves with constant speed between markers Ground truth resolution 2 meters

37. Objectives of Feasibility Study Two research questions 1.Is it possible for a human user to be successfully guided with the overall approach? How do the type of directions provided effect the probability of success? 2.Is it possible to localize the user with an accuracy that will help us adapt directions on the fly?

38. 1.Success Ratio of Direction Provision 84% of experiments reached a distance smaller than 2 meters 92% of experiments reached less than 3.5 meters Distance from goal

39. 1.Success Ratio of Direction Provision Duration of execution

40. 2.Localization Accuracy Particle filter improves considerably results acquired just by sensor readings Paths with distinctive landmarks had considerably lower error than paths with repetitive landmarks Distance from Ground Truth

41. Localization Error Average error: 3.69 meters 2.Localization Accuracy

42. Discussion Proposed a minimalistic indoor navigation system for individuals with VI Sensors used are inexpensive and common though erroneous Answers to research questions: It is possible for an individual with VI to be guided successfully with this scheme It is possible to localize the user with accuracy

43. Update Submitted versionCurrent version Average error: 3.69 metersAverage error: 2.05 meters

44. Future Work Automatic direction provision based on localization estimates User studies with a larger number of users with VI More complex environments Buildings with multiple floors Elevators, stairs, ramps 3D maps with more details about the world

45. Thank you for your attention!