1. Image-Based Segmentation of Indoor Corridor Floors for a Mobile Robot Yinxiao Li and Stanley T. Birchfield The Holcombe Department of Electrical and Computer Engineering Clemson University { yinxial, stb }@clemson.edu

2. Typical Corridor Images Low contrast Common Reflections Specular Reflections

3. Outline Previous work Detecting Line Segments Score model for Evaluating Line Segments –Structure Score –Bottom Score –Homogenous Score Experimental Results Conclusion

4. Outline Previous work Detecting Line Segments Score model for Evaluating Line Segments –Structure Score –Bottom Score –Homogenous Score Experimental Results Conclusion

5. Previous Work Free space detection Previous approaches Stereo Matching (Sabe 2004) Optical Flow (Stoffler 2000, Santos-Victor 1995 Combination of color and histogram (Lorigo 1997) Limitations Need motion require different colors for floor and wall Computational efficiency Calibrated cameras Floor detection Previous approaches Apply planar homographies to optical flow vectors (Kim 2009, Zhou 2006) Stereo homographies (Fazl-Ersi 2009) Geometric reasoning (Lee 2009) Limitations Assume ceiling visible Computational efficiency Calibrated cameras

6. Outline Previous work Detecting Line Segments Score model for Evaluating Line Segments –Structure Score –Bottom Score –Homogenous Score Experimental Results Conclusion

7. Douglas-Peucker Algorithm http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973) An algorithm that: Reduces the number of points in a curve that is approximated by a series of points Focus on maximum distance between the original curve and the simplified curve Finds a similar curve with fewer points

8. Douglas-Peucker Algorithm http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973) An algorithm that: Reduces the number of points in a curve that is approximated by a series of points Focus on maximum distance between the original curve and the simplified curve Finds a similar curve with fewer points

9. Douglas-Peucker Algorithm http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973) An algorithm that: Reduces the number of points in a curve that is approximated by a series of points Focus on maximum distance between the original curve and the simplified curve Finds a similar curve with fewer points

10. Douglas-Peucker Algorithm http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973) An algorithm that: Reduces the number of points in a curve that is approximated by a series of points Focus on maximum distance between the original curve and the simplified curve Finds a similar curve with fewer points

11. Douglas-Peucker Algorithm http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973) An algorithm that: Reduces the number of points in a curve that is approximated by a series of points Focus on maximum distance between the original curve and the simplified curve Finds a similar curve with fewer points

12. Douglas-Peucker Algorithm http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm David Douglas & Thomas Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112–122 (1973) An algorithm that: Reduces the number of points in a curve that is approximated by a series of points Focus on maximum distance between the original curve and the simplified curve Finds a similar curve with fewer points

13. Modified Douglas-Peucker Algorithm original algorithm: d allowed is constant modified algorithm: d allowed is given by half-sigmoid function d allowed original modified

14. Detecting Line Segments (LS)

15. 1.Compute Canny edges

16. Detecting Line Segments (LS) 1.Compute Canny edges 2.Modified Douglas-Peucker algorithm to detect line segments Vertical LS: slope within ±5° of vertical direction Horizontal LS: Slope within ±45° of horizontal direction d allowed

17. Detecting Line Segments (LS) 1.Compute Canny edges 2.Modified Douglas-Peucker algorithm to detect line segments Vertical LS: slope within ±5° of vertical direction Horizontal LS: Slope within ±45° of horizontal direction 3. Pruning line segments (320x240) Vertical LS: length 60 pixels Horizontal LS: length 15 pixels Vanishing point (height selection): The vanishing point is computed as the mean of the intersection of pairs of non- vertical lines d allowed

18. Detecting Line Segments (LS) 1.Compute Canny edges 2.Modified Douglas-Peucker algorithm to detect line segments Vertical LS: slope within ±5° of vertical direction Horizontal LS: Slope within ±45° of horizontal direction 3. Pruning line segments (320x240) Vertical LS: length 60 pixels Horizontal LS: length 15 pixels Vanishing point (height selection): The vanishing point is computed as the mean of the intersection of pairs of non- vertical lines d allowed

19. Outline Previous work Detecting Line Segments Score model for Evaluating Line Segments –Structure Score –Bottom Score –Homogenous Score Experimental Results Conclusion

20. Score Model is assigned to each horizontal line segment and we use a threshold to select those ones that beyond the Structure Score Bottom Score Homogeneous Score Weights

21. Score Model – Structure Given a typical corridor image Structure =>Threshold=?

22. Score Model – Structure Given a typical corridor image 1.Find gradient magnitude greater than a threshold T1 Structure =>Threshold=?

23. Score Model – Structure Structure =>Threshold=? Given a typical corridor image 1.Find gradient magnitude greater than a threshold T1 2.Find gradient magnitude greater than a threshold T1 and intensity less than a threshold T2. The figure shows the mask pixels image. We have a training data over 200 images to determine T1 and T2 by SVM over 800 points.

24. Score Model – Structure Given a typical corridor image 1.Find gradient magnitude greater than a threshold T1 2.Find gradient magnitude greater than a threshold T1 and intensity less than a threshold T2. The figure shows the mask pixels image. 3.Compute the average intensity of the make pixels image over original image and get a threshold value T LC Structure =>Threshold= T LC

25. Score Model – Structure Given a typical corridor image 1.Find gradient magnitude greater than a threshold T1 2.Find gradient magnitude greater than a threshold T1 and intensity less than a threshold T2. The figure shows the mask pixels image 3.Compute the average intensity of the make pixels image over original image and get a threshold value T LC 4.Compute the distance between the line segments and structure blocks Structure =>Threshold= T LC

26. Score Model – Structure Ridler-Calvard Otsu Our method Our method is able to get rid of the spurious pixels on the floor caused by reflections or shadows MethodR-COtsuOurs Correctness62%66%82% Comparison of different threshold methods

27. Outline Previous work Detecting Line Segments Score model for Evaluating Line Segments –Structure Score –Bottom Score –Homogenous Score Experimental Results Conclusion

28. Score Model – Bottom 1.Using vertical and horizontal line segments 2.Connect each bottom point of the vertical line segments as a “bottom” wall-floor boundary

29. Score Model – Bottom 1.Using vertical and horizontal line segments 2.Connect each bottom point of the vertical line segments as a “bottom” wall-floor boundary 3.Compute the distance of each horizontal lines segments to the “bottom” wall-floor boundary (red circled horizontal line segments indicates a positive contribution)

30. Outline Previous work Detecting Line Segments Score model for Evaluating Line Segments –Structure Score –Bottom Score –Homogenous Score Experimental Results Conclusion

31. Score Model – Homogeneous 1.Using efficient graph-based segmentation ( Felzenszwalb [5] ), we may get a bigger segmented area in floor area, due to decorations, posters, windows etc. on the wall. 2.Compute the ratio of area of the segmented region below a horizontal line segment over the biggest segmented region in the image. 3.The ratio is assigned to each horizontal line segment as a homogenous score.

32. Detecting wall-floor boundary Given a threshold to the sum of each score Connect the remaining line segments and extend the endpoints to the edges of the image Future work: A larger database and AdaBoost algorithm to generate weights for each score

33. Outline Previous work Detecting Line Segments Score model for Evaluating Line Segments –Structure Score –Bottom Score –Homogenous Score Experimental Results Conclusion

34. Evaluation Criterion Wall- floor Boundary –Green line: Detected wall-floor boundary –Red line: Ground truth

35. Evaluation Criterion Wall- floor Boundary –Green line: Detected wall-floor boundary –Red line: Ground truth Area –Blue shade area: misclassified area –Red shade area: ground truth floor area Error Rate

36. Sample Results Corridor images from our database

37. Sample Results Images download from internet.

38. Sample Results Some successful results of failure examples from Lee[1]. Original Image Lee’s Ours

39. Sample Results Failure examples Checked floor Texture wall Overly dark Bright lights

40. Sample Results Video clip 1

41. Sample Results Video clip 2

42. Outline Previous work Detecting Line Segments Score model for Evaluating Line Segments –Structure Score –Bottom Score –Homogenous Score Experimental Results Conclusion

43. Summaries - Over 90% of the corridor images in our database can be correctly detected. - Speed: around 7 frame/sec. - Weakness: not robust to highly textured floor, low resolution and dark environment. Future work - Speed up the current algorithm - Improving score model by add more visual cues - Mapping the whole corridor structure for robot navigation - …

44. Acknowledgement  Clemson Computer Vision Group reviewers Zhichao Chen Vidya Murali  Anonymous reviewers