1. GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen, CMU Wende Zhang, GM Inna Stainvas, GM ContributorsJongHo Lee, CMU 1

2. GM-Carnegie Mellon Autonomous Driving CRL Schedule Curb locationSensor locationMethodologyDate On the sideBottom of the side mirrorVisual appearance~ Jan. 2014 In frontThe front bumperGeometric structure~ May. 2014 In frontThe front bumperAppearance + Geometry with production camera ~ Nov. 2014 Deliverables Demonstration: In-vehicle curb detection Annual reports 2

3. GM-Carnegie Mellon Autonomous Driving CRL Objectives Develop reliable methods of detecting, localizing, and classifying features associated with curbs using in- vehicle, low-cost, monocular vision sensor Localize curbs within a range of 5 meters with 99% accuracy 3

4. GM-Carnegie Mellon Autonomous Driving CRL Approaches for Curb Detection Using Mono Camera Images Appearance-based image analysis (~ Nov. 2013) Extract features Evaluate performance Geometry-based image analysis (~ May. 2014) Structure-from-motion to estimate camera motion Multi-resolution plane sweeping algorithm to create 3-D point cloud Plane fitting to detect curb Combine appearance and geometric analysis (This Review) 4

5. GM-Carnegie Mellon Autonomous Driving CRL Appearance-based image analysis (~ Nov. 2013) Extract features Evaluate performance Geometry-based image analysis (~ May. 2014) Structure-from-motion to estimate camera motion Multi-resolution plane sweeping algorithm to create 3-D point cloud Plane fitting to detect curb Combine appearance and geometric analysis (This Review) Approaches for Curb Detection Using Mono Camera Images 5

6. GM-Carnegie Mellon Autonomous Driving CRL Appearance-based image analysis 6

7. GM-Carnegie Mellon Autonomous Driving CRL Edge Detection 7

8. GM-Carnegie Mellon Autonomous Driving CRL Detect Curb Using HOG * Feature * Histogram of Oriented Gradients Input image HOG image Curb model 8

9. GM-Carnegie Mellon Autonomous Driving CRL Appearance-based image analysis (~ Nov. 2013) Extract features Evaluate performance Geometry-based image analysis (~ May. 2014) Structure-from-motion to estimate camera motion Multi-resolution plane sweeping algorithm to create 3-D point cloud Plane fitting to detect curb Combine appearance and geometric analysis (This Review) Approaches for Curb Detection Using Mono Camera Images 9

10. GM-Carnegie Mellon Autonomous Driving CRL Geometry-based image analysis Input imageDepth image Ground plane estimation 3-D point cloud 10

11. GM-Carnegie Mellon Autonomous Driving CRL Plane Fitting 11

12. GM-Carnegie Mellon Autonomous Driving CRL Appearance-based image analysis (~ Nov. 2013) Extract features Evaluate performance Geometry-based image analysis (~ May. 2014) Structure-from-motion to estimate camera motion Multi-resolution plane sweeping algorithm to create 3-D point cloud Plane fitting to detect curb Combine appearance and geometric analysis (This Review) Approaches for Curb Detection Using Mono Camera Images 12

13. GM-Carnegie Mellon Autonomous Driving CRL Appearance at t+1 Schematic Overview Input at t+1 Input at tAppearance at t Geometry Candidate regions Annotate curb region

14. GM-Carnegie Mellon Autonomous Driving CRL 14 Appearance -For each image, divide into m x n grids -m: image height / grid size (pixels) -n: image width / grid size (pixels) Image at t

15. GM-Carnegie Mellon Autonomous Driving CRL 15 Appearance -For each grid, classify among two classes (road, curb) -uniform Local Binary Pattern (LBP) Image at t

16. GM-Carnegie Mellon Autonomous Driving CRL 16 Local Binary Pattern 646352 85152227 189167205 000 01 111 threshold Binary: 00011110 Decimal: 30 0123 255

17. GM-Carnegie Mellon Autonomous Driving CRL 17 Appearance -Once all the grids of two images are classified, get the intersection of them Output at t+1Output at tIntersect

18. GM-Carnegie Mellon Autonomous Driving CRL Geometry -Green lines shows the vectors from the interesting points of image at time t (blue dots) to those of image at time t+1 (red dots) -Calculate the 3-D points using camera matrix

19. GM-Carnegie Mellon Autonomous Driving CRL Appearance + Geometry -For each grid, -Fit the best plane using 3-D points -Compute the normal vector -Determine the normal vector is a road surface or a curb surface

20. GM-Carnegie Mellon Autonomous Driving CRL Appearance + Geometry

21. GM-Carnegie Mellon Autonomous Driving CRL Extend Curb Region -If the appearances are similar, extend the curb region -Calculate the distance of LBPs using chi-square

22. GM-Carnegie Mellon Autonomous Driving CRL Extend Curb Region

23. GM-Carnegie Mellon Autonomous Driving CRL Track Curb Region -For the next frames, tracking the appearance of the curbs -When tracking, keep checking the geometry constraint to remove the false positives if exist Input at t+2Input at t+3Input at t+4

24. GM-Carnegie Mellon Autonomous Driving CRL Curved curb case Extend Curb RegionCombine Analyses

25. GM-Carnegie Mellon Autonomous Driving CRL Curb Detection using Production Camera Image size : 480 by 640 FOV: 180 degree

26. GM-Carnegie Mellon Autonomous Driving CRL Test Curb Detection Image Size: 640 x 480 (pixels) ROI: 640 x 160 (pixels) Size of grid: 20 x 20 (pixels) Number of grids: 32 x 8 Output of the appearance- based curb detection

27. GM-Carnegie Mellon Autonomous Driving CRL Test Curb Detection Remove outliers based on cluster size Find edges using Canny operator inside candidate region

28. GM-Carnegie Mellon Autonomous Driving CRL Test Curb Detection - Fit polynomials to each segments, and check lines for similar curvatures (blue), and remove high curvatures (red) Annotate curb region on the original input image

29. GM-Carnegie Mellon Autonomous Driving CRL Future Work Application: Operate real-time curb detection in vehicle ~ May 2015 29

30. GM-Carnegie Mellon Autonomous Driving CRL Thank you Questions ? 30