1. Structured Hough Voting for Vision-based Highway Border Detection
Zhiding Yu
Carnegie Mellon University

2. Autonomous Driving: Not If, But When

3. GM-CMU Collaborative Research
And this paper is a joint collaborative research between General Motors Company and CMU. We are working together to develop future autonomous driving vehicles and the paper is part of this research project.

4. Sensors Setup on SRX Platform
Images from: Junqing Wei et al., “Towards a Viable Autonomous Driving Research Platform,” IEEE Intelligent Vehicles Symposium (IV), 2013

5. Sensors: Price vs Information
Camera
Lidar
And this paper is a joint collaborative research between General Motors Company and CMU. We are working together to develop future autonomous driving vehicles and the paper is part of this research project.
Radar
Price

6. Computer Vision Applications
Object detection (pedestrian, vehicle, bicycle…)
Road parsing (lane/border detection, road segmentation, vanishing point estimation…)
Localization and tracking
Driver status monitoring
Many other applications……
And this paper is a joint collaborative research between General Motors Company and CMU. We are working together to develop future autonomous driving vehicles and the paper is part of this research project.

7. Motivation, Description and Goal
Development for future driving assistance system and autonomous driving system
Robust detection within 0.5 to 6 meters detection range. Achieve near 100% accuracy in daytime and over 90% in nighttime on the right most lane
Handling various scenarios including highway entrance and exit
Extend to the joint system with front view
What we are trying to do is that we have a monocular camera looking towards the side and we seek to use vision and learning algorithms to automatically detect the border and shoulder of a highway. By saying border we mean it is the physical end of the paved road. As you can see the red line in the image is the border returned by our algorithm.
Another thing we aim to detect is the road shoulder. A shoulder is defined as the region between the right most solid lane and the border. For example, the blue line here is the lane marking. And the green region is the shoulder.

8. High-Level Idea: Learning based Method
Concrete Barrier
Guard Rail
Soft Shoulder
Guard Rail
Soft Shoulder
Concrete
Barrier
Lane Marking
How does the algorithm work? In this paper, we train both the border detector and the lane marker detector, and perform scanning window detection. Our trained detector handles various types of borders as shown on the top row. The scanning window detection will return densely triggered detectors and each triggered detector will return a voting point indicating approximately where the border and lane marking are. We then use our proposed structured Hough voting model to finally output both the border as well as the shoulder.
Structured Hough Voting
Densely Fired scanning windows
Returned Voting Points
Border / lane marking hypotheses

9. Dataset Collection Overall 1592 training images:
Concrete Barrier (839 images)
Guard Rail (300 images)
Soft Shoulder (453 images)
Overall 2638 testing images:

10. Training Patch Alignment
For each sample patch, fix the width-height ratio to be 2
Center each patch with respect to y-coordinates of ground truth
1-3 positive samples from each image. Separate on x-coordinates to cover as much as possible.
Each positive sample is associated with 3 negative samples with the same size, randomly selected from background.
Concrete
Natural
Steel
Lane Marker
Positive Samples:
Negative Samples:

11. Concatenated Filter Bank Feature Concatenated HOG Feature
Feature Extraction
Filter Bank
Concatenated Filter Bank Feature
Concatenated HOG Feature
HOG
Patches that are discriminative to HOG
Patches that are discriminative to filter banks

12. Classification & Detection
Extract features from all training patches (based on previous page)
Perform Fisher discriminant analysis
Train an RBF kernel SVM
Scanning window detection (Deliberately having a lot of positive firing)
Guard Rail
Soft Shoulder
Concrete
Barrier
Lane Marking

13. Hough Voting

14. Structured Hough Voting: Intuitions
Basic philosophy: A model that assumes voting results are correlated rather than independent
Inter-frame structural info on hypotheses (Temporal smoothness)
Intra-frame structural info (Geometric relationship)
Multiple candidate hypotheses generation (Proposals with diversity)
Constrained Hough Voting on detected voting points (Detection + Tracking)
Arbitrary Hough Voting on detected voting points (Detection)
Constrained Hough Voting on image gradients (Pure Tracking)

15. Purpose of Candidate 1
Deals most of the frames where hypotheses from consecutive frames have strong correlation.

16. Purpose of Candidate 2
Automatically corrects result through searching for “much better” voting configurations (This is the power of detection, avoids error from tracking)

17. Purpose of Candidate 3
In the worst case where Type 1 voters fail, perform tracking by gradients from previous pose configuration.

18. Modeling under CRF: Background
A Conditional Random Field (CRF) discriminatively defines the joint posterior probability as the product of a set of potentials
The potentials are functions with hypotheses Hi being the variables. They are modeled in such a way that a larger potential value generally indicates a better hypothesis configuration.
CRF inference seeks to find the joint hypothesis configuration H that maximizes
Unary Potential
Pairwise Potential H1 H2 … HN X1 X2 XN

19. Modeling under CRF: Intuition
What are the hypothesis Hi?
E.g.: image pixel labels (FG/BG, Object Class, etc.), if it is a segmentation problem.
In our problem, Hi is the Hough Voting hypothesis: Hi = (r, θ).
X is the observation of voting point coordinates and their weights.
The unary potential corresponds to the exponential of Hough voting weights: exp(v(Hi)).
The pairwise potential corresponds to the inter-frame smoothness (tracking) constraint. H1 H2 … HN X1 X2 XN

20. No Structural Information
Hbd,1
Hbd,2
Hbd,N … X1 X2 XN
Hln,1
Hln,2
Hln,N … X1 X2 XN
Simplest Case: frame-wise independent Hough voting

21. Adding Inter-frame Structural Info.
Hbd,1
Hbd,2
Hbd,N … X1 X2 XN
Hln,1
Hln,2
Hln,N … X1 X2 XN
Adding temporal smoothness: Hough voting constrained by neighboring frames

22. Adding Intra-frame Structural Info.
Hbd,1
Hbd,2
Hbd,N … X1 X2 XN
Hln,1
Hln,2
Hln,N … X1 X2 XN
Adding Geometric Constraint: Hough voting constrained by both neighboring frames and intra-frame hypotheses

23. The Structured Hough Voting Model
Candidate Hypotheses Generation Unit
• • •
Coupled Structure Potential
Mode Selection Potential

24. The Structured Hough Voting Model

25. Candidate Hypotheses Generation Unit

26. Mode Selection Potential
Use decision tree to guide the mode selection.
The mode selection basically forces the output to be one of the candidate hypotheses, but allows discrepancy with the decision tree prediction with a penalty.

27. Coupled Structure Potential
The coupled structure potential captures two most important relations between a border hypothesis and a lane hypothesis
Parallelism
Distance

28. Inference
Conducting a whole inference each time given a new frame is computationally infeasible.
Relaxation: Initialize with the inferred state variable configuration of the previous t-1 frames and infer the current state variables, updating in an incremental way.
Inference procedure at t = 1: 1. Perform Hough voting for both border and lane marking 2. Perturbate hypotheses if geometric relationship violated (optional)
Inference procedure at t > 1: 1. Generate the 3 candidate hypotheses for both border and lane marking 2. Use decision tree to help selecting the best candidate 3. Perturbate candidate hypotheses if geometric relationship violated (optional) 4. Re-select the best candidate

29. Experiments: Adding Coupled Structure

30. Experiments: Qualitative Results
Ground Truth and Baseline methods:
Ground Truth
Independent Hough voting in each frame using the fired detector voting points
Hough voting using the triggered detector voting points constrained by previous frame
Adding gradient tracking to Baseline 2.
Kalman filter.
Proposed Method

31. Experiments: Quantitative Results

32. Highway Entrance Detection and Lane State Tracking

33. Summary Proposed the Structured Hough Voting Model
The proposed model can be theoretically formulated under a CRF
Fast real-time feature extraction and online inference
Achieves very robust and good performance under challenging scenarios and low quality inputs from production camera

34. Thank You! Q & A