1. Dr. Stanley Birchfield (Advisor)
Vision-Based Detection, Tracking and Classification of Vehicles using Features and Patterns with Automatic Camera Calibration
Neeraj K. Kanhere
Committee members
Dr. Stanley Birchfield (Advisor)
Dr. John Gowdy
Dr. Robert Schalkoff
Dr. Wayne Sarasua
Clemson University
July 10th 2008

3. Vehicle tracking Why detect and track vehicles ? Non-vision sensors
Intelligent Transportation Systems (ITS)
Data collection for transportation engineering applications
Incident detection and emergency response
Non-vision sensors
Inductive loop detectors
Piezoelectric and Fiber Optic sensors
The Infra-Red Traffic Logger (TIRTL)
Radar
Laser
Other sensors provide classification up to 13 classes when the traffic is sparse.
Problem with intrusive technologies
Safety
Easily damaged with street sweepers
Road geometry makes it impossible to collect data sometimes
Impractical for tracking
Vision-based sensors
No traffic disruption for installation and maintenance
Wide area detection with a single sensor
Rich in information for manual inspection

4. Autoscope (Econolite)
Available video commercial systems
Autoscope (Econolite)
Citilog
Vantage (Iteris)
Traficon
- All systems rely on manually specified detection zones which are prone to errors due to spillover and occlusions

5. Problems with commercial systems
Video

6. Related research Region/contour (Magee 04, Gupte et al. 02)
Computationally efficient
Good results when vehicles are well separated
3D model (Ferryman et al. 98)
Large number of models needed for different vehicle types
Limited experimental results
Markov random field (Kamijo et al. 01)
Good results on low angle sequences
Accuracy drops by 50% when sequence is processed
in true order
Feature tracking (Kim 08, Beymer et al. 97)
Handles partial occlusions
Good accuracy for free flowing as well as congested traffic conditions

7. Overview of the research
Scope of this research includes three problems
Vehicle detection and tracking
Camera calibration
Features
Patterns
Vehicle classification and traffic parameter extraction

8. Overview of the research
Scope of this research includes three problems
Vehicle detection and tracking
Camera calibration
Features
Patterns
Vehicle classification and traffic parameter extraction

9. Problem of depth ambiguity
Image plane
Focal point
Road
Pinhole camera model
All points along the ray map to the same image location

10. An image point on the roof of the trailer is in the second lane
Problem of depth ambiguity
Perspective view
Top view
An image point on the roof of the trailer is in the second lane

11. The same image point is now in the last lane
Problem of depth ambiguity
Perspective view
Top view
The same image point is now in the last lane

12. Problem of depth ambiguity

13. Problem of scale change
Grouping based on pixel distances fails when there is a large scale change in the scene.

14. Feature segmentation using 3D coordinates
Background model
Calibration 1
Background subtraction 5
make differences more clear.
Make it clear next slides talk about new system
Correspondence 4
Normalized cuts on
affinity matrix 2
Single frame estimation 3
Rigid motion constraint
Neeraj Kanhere, Stanley Birchfield and Shrinivas Pundlik (CVPR 2005)
Neeraj Kanhere, Stanley Birchfield and Wayne Sarasua (TRR 2006)

15. Improved real-time implementation
Image frame
Feature tracking
Background subtraction
Filtering
Group stable features
Differences: handling shadows as a preprocessing step rather than post-processing
Focus on segmenting features which can be segmented reliably rather than segmenting all features
PLP estimation
Correspondence, Validation and Classification
Group unstable features
Vehicle trajectories and data
Calibration
Neeraj Kanhere and Stanley Birchfield (IEEE Transactions on Intelligent Transportation Systems, 2008)

16. Offline camera calibration
1) User draws two lines (red) corresponding to the edges of the road
2) User draws a line (green) corresponding to a known length along the road
3) Using either road width or camera height, a calibrated detection zone is computed
Differences: handling shadows as a preprocessing step rather than post-processing
Focus on segmenting features which can be segmented reliably rather than segmenting all features

17. Background subtraction and filtering
colorize right figure
Differences: handling shadows as a preprocessing step rather than post-processing
Focus on segmenting features which can be segmented reliably rather than segmenting all features
Background features
Vehicle features
Shadow features
Only vehicles features are considered in further processing, reducing distraction from shadows

18. Plumb line projection (PLP)
use notation to relate p-u and q-v
note that p is unknown
PLP is the projection of a feature on the road in the foreground image.
With this projection, an estimate of 3D location of the feature is obtained.

19. Error in 3D estimation with PLP
error is greater for points higher up
show z-tilde
Differences: handling shadows as a preprocessing step rather than post-processing
Focus on segmenting features which can be segmented reliably rather than segmenting all features

20. Selecting stable features
Feature is stable if
&
zai
epsilon
Features are stable if
close to the ground, and
slope is small at plumb line projection

21. Grouping of stable features
grouping rather than segmentation
show steps in diagram
Within each lane: Seed growing is used to group features with similar Y coordinate
Across lanes: Groups with similar Y coordinate are merged if their combined width is acceptable

22. Grouping unstable features
Location of an unstable feature is estimated with respect to each stable group using rigid motion constraint.
Centroid of a stable feature group
Unstable feature
split into two slides and show figure more clearly
define all terms

23. Grouping unstable features
Likelihood of the unstable feature is computed based on the estimated 3D location.
score for group i
validity of location
bias terms for large vehicles
split into two slides and show figure more clearly
define all terms
Unstable feature is assigned to the group if it is likely to belong to that group
Unlikely to belong to any other group
&
a is best matching stable group. b is second best matching stable group.

24. Overview of the research
Scope of this research includes three problems
Vehicle detection and tracking
Camera calibration
Features
Patterns
Vehicle classification and traffic parameter extraction

25. Combining pattern recognition
Feature grouping
Pattern recognition
Works under varying camera
placement
Needs a trained detector for
significantly different viewpoints
Eliminates false counts due to shadows but headlight reflections are still a problem
Does not get distracted by headlight reflections
Does not need calibration
Needs calibration
Handles lateral occlusions but fails in case of back-to-back occlusions
Handles back-to-back occlusions but difficult to handle lateral occlusions

26. Back-to-back occlusion
Combining pattern recognition
Lateral occlusion
Back-to-back occlusion
Handles lateral occlusions but fails in case of back-to-back occlusions
Handles back-to-back occlusions but difficult to handle lateral occlusions A B A B

27. Boosted Cascade Vehicle Detector (BCVD)
Offline supervised training of the detector using training images
Vehicles detected in new images
Training
Run-time
Positive training samples
BCVD
Negative training samples
Cascade architecture
Stage 1
Stage 2
Stage n
Detection ….
Rejected sub-windows

28. Rectangular features with Integral images
Haar-like rectangular features
Fast computation and fast scaling A B C D 1 2 3 4
sum(A) = val(1)
sum(A+B) = val(2)
sum(A+C) = val(3)
sum(A+B+C+D) = val(4)
sum(D) = val(4) – val(3) – val(2) + val(1)
Viola and Jones, CVPR 2001

29. Sample results for static vehicle detection
min size and not trained for view from behind

30. Overview of the research
Scope of this research includes three problems
Vehicle detection and tracking
Camera calibration
remind audience that two sections are not as lengthy
Vehicle classification and traffic parameter extraction

31. Two calibration approaches
Image-world correspondences
f, h, Φ, θ …
M[3x4]
M[3x4]
Direct estimation of
projective transform
Estimation of parameters for the assumed camera model
Goal is to estimate 11 elements of a matrix which transforms points in 3D to a 2D plane
Harder to incorporate scene-specific knowledge
Goal is to estimate camera parameters such as focal length and pose
Easier to incorporate known quantities and constraints

32. Direct estimation of projective matrix
Advantage is that we don’t need to assume zero roll, square pixels etc.
replace word manual
explain modes (may be cover 3, 4 modes which are important)
Atleast six points are required to estimate the 11 unknown parametes of the projective matrix

33. Camera calibration modes
replace word manual
explain modes (may be cover 3, 4 modes which are important)
Assumptions:
Flat road surface, zero skew, square pixels, and principal point at image center
Known quantities:
Width (W) or, Length (L), or Camera height (H)

34. Camera calibration modes
replace word manual
explain modes (may be cover 3, 4 modes which are important)
Assumptions:
Flat road surface, zero skew, square pixels, principal point at image center, and zero roll angle
Known quantities:
W or L or H

35. Camera calibration modes
replace word manual
explain modes (may be cover 3, 4 modes which are important)
Assumptions:
Flat road surface, zero skew, square pixels, principal point at image center, and zero roll angle
Known quantities:
{W, L} or {W, H} or {L, H}

36. Schoepflin and Dailey (2003)
Previous approaches to automatic calibration
Schoepflin and Dailey (2003)
Dailey et al. (2000)
Song et al. (2006)
Zhang et al. (2008)
Common to all: Do not work in night time
Need this?
Instead cite prv work in next slide... in contrast to our approach:
Previous approaches:
Need background image
Sensitive to image processing parameters
Affected by spillover
Do not work at night

37. Neeraj Kanhere, Stanley Birchfield and Wayne Sarasua (TRR 2008)
Our approach to automatic calibration
Point out under-laying assumptions (zero roll, square pixels and sufficient pan angle)
Fix labels
Mention VVWF\
Fix decision box
Does not depend on road markings
Does not require scene specific parameters such as lane dimensions
Works in presence of significant spill-over (low height)
Works under night-time condition (no ambient light)
Neeraj Kanhere, Stanley Birchfield and Wayne Sarasua (TRR 2008)

38. Estimating vanishing points
Vanishing point in the direction of travel is estimated using vehicle tracks
Orthogonal vanishing point is estimated using strong gradients or headlights

39. Automatic calibration algorithm
Focal length (pixels)
Pan angle
Tilt angle
Camera height

40. Overview of the research
Vehicle detection and tracking
Camera calibration
Vehicle classification and traffic parameter extraction

41. FHWA highway manual lists 13 vehicle classes based on axle counts:
Vehicle classification based on axle counts
FHWA highway manual lists 13 vehicle classes based on axle counts:
Motorcycles
Passenger cars
Other two-axle, four-tire single unit vehicles
Buses
Two-axle, six-tire, single-unit trucks
Three-axle single-unit trucks
Four or more axle single-unit trucks
Four or fewer axle single-trailer trucks
Five-axle single-trailer trucks
Six or more axle single-trailer trucks
Five or fewer axle multi-trailer trucks
Six axle multi-trailer trucks
Seven or more axle multi-trailer trucks

42. Vehicle classification based on length
four boxes
mention last class not evaluated yet
mention how diff agencies classify on length
show graph or other graphical representation
Thanks to Steven Jessberger (FHWA)

43. Four classes for length-based classification
Vehicle classification based on length
Four classes for length-based classification
Motorcycles
Passenger cars
Other two-axle, four-tire single unit vehicles
Buses
Two-axle, six-tire, single-unit trucks
Three-axle single-unit trucks
Four or more axle single-unit trucks
Four or fewer axle single-trailer trucks
Five-axle single-trailer trucks
Six or more axle single-trailer trucks
Five or fewer axle multi-trailer trucks
Six axle multi-trailer trucks
Seven or more axle multi-trailer trucks
four boxes
mention last class not evaluated yet
mention how diff agencies classify on length
show graph or other graphical representation

44. Traffic parameters Volumes Lane counts Speeds
Classification (three classes)

45. Results

47. Quantitative results
results at end (table after video)

48. Results for automatic camera calibration

49. Demo

50. Conclusion Research contributions:
A system for detection, tracking and classification of vehicles
Combination of feature tracking and background subtraction to group features in 3D
Pattern recognition-based approach to detection and tracking of vehicles
Automatic camera calibration technique which doesn’t need pavement markings and works even in absence of ambient light
Future work should be aimed at:
Extending automatic calibration to handle non-zero roll
Improving and extending vehicle classification
Long term testing of the system in day and night conditions
A framework for combining pattern recognition with features

51. Questions
and
Discussion

52. Thank You

54. Schoepflin and Dailey (2003)
Previous approaches to automatic calibration
Dailey et al. (2000)
Avoids calculating camera
parameters
Based on assumptions that reduce the problem to 1-D
geometry
Uses parameters from the
distribution of vehicle lengths.
Schoepflin and Dailey (2003)
Lane activity map
Peaks at lane centers
Uses two vanishing points
Lane activity map sensitive of spill-over
Correction of lane activity map needs background image
Common to all: Do not work in night time
Need this?
Instead cite prv work in next slide... in contrast to our approach:
Song et al. (2006)
Known camera height
Needs background image
Depends on detecting road markings

55. Plumb line projection (PLP)
use notation to relate p-u and q-v
note that p is unknown
PLP is the projection of a feature on the road in the foreground image.
With this projection, an estimate of 3D location of the feature is obtained.