Dr. Stanley Birchfield (Advisor) Vision-Based Detection, Tracking and Classification of Vehicles using Features and Patterns with Automatic Camera Calibration Neeraj K. Kanhere Committee members Dr. Stanley Birchfield (Advisor) Dr. John Gowdy Dr. Robert Schalkoff Dr. Wayne Sarasua Clemson University July 10th 2008

Vehicle tracking Why detect and track vehicles ? Non-vision sensors Intelligent Transportation Systems (ITS) Data collection for transportation engineering applications Incident detection and emergency response Non-vision sensors Inductive loop detectors Piezoelectric and Fiber Optic sensors The Infra-Red Traffic Logger (TIRTL) Radar Laser Other sensors provide classification up to 13 classes when the traffic is sparse. Problem with intrusive technologies Safety Easily damaged with street sweepers Road geometry makes it impossible to collect data sometimes Impractical for tracking Vision-based sensors No traffic disruption for installation and maintenance Wide area detection with a single sensor Rich in information for manual inspection

Autoscope (Econolite) Available video commercial systems Autoscope (Econolite) Citilog Vantage (Iteris) Traficon - All systems rely on manually specified detection zones which are prone to errors due to spillover and occlusions

Problems with commercial systems Video

Related research Region/contour (Magee 04, Gupte et al. 02) Computationally efficient Good results when vehicles are well separated 3D model (Ferryman et al. 98) Large number of models needed for different vehicle types Limited experimental results Markov random field (Kamijo et al. 01) Good results on low angle sequences Accuracy drops by 50% when sequence is processed in true order Feature tracking (Kim 08, Beymer et al. 97) Handles partial occlusions Good accuracy for free flowing as well as congested traffic conditions

Overview of the research Scope of this research includes three problems Vehicle detection and tracking Camera calibration Features Patterns Vehicle classification and traffic parameter extraction

Overview of the research Scope of this research includes three problems Vehicle detection and tracking Camera calibration Features Patterns Vehicle classification and traffic parameter extraction

Problem of depth ambiguity Image plane Focal point Road Pinhole camera model All points along the ray map to the same image location

An image point on the roof of the trailer is in the second lane Problem of depth ambiguity 1 2 3 4 Perspective view Top view An image point on the roof of the trailer is in the second lane

The same image point is now in the last lane Problem of depth ambiguity 1 2 3 4 Perspective view Top view The same image point is now in the last lane

Problem of depth ambiguity 1 2 3 4 1 2 3 4

Problem of scale change Grouping based on pixel distances fails when there is a large scale change in the scene.

Feature segmentation using 3D coordinates Background model Calibration 1 Background subtraction 5 make differences more clear. Make it clear next slides talk about new system Correspondence 4 Normalized cuts on affinity matrix 2 Single frame estimation 3 Rigid motion constraint Neeraj Kanhere, Stanley Birchfield and Shrinivas Pundlik (CVPR 2005) Neeraj Kanhere, Stanley Birchfield and Wayne Sarasua (TRR 2006)

Improved real-time implementation Image frame Feature tracking Background subtraction Filtering Group stable features Differences: handling shadows as a preprocessing step rather than post-processing Focus on segmenting features which can be segmented reliably rather than segmenting all features PLP estimation Correspondence, Validation and Classification Group unstable features Vehicle trajectories and data Calibration Neeraj Kanhere and Stanley Birchfield (IEEE Transactions on Intelligent Transportation Systems, 2008)

Offline camera calibration 1) User draws two lines (red) corresponding to the edges of the road 2) User draws a line (green) corresponding to a known length along the road 3) Using either road width or camera height, a calibrated detection zone is computed Differences: handling shadows as a preprocessing step rather than post-processing Focus on segmenting features which can be segmented reliably rather than segmenting all features

Background subtraction and filtering colorize right figure Differences: handling shadows as a preprocessing step rather than post-processing Focus on segmenting features which can be segmented reliably rather than segmenting all features Background features Vehicle features Shadow features Only vehicles features are considered in further processing, reducing distraction from shadows

Plumb line projection (PLP) use notation to relate p-u and q-v note that p is unknown PLP is the projection of a feature on the road in the foreground image. With this projection, an estimate of 3D location of the feature is obtained.

Error in 3D estimation with PLP error is greater for points higher up show z-tilde Differences: handling shadows as a preprocessing step rather than post-processing Focus on segmenting features which can be segmented reliably rather than segmenting all features

Selecting stable features Feature is stable if & zai epsilon Features are stable if close to the ground, and slope is small at plumb line projection

Grouping of stable features grouping rather than segmentation show steps in diagram Within each lane: Seed growing is used to group features with similar Y coordinate Across lanes: Groups with similar Y coordinate are merged if their combined width is acceptable

Grouping unstable features Location of an unstable feature is estimated with respect to each stable group using rigid motion constraint. Centroid of a stable feature group Unstable feature split into two slides and show figure more clearly define all terms

Grouping unstable features Likelihood of the unstable feature is computed based on the estimated 3D location. score for group i validity of location bias terms for large vehicles split into two slides and show figure more clearly define all terms Unstable feature is assigned to the group if it is likely to belong to that group Unlikely to belong to any other group & a is best matching stable group. b is second best matching stable group.

Overview of the research Scope of this research includes three problems Vehicle detection and tracking Camera calibration Features Patterns Vehicle classification and traffic parameter extraction

Combining pattern recognition Feature grouping Pattern recognition Works under varying camera placement Needs a trained detector for significantly different viewpoints Eliminates false counts due to shadows but headlight reflections are still a problem Does not get distracted by headlight reflections Does not need calibration Needs calibration Handles lateral occlusions but fails in case of back-to-back occlusions Handles back-to-back occlusions but difficult to handle lateral occlusions

Back-to-back occlusion Combining pattern recognition Lateral occlusion Back-to-back occlusion Handles lateral occlusions but fails in case of back-to-back occlusions Handles back-to-back occlusions but difficult to handle lateral occlusions A B A B

Boosted Cascade Vehicle Detector (BCVD) Offline supervised training of the detector using training images Vehicles detected in new images Training Run-time Positive training samples BCVD Negative training samples Cascade architecture Stage 1 Stage 2 Stage n Detection …. Rejected sub-windows

Rectangular features with Integral images Haar-like rectangular features Fast computation and fast scaling A B C D 1 2 3 4 sum(A) = val(1) sum(A+B) = val(2) sum(A+C) = val(3) sum(A+B+C+D) = val(4) sum(D) = val(4) – val(3) – val(2) + val(1) Viola and Jones, CVPR 2001

Sample results for static vehicle detection min size and not trained for view from behind

Overview of the research Scope of this research includes three problems Vehicle detection and tracking Camera calibration remind audience that two sections are not as lengthy Vehicle classification and traffic parameter extraction

Two calibration approaches Image-world correspondences f, h, Φ, θ … M[3x4] M[3x4] Direct estimation of projective transform Estimation of parameters for the assumed camera model Goal is to estimate 11 elements of a matrix which transforms points in 3D to a 2D plane Harder to incorporate scene-specific knowledge Goal is to estimate camera parameters such as focal length and pose Easier to incorporate known quantities and constraints

Direct estimation of projective matrix Advantage is that we don’t need to assume zero roll, square pixels etc. replace word manual explain modes (may be cover 3, 4 modes which are important) Atleast six points are required to estimate the 11 unknown parametes of the projective matrix

Camera calibration modes replace word manual explain modes (may be cover 3, 4 modes which are important) Assumptions: Flat road surface, zero skew, square pixels, and principal point at image center Known quantities: Width (W) or, Length (L), or Camera height (H)

Camera calibration modes replace word manual explain modes (may be cover 3, 4 modes which are important) Assumptions: Flat road surface, zero skew, square pixels, principal point at image center, and zero roll angle Known quantities: W or L or H

Camera calibration modes replace word manual explain modes (may be cover 3, 4 modes which are important) Assumptions: Flat road surface, zero skew, square pixels, principal point at image center, and zero roll angle Known quantities: {W, L} or {W, H} or {L, H}

Schoepflin and Dailey (2003) Previous approaches to automatic calibration Schoepflin and Dailey (2003) Dailey et al. (2000) Song et al. (2006) Zhang et al. (2008) Common to all: Do not work in night time Need this? Instead cite prv work in next slide... in contrast to ..... our approach: Previous approaches: Need background image Sensitive to image processing parameters Affected by spillover Do not work at night

Neeraj Kanhere, Stanley Birchfield and Wayne Sarasua (TRR 2008) Our approach to automatic calibration Point out under-laying assumptions (zero roll, square pixels and sufficient pan angle) Fix labels Mention VVWF\ Fix decision box Does not depend on road markings Does not require scene specific parameters such as lane dimensions Works in presence of significant spill-over (low height) Works under night-time condition (no ambient light) Neeraj Kanhere, Stanley Birchfield and Wayne Sarasua (TRR 2008)

Estimating vanishing points Vanishing point in the direction of travel is estimated using vehicle tracks Orthogonal vanishing point is estimated using strong gradients or headlights

Automatic calibration algorithm Focal length (pixels) Pan angle Tilt angle Camera height

Overview of the research Vehicle detection and tracking Camera calibration Vehicle classification and traffic parameter extraction

FHWA highway manual lists 13 vehicle classes based on axle counts: Vehicle classification based on axle counts FHWA highway manual lists 13 vehicle classes based on axle counts: Motorcycles Passenger cars Other two-axle, four-tire single unit vehicles Buses Two-axle, six-tire, single-unit trucks Three-axle single-unit trucks Four or more axle single-unit trucks Four or fewer axle single-trailer trucks Five-axle single-trailer trucks Six or more axle single-trailer trucks Five or fewer axle multi-trailer trucks Six axle multi-trailer trucks Seven or more axle multi-trailer trucks

Vehicle classification based on length four boxes mention last class not evaluated yet mention how diff agencies classify on length show graph or other graphical representation Thanks to Steven Jessberger (FHWA)

Four classes for length-based classification Vehicle classification based on length Four classes for length-based classification Motorcycles Passenger cars Other two-axle, four-tire single unit vehicles Buses Two-axle, six-tire, single-unit trucks Three-axle single-unit trucks Four or more axle single-unit trucks Four or fewer axle single-trailer trucks Five-axle single-trailer trucks Six or more axle single-trailer trucks Five or fewer axle multi-trailer trucks Six axle multi-trailer trucks Seven or more axle multi-trailer trucks four boxes mention last class not evaluated yet mention how diff agencies classify on length show graph or other graphical representation

Traffic parameters Volumes Lane counts Speeds Classification (three classes)

Results

Quantitative results results at end (table after video)

Results for automatic camera calibration

Demo

Conclusion Research contributions: A system for detection, tracking and classification of vehicles Combination of feature tracking and background subtraction to group features in 3D Pattern recognition-based approach to detection and tracking of vehicles Automatic camera calibration technique which doesn’t need pavement markings and works even in absence of ambient light Future work should be aimed at: Extending automatic calibration to handle non-zero roll Improving and extending vehicle classification Long term testing of the system in day and night conditions A framework for combining pattern recognition with features

Questions and Discussion

Thank You

Schoepflin and Dailey (2003) Previous approaches to automatic calibration Dailey et al. (2000) Avoids calculating camera parameters Based on assumptions that reduce the problem to 1-D geometry Uses parameters from the distribution of vehicle lengths. Schoepflin and Dailey (2003) Lane activity map Peaks at lane centers Uses two vanishing points Lane activity map sensitive of spill-over Correction of lane activity map needs background image Common to all: Do not work in night time Need this? Instead cite prv work in next slide... in contrast to ..... our approach: Song et al. (2006) Known camera height Needs background image Depends on detecting road markings

Plumb line projection (PLP) use notation to relate p-u and q-v note that p is unknown PLP is the projection of a feature on the road in the foreground image. With this projection, an estimate of 3D location of the feature is obtained.