Presentation is loading. Please wait.

Presentation is loading. Please wait.

Vision-Based Detection, Tracking and Classification of Vehicles using Features and Patterns with Automatic Camera Calibration Neeraj K. Kanhere Committee.

Similar presentations


Presentation on theme: "Vision-Based Detection, Tracking and Classification of Vehicles using Features and Patterns with Automatic Camera Calibration Neeraj K. Kanhere Committee."— Presentation transcript:

1 Vision-Based Detection, Tracking and Classification of Vehicles using Features and Patterns with Automatic Camera Calibration Neeraj K. Kanhere Committee members Dr. Stanley Birchfield (Advisor) Dr. John Gowdy Dr. Robert Schalkoff Dr. Wayne Sarasua Clemson University July 10 th 2008

2

3 Why detect and track vehicles ? Intelligent Transportation Systems (ITS)Intelligent Transportation Systems (ITS) Data collection for transportation engineering applicationsData collection for transportation engineering applications Incident detection and emergency responseIncident detection and emergency response Vehicle tracking Non-vision sensors Vision-based sensors No traffic disruption for installation and maintenance No traffic disruption for installation and maintenance Wide area detection with a single sensor Wide area detection with a single sensor Rich in information for manual inspection Rich in information for manual inspection Inductive loop detectors Inductive loop detectors Piezoelectric and Fiber Optic sensors Piezoelectric and Fiber Optic sensors The Infra-Red Traffic Logger (TIRTL) The Infra-Red Traffic Logger (TIRTL) Radar Radar Laser Laser

4 Available video commercial systems Autoscope (Econolite) Citilog Vantage (Iteris) Traficon

5 Problems with commercial systems Video

6 Related research Region/contour (Magee 04, Gupte et al. 02) Computationally efficient Computationally efficient Good results when vehicles are well separated Good results when vehicles are well separated 3D model (Ferryman et al. 98) Large number of models needed for different vehicle types Large number of models needed for different vehicle types Limited experimental results Limited experimental results Markov random field (Kamijo et al. 01) Good results on low angle sequences Good results on low angle sequences Accuracy drops by 50% when sequence is processed Accuracy drops by 50% when sequence is processed in true order in true order Feature tracking (Kim 08, Beymer et al. 97) Handles partial occlusions Handles partial occlusions Good accuracy for free flowing as well as congested traffic conditions Good accuracy for free flowing as well as congested traffic conditions

7 Overview of the research Vehicle detection and tracking Camera calibration Vehicle classification and traffic parameter extraction Scope of this research includes three problems FeaturesFeatures PatternsPatterns

8 Overview of the research Vehicle detection and tracking Camera calibration Vehicle classification and traffic parameter extraction Scope of this research includes three problems FeaturesFeatures PatternsPatterns

9 Problem of depth ambiguity Image plane Road Focal point Pinhole camera model All points along the ray map to the same image location Pinhole camera model All points along the ray map to the same image location

10 Problem of depth ambiguity An image point on the roof of the trailer is in the second lane Perspective view Top view 1 2 3 4

11 Problem of depth ambiguity The same image point is now in the last lane Perspective view Top view 1 2 3 4

12 Problem of depth ambiguity 1 2 3 4

13 Problem of scale change Grouping based on pixel distances fails when there is a large scale change in the scene.

14 Feature segmentation using 3D coordinates Normalized cuts on affinity matrix 4 Background modelCalibration Rigid motion constraint Background subtraction 3 Correspondence 5 Single frame estimation 2 1 Neeraj Kanhere, Stanley Birchfield and Shrinivas Pundlik (CVPR 2005) Neeraj Kanhere, Stanley Birchfield and Wayne Sarasua (TRR 2006)

15 Improved real-time implementation Neeraj Kanhere and Stanley Birchfield (IEEE Transactions on Intelligent Transportation Systems, 2008) Image frame Feature tracking PLP estimation Group stable features Correspondence, Validation and Classification Group unstable features Background subtraction Filtering Calibration Vehicle trajectories and data

16 Offline camera calibration 1) User draws two lines (red) corresponding to the edges of the road 2) User draws a line (green) corresponding to a known length along the road 3) Using either road width or camera height, a calibrated detection zone is computed

17 Background subtraction and filtering Only vehicles features are considered in further processing, reducing distraction from shadows Background features Shadow features Vehicle features

18 Plumb line projection (PLP) PLP is the projection of a feature on the road in the foreground image. With this projection, an estimate of 3D location of the feature is obtained. PLP is the projection of a feature on the road in the foreground image. With this projection, an estimate of 3D location of the feature is obtained.

19 Error in 3D estimation with PLP

20 Selecting stable features Features are stable if close to the ground, and slope is small at plumb line projection Features are stable if close to the ground, and slope is small at plumb line projection & Feature is stable if

21 Grouping of stable features Within each lane: Seed growing is used to group features with similar Y coordinate Across lanes: Groups with similar Y coordinate are merged if their combined width is acceptable

22 Grouping unstable features Location of an unstable feature is estimated with respect to each stable group using rigid motion constraint. Centroid of a stable feature group Unstable feature

23 Grouping unstable features Likelihood of the unstable feature is computed based on the estimated 3D location. Unstable feature is assigned to the group if it is likely to belong to that group a is best matching stable group. b is second best matching stable group. Unlikely to belong to any other group & & score for group i validity of location bias terms for large vehicles

24 Overview of the research Vehicle detection and tracking Camera calibration Vehicle classification and traffic parameter extraction Scope of this research includes three problems FeaturesFeatures PatternsPatterns

25 Feature grouping Pattern recognition Works under varying camera placement Needs a trained detector for significantly different viewpoints Eliminates false counts due to shadows but headlight reflections are still a problem Does not get distracted by headlight reflections Needs calibration Does not need calibration Combining pattern recognition Handles lateral occlusions but fails in case of back-to-back occlusions Handles back-to-back occlusions but difficult to handle lateral occlusions

26 Combining pattern recognition Lateral occlusion Back-to-back occlusion Handles lateral occlusions but fails in case of back-to-back occlusions Handles back-to-back occlusions but difficult to handle lateral occlusions A B A B

27 Boosted Cascade Vehicle Detector (BCVD) Negative training samples Positive training samples BCVD Offline supervised training of the detector using training images Vehicles detected in new images Training Detection Stage 1Stage 2 Stage n Rejected sub-windows …. Cascade architecture Run-time

28 Rectangular features with Integral images Haar-like rectangular features Viola and Jones, CVPR 2001 AB C D 12 3 4 sum(A) = val(1) sum(A+B) = val(2) sum(A+C) = val(3) sum(A+B+C+D) = val(4) sum(D) = val(4) – val(3) – val(2) + val(1) Fast computation and fast scaling

29 Sample results for static vehicle detection

30 Overview of the research Vehicle detection and tracking Camera calibration Vehicle classification and traffic parameter extraction Scope of this research includes three problems

31 Two calibration approaches Estimation of parameters for the assumed camera model Direct estimation of projective transform Goal is to estimate 11 elements of a matrix which transforms points in 3D to a 2D plane Harder to incorporate scene- specific knowledge Goal is to estimate camera parameters such as focal length and pose Easier to incorporate known quantities and constraints Image-world correspondences M [3x4] f, h, Φ, θ …

32 Direct estimation of projective matrix Atleast six points are required to estimate the 11 unknown parametes of the projective matrix

33 Camera calibration modes Assumptions: Flat road surface, zero skew, square pixels, and principal point at image center Known quantities: Width (W) or, Length (L), or Camera height (H)

34 Camera calibration modes Assumptions: Flat road surface, zero skew, square pixels, principal point at image center, and zero roll angle Known quantities: W or L or H

35 Camera calibration modes Assumptions: Flat road surface, zero skew, square pixels, principal point at image center, and zero roll angle Known quantities: {W, L} or {W, H} or {L, H}

36 Previous approaches to automatic calibration Song et al. (2006) Schoepflin and Dailey (2003) Dailey et al. (2000) Previous approaches: Need background image Sensitive to image processing parameters Affected by spillover Do not work at night Zhang et al. (2008)

37 Our approach to automatic calibration Does not depend on road markings Does not require scene specific parameters such as lane dimensions Works in presence of significant spill-over (low height) Works under night-time condition (no ambient light) Does not depend on road markings Does not require scene specific parameters such as lane dimensions Works in presence of significant spill-over (low height) Works under night-time condition (no ambient light) Neeraj Kanhere, Stanley Birchfield and Wayne Sarasua (TRR 2008)

38 Estimating vanishing points Vanishing point in the direction of travel is estimated using vehicle tracks Orthogonal vanishing point is estimated using strong gradients or headlights

39 Automatic calibration algorithm Focal length (pixels) Pan angle Tilt angle Camera height

40 Vehicle detection and tracking Camera calibration Vehicle classification and traffic parameter extraction Overview of the research

41 Vehicle classification based on axle counts 1.Motorcycles 2.Passenger cars 3.Other two-axle, four-tire single unit vehicles 4.Buses 5.Two-axle, six-tire, single-unit trucks 6.Three-axle single-unit trucks 7.Four or more axle single-unit trucks 8.Four or fewer axle single-trailer trucks 9.Five-axle single-trailer trucks 10.Six or more axle single-trailer trucks 11.Five or fewer axle multi-trailer trucks 12.Six axle multi-trailer trucks 13.Seven or more axle multi-trailer trucks FHWA highway manual lists 13 vehicle classes based on axle counts:

42 Vehicle classification based on length Thanks to Steven Jessberger (FHWA)

43 Vehicle classification based on length Four classes for length-based classification 1.Motorcycles 2.Passenger cars 3.Other two-axle, four-tire single unit vehicles 4.Buses 5.Two-axle, six-tire, single-unit trucks 6.Three-axle single-unit trucks 7.Four or more axle single-unit trucks 8.Four or fewer axle single-trailer trucks 9.Five-axle single-trailer trucks 10.Six or more axle single-trailer trucks 11.Five or fewer axle multi-trailer trucks 12.Six axle multi-trailer trucks 13.Seven or more axle multi-trailer trucks

44 Traffic parameters Volumes Lane counts Speeds Classification (three classes) Volumes Lane counts Speeds Classification (three classes)

45 Results

46

47 Quantitative results

48 Results for automatic camera calibration

49 Demo

50 Conclusion Future work should be aimed at: Extending automatic calibration to handle non-zero roll Improving and extending vehicle classification Long term testing of the system in day and night conditions A framework for combining pattern recognition with features Future work should be aimed at: Extending automatic calibration to handle non-zero roll Improving and extending vehicle classification Long term testing of the system in day and night conditions A framework for combining pattern recognition with features Research contributions: A system for detection, tracking and classification of vehicles Combination of feature tracking and background subtraction to group features in 3D Pattern recognition-based approach to detection and tracking of vehicles Automatic camera calibration technique which doesn’t need pavement markings and works even in absence of ambient light Research contributions: A system for detection, tracking and classification of vehicles Combination of feature tracking and background subtraction to group features in 3D Pattern recognition-based approach to detection and tracking of vehicles Automatic camera calibration technique which doesn’t need pavement markings and works even in absence of ambient light

51 QuestionsandDiscussion

52 Thank You

53

54 Previous approaches to automatic calibration Song et al. (2006) Known camera height Known camera height Needs background image Needs background image Depends on detecting road markings Depends on detecting road markings Schoepflin and Dailey (2003) Dailey et al. (2000) Avoids calculating camera Avoids calculating cameraparameters Based on assumptions that reduce the problem to 1-D Based on assumptions that reduce the problem to 1-Dgeometry Uses parameters from the Uses parameters from the distribution of vehicle lengths. Uses two vanishing points Uses two vanishing points Lane activity map sensitive of spill-over Lane activity map sensitive of spill-over Correction of lane activity map needs background image Correction of lane activity map needs background image Lane activity map Peaks at lane centers

55 Plumb line projection (PLP) PLP is the projection of a feature on the road in the foreground image. With this projection, an estimate of 3D location of the feature is obtained. PLP is the projection of a feature on the road in the foreground image. With this projection, an estimate of 3D location of the feature is obtained.


Download ppt "Vision-Based Detection, Tracking and Classification of Vehicles using Features and Patterns with Automatic Camera Calibration Neeraj K. Kanhere Committee."

Similar presentations


Ads by Google