Vehicle Segmentation and Tracking From a Low-Angle Off-Axis Camera Neeraj K. Kanhere Committee members Dr. Stanley Birchfield Dr. Robert Schalkoff Dr. Wayne Sarasua Clemson University July 14 th 2005

Why detect and track vehicles ?  Intelligent Transportation Systems (ITS)  Data collection for transportation engineering applications  Because it's a challanging problem! Vehicle Tracking Loop detectors Tracking using Vision  Low per unit cost  Field experience  No traffic disruption  Wide area detection  Rich in information  No tracking  Maintenance difficult  Computationally demanding  Expensive

Available Commercial Systems AUTOSCOPE (Image Sensing Systems)  Has been around for more than a decade  Dedicated Hardware  Reliable operation  Good accuracy with favorable camera placement VANTAGE (Iteris)  New in market  Accuracy has been found to be lower than Autoscope

Related Research Region/Contour Based  Computationally efficient  Good results when vehicles are well separated 3D Model Based  Large number of models needed for different vehicle types  Limited experimental results Markov Random Field  Good results on low angle sequences  Accuracy drops by 50% when sequence is processed in true order Feature Tracking Based  Handles partial occlusions  Good accuracy for free flowing as well as congested traffic conditions

Factors To be Considered High angle Low angle  Planar motion assumption  Well-separated vehicles  Relatively easy  More depth variation  Occlusions  A difficult problem

Overview of the Approach Offline Calibration Background model Frame-Block #1 Frame-Block #3 Frame-Block #2 Feature Tracking Estimation of 3-D Location Grouping segmented #1 segmented #2 segmented #3 Counts, Speeds and ClassificationCounts, Classification Block Correspondence and Post Processing

Processing a Frame-Block  Multiple frames are needed for motion information  Tradeoff between number of features and amount of motion  Typically 5-15 frames yield good results Block # n Block # n+1 frames #features in block #features in block #frames in block #frames in block Overlap

Background Model Time Domain Median filtering  For each pixel, values observed over time  Median value among observations  Simple and effective for the sequences considered  Adaptive algorithm required for long term modeling

Frame Differencing  Partially occluded vehicles appear as single blob  Effectively segments well-separated vehicles  Goal is to get filled connected components

` Offline Calibration  Required for estimation of world coordinates  Provides geometric information about the scene  Involves estimating 11 unknown parameters  Needs atleast six world-image correspondances Control points

Calibration Process  Using scene features to estimate correspondences  Standard lane width (e.g. 12 feet on an Interstate)  Vehicle class dimensions (truck length of 70 feet)  Relies on human judgment and prone to errors  Approximate calibration is good enough

Estimation using Single Frame  Box-model for vehicles  Road projection using foreground mask  Works for orthogonal surfaces cameravehicle Road plane

Selecting Stable Features  Shadows, partial occlusions will result into wrong estimates  Planar motion assumption is violated more for features higher up  Select stable features, which are closer to road  Use stable features to re-estimate world coordinates of other features

Estimation Using Motion ➢ Estimate coordinates with respect to each stable feature ➢ Choose coordinates which minimized weighted sum of euclidean distance and trajectory error  Rigid body under translation  Estimate coordinates with respect to each stable feature  Select the coordinates minimizing weighted sum of Euclidean distance and trajectory error Coordinates of P are unknown Coordinates of Q are known R and H denote backprojections 0 : first frame of the block t : last frame of the block Δ: Translation of corresponding point

Affinity Matrix  Each element represents the similarity between corresponding features  Three quantities contribute to the affinity matrix  Euclidean distance (A D ), Trajectory Error (A E ) and Background- Content (A B )  Normalized Cut is used for segmentation  Number of Cuts is not known

Incremental Cuts  We apply normalized cut to initial A with increasing number of cuts  For each successive cut, segmented groups are analyzed till valid groups are found  Valid Group: meeting dimensional criteria  Elements corresponding to valid groups are removed from A and process repeated starting from single cut Avoids specifying a threshold for the number of cuts

Correspondence Over Blocks  Formulated as a problem of finding maximum wieght graph  Nodes represent segmented groups  Edge weights represent number features common over two blocks a n : groups in block N b n : groups in block N+1

Results

Results

Conclusion  A novel approach based on feature point tracking  Key part of the technique is estimation of 3-D world coordinates  Results demonstrate the ability to correctly segment vehicles under severe partial occlusions  Handling shadows explicitly  Improving processing speed  Robust block-correspondance Future Work

Questions ?

Thank You !