1. Vision-based 3D Bicycle Tracking using Deformable Part Model and Interacting Multiple Model Filter May 11, 2011 Hyunggi Cho 1, Paul Rybski 1,2, and Wende Zhang 3 1 Electrical and Computer Engineering School of Engineering Carnegie Mellon University 3 The Electrical and Controls Integration Lab. General Motors R&D 2 Robotics Institute School of Computer Science Carnegie Mellon University

2. Outline  Motivation and Overview  Bicycle Detection  Bicycle Tracking  Experimental Results  Conclusion and Future work

3. Motivation  Motivation -In 2009, 630 bicyclists were killed and 51,000 were injured in traffic accidents in the United States*. -Bicyclists and pedestrians are the most vulnerable traffic participants. -There is less research on bicyclist detection and tracking compared to that of pedestrians. *http://www.nhtsa.gov Movie clip: Bicycle messengers in New York City (Youtube)

4. Sensors on Cars Source : http://www.tartanracing.org

5. System Overview Input : video  Track bicycles using a single video camera mounted on a vehicle System output : position & velocity  System block diagram Bicycle Detector Bicycle Tracker Bicycle’s position & velocity

6. Bicycle Detection – Deformable Part Model HOG Detector  Eight view-based bicycle detection root filters coarse resolution part filters finer resolution deformation models P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained Part Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010

7. DPM HOG Detector – Object hypothesis  Bicycle detection process P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained Part Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010

8. DPM HOG Detector – Performance Analysis  Examples of bicycle detection Test images from Google image

9. DPM HOG Detector – Performance Analysis Terminology Eight view  Precision-Recall Curve (VOC2009 + Ours) Recall (True Positive Rate) Precision True Positive False Negative False Positive Total No. of Positive Average Precision ( Area Under Curve ) Training Set : 350 positive / 3300 negative Test Set : : VOC2009 ‘val’ dataset

10. Overview of our single bicycle tracking system Prediction stageUpdate stage y x  Kalman filter-based tracking state space : 2D image space

11. Overview of our single bicycle tracking system Model  Dynamic system model  Motion model : constant velocity y x : Dynamic equation : Measurement equation : Initial state : Process noise : Measurement noise

12. Overview of our single bicycle tracking system  Measurement model : perspective projection rotation matrix translation vector focal length optical center Horizon Image plane height Image

13. IMM - Choosing a model set Constant Velocity Coordinated Turn Constant Velocity Simplified Bicycle with CV and CY angle  Model Set I  Model Set II  Model Set III GM CV CA Constant Velocity Constant Acceleration CV CT CV SB

14. IMM - Choosing a model set  Constant Velocity model :  Simplified Bicycle model :

15. IMM - Performance analysis We tested the IMM method on the GM bicycle dataset Test Set : 6 sequences with a stationary GM test vehicle Data statistics : Size : 320x240, FPS : 10~12, No. of bicycles : 1  IMM Tracking performance  Details of six bicycle sequences ( SM vs. IMM ) Seq.ego-vehicle bicycle RMSE(SM)RMSE(IMM) ‘seq1’stationarylaterally0.01830.0216 ‘seq2’stationary longitudinally 6.62076.6196 ‘seq3’stationaryrandomly0.15150.1443 ‘seq4’movinglaterally2.34932.3860 ‘seq5’moving longitudinally 7.08846.860 ‘seq6’movingrandomly11.092910.6281

16. IMM - Performance analysis  Sequence 3 case

17. Multiple bicycle tracking using a Rao-Blackwellized particle filter Single bicycle tracking : We solved this problem. Data association : Given a measurement, which target produced it, if any ? Unknown number of targets : How many bicycles are there ?  Multiple bicycle tracking problem  In our multiple bicycle tracking case Particle filter Kalman filter Joint state vector : Data association indicator :Target visibility indicator Simo Särkkä, Aki Vehtari, and Jouko Lampinen (2007). Rao-Blackwellized Particle Filter for Multiple Target Tracking. Information Fusion Journal, Volume 8, Issue 1, Pages 2-15

18. Multiple bicycle tracking using a Rao-Blackwellized particle filter Only one target can die is associated with : (a) Clutter (b) One of the existing targets (c) A newborn target  All possible events between two measurements and  Example t-2 t-1 t t-2 t-1 t y1y1 y2y2 y3y3 : Target : Measurement : Trajectory  Particle filter for data association problem

19. Experimental Results We tested our detection/tracking system on our bicycle dataset Test Set : A challenging sequence from a moving Boss (so called ‘Free for all’) Data statistics : Size : 320 x 240, Frame rate : 13~15 frame per second Sensor coverage area  Tracking performance 15 m 5 m 0 m 4 m Minimum pixel size HOG Detector : 32x64 4 m

20. Experimental Results - data collection Rank & RateDescriptionIllustration 10 (3.9%) Motorist Overtaking-Other The motorist was overtaking a bicyclis ts. 9 (4.3%) Bicyclist Left Turn in front of traffic The bicyclist made a left turn in front o f traffic travelling in the same direction. 8 (4.4%) Ride Out At Midblock The bicyclist entered the roadway at a shoulder or curb midblock location.  US Bicyclists Crash Types – Top 10covering 61% database samples W.H. Hunter, W.E. Pein, and J.C. Stutts, Bicycle Crash Types: A 1990's Informational Guide, Publication No. FHWA-RD-96-104, Federal Highway Administration, Washington, DC, April, 1997

21. Experimental Results - data collection Rank & RateDescriptionIllustration 7 (4.7%) Motorist Right Turn The motorist was making a right turn and the bicyclist was riding in either the same or opposing direction. 6 (5.1%) Ride Out At Residential Driveway The bicyclist entered the roadway from a residential driveway or alley. 5 (5.9%) Motorist Left Turn– Facing Bicyclist The motorist made a left turn while facing the approaching bicyclist. 4 (6.9%) Ride Out At Midblock The motorist was entering the roadway from a driveway or alley W.H. Hunter, W.E. Pein, and J.C. Stutts, Bicycle Crash Types: A 1990's Informational Guide, Publication No. FHWA-RD-96-104, Federal Highway Administration, Washington, DC, April, 1997

22. Experimental Results - data collection Rank & RateDescriptionIllustration 3 (7.1%) Ride Out At Intersection - Other The crash occurred at an intersection, signalized or uncontrolled, at which the bicyclist failed to yield. 2 (9.3%) Drive Out At Stop Sign The crash occurred at an intersection at which the motorist was facing a stop sign. 1 (9.7%) Ride Out At Stop Sign The crash occurred at an intersection at which the bicyclist was facing a stop sign or flashing red light. We categorized the upper scenarios into 4 different classes in terms of bicycle motion patterns !!! W.H. Hunter, W.E. Pein, and J.C. Stutts, Bicycle Crash Types: A 1990's Informational Guide, Publication No. FHWA-RD-96-104, Federal Highway Administration, Washington, DC, April, 1997

23. Experimental Results - data collection

24. Experimental Results – Performance analysis  Scenario – Random moving case (‘Free for all’) 2D Bounding box {view (x coordinate, y coordinate)} Uncertainty level

25. Experimental Results – Performance analysis  Scenario – Random moving case with 3D visualization 2D Bounding box {view (x coordinate, y coordinate)} Uncertainty level

26. Summary and Future Work  Summary  Data collection - Based on bicycle accident statistics  Detection part - Applied DPM HOG detector into a multiple bicycle tracking system  Tracking part - Incorporate Interacting Multiple Model (IMM) algorithm into our multiple bicycle tracking system to exploit several types of motion models - RBPF data association algorithm  Future work  Real-time C++ implementation ( > 10fps)  Integration the system into the perception system of our autonomous vehicles at CMU

27. Q&A

28. Single bicycle tracking using an IMM True motion of a bicycle cannot be exactly modeled by just one model, only be sufficiently approximated by using several motion models for representing dynamic driving behaviors of a target (i.e., maneuverings of a bicycle). The IMM filter runs several motion models in parallel and estimates a state by computing a weighted sum of several filter results which are based on different motion models.  Main idea of Interacting Multiple Model (IMM)

29. Integral HOG Detector - Performance Analysis II

30.  Related works Vision-based Bicycle Detector PublicationSensorsFeaturesAttention Focusing StageClassification stage Gavrila IV2004 StereoEdge map Stereo-based depth Chamfer matching Texture classification Papageorgiou IJCV2000 MonocularHaar wavelet Can add motion/stereo modules for preprocessing SVM classifier on Haar wavelet features Viola & Jones CVPR2001 MonocularHaar-like waveletNAAdaBoost Dalal & Trigg CVPR2005 MonocularHOGNALinear SVM on HOG Zhu & Avidan CVPR2006 MonocularIntegral HOGNA AdaBoost with linear SVM as a weak classifier Miko. ECCV2004 Monocular SIFT-like orientation feature NAAdaBoost Wu & Nevatia ICCV2005 MonocularEdgeletsNA AdaBoost with hard- coded mid-level features Felzenszwalb CVPR2008 MonocularHOGNA Deformable part model with LSVM