1. Personal Driving Diary: Constructing a Video Archive of Everyday Driving Events IEEE workshop on Motion and Video Computing ( WMVC) 2011 IEEE Workshop on Applications of Computer Vision (WACV) 2011 Electronics and Telecommunications Research Institute M. S. Ryoo, Jae-Yeong Lee, Ji Hoon Joung, Sunglok Choi, and Wonpil Yu

2. Introduction It illustrates important driving events of the user. – Enable interactive search of video segments – Help the user to analyze his/her driving habits and patterns The objective is to construct a system that automatically annotates and summarizes videos.

3. Framework

4. geometry component(1/2) visual odometry [9] – To measure the self-motion of the camera.

5. geometry component(2/2) visual odometry – Feature (SIFT) detection for each frame – Matching is performed using KLT optical flows Estimating a ground plane using regular patterns on the ground (e.g. lane and crosswalk) – It enables global localization of other objects on it.

6. Detection component(1/3)

7. Detection component(2/3) Detect pedestrians – Adopt histogram of oriented gradients (HOG) features [3] and apply a sliding windows method – Filtering out windows with little vertical edges

8. Detection component(3/3) Vehicle detection – Apply the Viola and Jones’ method [15] to detect rear-view of appearing vehicles [15] P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In CVPR, 2001. Rectangle features

9. Tracking component A single hypothesis for each object Relies on color appearance model of the object – Each object hypothesis is computed using its position, size, and color histogram

10. Event analysis component The role is to label all ongoing events of the vehicle given a continuous video sequence. – They are recognized by hierarchically analyzing the relationships among the detected sub-events. – Spatio-Temporal Relationship Decision Tree.

11. Decision Trees Rules for classifying data using attributes. The tree consists of decision nodes and leaf nodes. –A decision node has two (or more branches), each representing values for the attribute tested. –A leaf node attribute produces a homogeneous result (all in one class), which does not require additional classification testing. intermediate node

12. Decision Tree Example overcast highnormal false true sunny rain No Yes Outlook Humidity Windy feature event result

13. Entropy Entropy = -1*(0.5log 2 (0.5) + 0.5log 2 (0.5)) = +1 Entropy = -1*(0.1log 2 (0.1) + 0.9log 2 (0.9)) = 0.47 Entropy: a formula to calculate the homogeneity of a sample. Maximizes the gain E(Current set) – E(All child sets)

14. Spatio-Temporal Relationship Decision Tree elementary sub-events car passing another car passed by another car is at front of another car at behind of another cars side-by-side accelerating decelerating vehicle stopped pedestrian in front Describing a condition of a particular sub- event (e.g. its duration greater than a certain threshold) Binary decision tree

15. Spatio-Temporal Relationship Decision Tree The system recognizes the sub-events using four types of features. – Extracted from local 3-D XYT trajectories. Time intervals of all occurring sub-events are recognized, and are provided to the system for the further analysis. – Describing a condition of a particular sub-event – A relationship between two sub-events orientationvelocityaccelerationrelative XY coordinate

16. Experiments Dataset of driving events The dataset is segmented into 52 scenes, where each of them contains 0 to 3 events. long stopping overtake overtaken sudden acceleration sudden stop - pedestrian sudden stop - vehicle