Detecting Pedestrians Using Patterns of Motion and Appearance Paul Viola Microsoft Research Irfan Ullah Dept. of Info. and Comm. Engr. Myongji University Michael J. Jones, and Daniel Snow Mitsubishi Electric Research Laboratories Copyright © solarlits.com

Contents 1.Introduction 2.Background 3.System architecture 4.Objective 5.Rectangle features 6.Boosting algorithm 7.Training algorithm 8.Detection results 9.Conclusions

Pattern recognition approaches Face, automobile, and pedestrian detection Works well for face detection Introduction Automobile Face detection Pedestrian detection Training examples Detector Scanning Pattern of intensities

Researchers presumed that moving object is detected Recognize, categorize, or analyze the long-term pattern of motion Background Low resolution 9 x 15 pixels R. Cutler and L. S. Davis, 2000 Gavrila and Philomen (1999) Pedestriain detection in static images Detection rates: 75% False positive rate: 2 per image support vector machine False positive rate was higher in face detection Papageorgiou et al. (1998) Rectangle features and AdaBoost Paul Viola, Michael J. Jones, 2004

System architecture Input Image Rectangle filter Two-rectangl Three-triangle features Motion filters Integral image 1. Difference 2. Motion 3. direction of motion , U, D, L and R 1. f i 2. f j 3. f k 4. f m Final classifier Pedestrian detection AdaBoost Classifier from features Threshold filter Training Process

Pedestrian detection system Integrates image intensity information with motion information Detection style algorithm (using AdaBoost) Detectors based on motion information and detectors based on appearance information 4 frames/second with 20 x 15 pixels Representation of image motion Pedestrian detection system Under conditions (rain and snow) Full human figures Objective Example

Rectangle features difference between the sum of the pixels within two rectangular regions Two-rectangle feature Three-rectangle feature sum within two outside rectangles subtracted from the sum in a center rectangle Four-rectangle feature difference between diagonal pairs of rectangles Dark-Bright (Bright1+Bright2)-Dark

Integral image “Intermediate representation for the image” Integral image Original image above and to the left of x, y Cumulative row sum sum of the pixels within rectangle D Sum of pixels in A A+B A+CA+B+C+D 4+1-(2+3) integral image: double integral of the image first along rows and then along columns i is the image and r is the box Simard et al. (1999)

Rectangle filters on motion pair Two-rectangle filters Sum of the pixels within the lighter rectangles - Sum of pixels in the darker rectangles Three-rectangle filters (Sum of pixels in the darker rectangle)  2 to account for twice as many lighter pixels Detection of Motion Patterns Bright-dark

Motion information Optical flow 100s or 1000s of operations per pixel Block motion estimation This is not entirely compatible with multi-scale object 1.Differences between pairs of images in time 2.Motion: Regions where the sum of the absolute values of the differences is large 3.Direction of motion: Difference between shifted versions of the second image in time with the first image Detection of Motion Patterns

Filters r i () is a single box of rectangular sum within the detection window S is one of {U, L, R, D} Region moving in a given direction Measures closer to motion shear φ j is one of the rectangle filters Magnitude of motion in one of the motion images r k () is a single box rectangular sum within the detection window Appearance filter Integral image

Classifier Feature is a thresholded filter that outputs one of two votes Classifier is a thresholded sum of features t i ∈ R is a feature threshold f i is one of the motion or appearance filters Real-valued α and β are computed during AdaBoost learning filter threshold t i and classifier threshold θ

Detection at multiple scales Scaling training images during tanning process 20 × 15 training images Pyramids are computed Scale factor: 0.8 to generate each successive layer of the pyramid where X l refers to the l th level of the pyramid

“Select the features and to train the classifier” Combining a collection of weak classification functions to form a stronger classifier AdaBoost Week classifier f: feature θ: threshold P: polarity (direction of the inequality) x is a (24 × 24) pixel sub-window of an image “Generates final classifier” Depends on designed system

Boosting algorithm Example imagesInitialize weights Final strong classifier m and l are the number of negatives and positives Normalize weights Best weak classifier Define h t (x) where f t, p t, and θ t are the minimizers of (error)  t Update weights e i = 0 if x i is classified correctly, e i = 1 otherwise Correctly classified

Training process To select a subset of features and construct the classifier AdaBoost Learning round Appearance filters Motion direction filters Motion shear filters Motion magnitude filters Threshold α and β votes of each feature Lowest weighted error Cascade architecture Fewest features False positive Detection rate “classifiers are applied to every sub-window” Initial classifier eliminates a large number of negative examples with very little processing

Training process False positive rate of the cascade Detection rate Expected number of features K: number of classifiers fi : falsepositive rate of the i th classifier on the examples di : detection rate of the i th classifier on the examples pi is the positive rate of the i th classifier n i are the number of features in the i th classifier Optimization framework the number of classifier stages the number of features, ni, of each stage the threshold of each stage

Training algorithm for building a cascaded detector Selects f and d per layer Overall false positive rate F target Acceptable false positive rate Minimum acceptable detection rate while F i > F target Train classifier with n i features using AdaBoost Use P and N Evaluate current classifier Decrease threshold until detection rate evaluate detector on set of non-face images put any false detections into the set N P = set of positive examples N = set of negative examples F 0 = 1.0 D 0 = 1.0 i = 0

8 set of video sequences of street with pedestrians Each contain 2000 frames 1 frame of each sequence is used for training Other two sequences were used to test the detectors Examples 2250 positive and 2250 negative examples 20 × 15 pedestrian images Experiments 6 sequences used for training

Variance normalization is performed To reduce contrast Experiments Positive training examples 2250 positive exemples 2250 false positive Détection threshold

Training Dynamic pedestrian detector: 54,624 filters Static detector: 24,328 filters 20 × 15 pixel window Training the cascade Difference in motion Pedestrians in the center Stand out from background The first 5 filters learned for the static pedestrian detector First 5 filters learned for the dynamic pedestrian detector Legs Chest

Dynamic detector few false positive Detection results Dynamic detector Static detector Rain Static detector More false positive

Detection results At 80% detection rate: dynamic detector: 1/400,000 static detector: 1/15,000. At 80% detection rate: both detectors: 1/400,000 false positive every 2 frames for the 360×240 “Sequence 2 has some highly textured areas such as the tree and grass”

Detection style algorithm Combines motion and appearance information Low false positive rate low computation time 0.25 seconds to detect pedestrians in 360 × 240 pixel image With 2.8 GHz P4 processor 0.1 seconds: scanning the cascade over all positions and scale the image 0.15 seconds: creating the pyramids of difference images Applications human motion (running, jumping) Facial expression classification Lip reading Conclusions

Thanks ?