Object Detection using Histograms of Oriented Gradients Navneet Dalal, Bill Triggs INRIA Rhône-Alpes Grenoble, France Introduce urself, bill and Matthijs Thanks audience Introduce the topic Thanks to Matthijs Douze for volunteering to help with the experiments 7 May, 2006 Pascal VOC 2006 Workshop ECCV 2006, Graz, Austria

Talk Outline Current approaches Overall architecture Histogram of oriented gradient Description of image encoding algorithm Multi-scale detection architecture Fusion of detections at multiple scales and locations Key findings on Pascal VOC 2006 Conclusions

Motivation Current Approaches Our Approach Dense feature sets based approaches Papageorgiou & Poggio, 2000; Viola & Jones, 2001 Template or image fragments based approaches Gavrila & Philomen, 1999; Vidal-Naquet & Ullman, 2003 Models based on key points Leibe et al, 2005; Fergus et al, 2003 Our Approach Focus on creating robust encoding of images Linear SVM as classifier on normalized image windows, is reliable & fast Moving window based detector with non-maximum suppression over scale space Focus on why i opted for my approach. Before working on modelling, it is important to have robust representation. We divide image into windows and check if each window is a person or not. How we use linear SVM. Advantages of linear SVM. Describe multi-scale approach Feature space and learning method Include CMU guys and Deva, Forsyth paper

Overall Architecture Learning Phase Detection Phase Create normalised training data set Scan image at all scales and locations Encode images into feature vectors Run classifier to obtain object/non-object decisions Learn binary classifier Fuse multiple detections in 3-D position & scale space Object/Non-object decision Object detections with bounding boxes

Descriptor Processing Chain Object/Non-object Linear SVM [ ..., ..., ..., ...] Collect HOGs over detection window Contrast normalize over overlapping spatial cells Weighted vote in spatial & orientation cells Let me start with an overview of the encoding scheme algorithm. Don’t say anything think of phrase that a cell would activate neuron. Why normalize… We want invariance to color… so gradient information is necessary and we through away color information The problem in detecting human is that appearance and pose changes lot. We want invariance to small movements. So spatial binning is required. David Lowe’s SIFT features are very good in describing people. But key here is because of different clothng, there are no interest points. Using the lessons learned from it, we do orientation binning. One can put a linear SVM at this stage. But I will show that it is a bad idea. We want contrast normalization. One may ask why we need overlapping. It is the same information being copied over, but key is with different normalization. Overlapping helps encode same information in multiple ways and brings invariance to small displacements of silhouttes. We call it HOG. Actual implementation differs bit. We perform steps 1—3 over the whole image. Compute gradients Gamma compression Image Window

HOG Descriptors Parameters Gradient scale R-HOG/SIFT Orientation bins HOG: Histogram of Oriented Gradients Parameters Gradient scale Orientation bins Block overlap area R-HOG/SIFT Cell Schemes RGB or Lab, Color/gray- space Block normalization L2-hys, or L1-sqrt, Block HOGs can have any shape, like Lowe’s SIFT or more psychological inspired 3D version of shape context of Belongie et al. The log-polar shape of CHOG are motivated from human fovea system – where there are more cells at center and resolution decreases at the peripheries. Even in RHOG, we find that putting a Gaussian weight on top of block help increase the perform. But HOG has lot of parameters… that is hard to tune in. Surprisingly, our experience shows we only need to vary some – even if we change the object class or our detection window size. We take inspiration from Biological… C-HOG Center bin

Fine grained features improve performance Lessons on HOGs No gradient smoothing, [1 0 -1] derivative mask Use gradient magnitude (no thresholding) Orientation voting into fine bins (20˚ wide bins) Spatial voting into coarser bins Strong local normalization Use overlapping blocks +1 -1 Fine grained features improve performance Point 4 and 1st are opposite of what we do in Haar wavelets. This provides significant performance gain. It may seem lots of parameters, but our experience on humans, faces, and other object classes suggest that only normalization and size of blocks/cells need to be tested. Out of this, the object size already provides a fair idea of size of the object.  Have 1-2 order lower false positives than other descriptors  Slower than integral images of Viola & Jones, 2001 N. Dalal and B. Triggs. Histograms of Oriented Gradients for Human Detection. CVPR, 2005

Multi-Scale Detection Clip Detection Score After dense multi-scale scan of detection window Map each detection to 3D [x,y,scale] space x y s (in log) Now we need to get rid of multiple detections. How can we do it in principled manner? Tell about hypothesis… This leads us to kernel density estimates… We create 3d log space.. Put a point dependent sigma pose the problem as mode detection of this weighted density estimates… Final detections Apply robust mode detection, like mean shift

Performance Evaluation Transformation functions Scale-space pyramid steps Hard clipping Sigmoid mapping Hard clipping is better than sigmoid mapping Fine scale transition is very important

Overall robust non-maximum suppression is important Effect of Smoothing Spatial smoothing proportional to window size performs best Relatively independent to smoothing across scales Detector’s normalized image window size Detector’s response at the given scale level Overall robust non-maximum suppression is important Classifier responses for a dense scan at a scale

Overall Performance Recall-precision on INRIA person database R/C-HOG have 1-2 order lower false positives than other descriptors

Descriptor Cues: Motorbikes Average gradients Weighted pos wts Weighted neg wts Dominant pos orientations Dominant neg orientations Input window Detection Examples

Key Descriptor Parameters Class Window Size Avg. Size # of Orient- ation Bins Orientat- ion Range Gamma Compre- ssion Normal- isation Method Person 64×128 Height 96 9 0˚-180˚ √RGB L2-Hys Car 104×56 Height 48 18 0˚-360˚ L1-Sqrt Bus 120×80 Height 64 Motorbike Width 112 Bicycle 104×64 Width 96 Cow 128×80 Width 56 Sheep 104×60 Height 56 Horse RGB Cat 96×56 Dog Mention that bar helps for persons, motorbikes, bicycles

Conclusions Contributions Future Work Robust feature encoding for object detection Gives good performance for variety of object classes Real time detection is possible Future Work Part based detector for handling partial occlusions Incorporate texture and color descriptors into the framework One single optimization phase based on AdaBoost to learn most relevant descriptors

Thank You

Descriptor Cues: Cars Detection Examples Average gradients Weighted pos wts Weighted neg wts Detection Examples

Effect of Parameters Gradient smoothing, σ Orientation bins, β Using simple smoothed gradients and many orientations helps! Gradient scale 3→ 0 ⇒ false positives drop by 10 times Orientation bins 45˚→ 20˚ ⇒ false positives drop by 10 times

... Continued Normalization method Block overlap Strong local normalization is essential Overlapping block increases performance, but descriptor size increases

Effect of Block and Cell Size 64 128 Trade off between need for local spatial invariance and need for finer spatial resolution

Descriptor Cues: Persons Input example Average gradients Weighted pos wts Weighted neg wts Outside-in weights Most important cues are head, shoulder, leg silhouettes Vertical gradients inside a person are counted as negative Overlapping blocks just outside the contour are most important Of course we want to know what is going behind the scenes… On left, we have an example window. Then average gradient…… In some separate tests, not shown here, we find that if we reduce the background information, the detector performance drops.