LPP-HOG: A New Local Image Descriptor for Fast Human Detection Andy Qing Jun Wang and Ru Bo Zhang IEEE International Symposium on Knowledge Acquisition and Modeling Workshop, pp Dec. 2008, Wuhan

2 Intelligent Systems Lab. Problem setting Goal: design algorithm for human detection able to perform in real-time Proposed solution: -Use a combination of Histogram of Oriented Gradients (HOG) as a feature vector. - Decrease feature-space dimensionality using Locality Preserving Projection (LPP) - Use Support Vector Machine (SVM) algorithm in reduced feature space to train the classifier

3 Intelligent Systems Lab. HOG general scheme

4 Intelligent Systems Lab. Typical person detection scheme using SVM In practice, effect is very small (about 1%) while some computational time is required* *Navneet Dalal and Bill Triggs. Histograms of Oriented Gradients for Human Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, SanDiego, USA, June Vol. II, pp

5 Intelligent Systems Lab. Computing gradients Mask Type 1D centered 1D uncentered 1D cubic ‑ corrected 2x2 diagonal3x3 Sobel Operator[-1, 0, 1][-1, 1][1, -8, 0, 8, -1] Miss rate at 10 −4 FPPW 11%12.5%12%12.5%14%

6 Intelligent Systems Lab. Accumulate weight votes over spatial cells How many bins should be in histogram? Should we use oriented or non-oriented gradients? How to select weights? Should we use overlapped blocks or not? If yes, then how big should be the overlap? What block size should we use? z

7 Intelligent Systems Lab. Accumulate weight votes over spatial cells How many bins should be in histogram? Should we use oriented or non-oriented gradients? How to select weights? Should we use overlapped blocks or not? If yes, then how big should be the overlap? What block size should we use?

8 Intelligent Systems Lab. Accumulate weight votes over spatial cells How many bins should be in histogram? Should we use oriented or non-oriented gradients? How to select weights? Should we use overlapped blocks or not? If yes, then how big should be the overlap? What block size should we use?

9 Intelligent Systems Lab. Contrast normalization - L2-norm followed by clipping (limiting the maximum values of v to 0.2) and renormalising

10 Intelligent Systems Lab. Making feature vector Variants of HOG descriptors. (a) A rectangular HOG (R-HOG) descriptor with 3 × 3 blocks of cells. (b) Circular HOG (C-HOG) descriptor with the central cell divided into angular sectors as in shape contexts. (c) A C-HOG descriptor with a single central cell.

11 Intelligent Systems Lab. HOG feature vector for one block AngleMagnitude Binary voting Magnitude voting Feature vector extends while window moves

12 Intelligent Systems Lab. HOG example In each triplet: (1) the input image, (2) the corresponding R-HOG feature vector (only the dominant orientation of each cell is shown), (3) the dominant orientations selected by the SVM (obtained by multiplying the feature vector by the corresponding weights from the linear SVM).

13 Intelligent Systems Lab. Support Vector Machine (SVM)

14 Intelligent Systems Lab. Problem setting for SVM x1x1 x2x2 w T x + b = 0 w T x + b < 0 w T x + b > 0 A hyper-plane in the feature space (Unit-length) normal vector of the hyper-plane: n denotes +1 denotes -1

15 Intelligent Systems Lab. x1x1 x2x2 How would you classify these points using a linear discriminant function in order to minimize the error rate? denotes +1 denotes -1 Infinite number of answers! Which one is the best? Problem setting for SVM

16 Intelligent Systems Lab. Large Margin Linear Classifier We know that The margin width is: x1x1 x2x2 denotes +1 denotes -1 Margin w T x + b = 0 w T x + b = -1 w T x + b = 1 x+x+ x+x+ x-x- n Support Vectors

17 Intelligent Systems Lab. Large Margin Linear Classifier Formulation: x1x1 x2x2 denotes +1 denotes -1 Margin w T x + b = 0 w T x + b = -1 w T x + b = 1 x+x+ x+x+ x-x- n such that

18 Intelligent Systems Lab. Large Margin Linear Classifier Formulation: x1x1 x2x2 denotes +1 denotes -1 Margin w T x + b = 0 w T x + b = -1 w T x + b = 1 x+x+ x+x+ x-x- n such that

19 Intelligent Systems Lab. Large Margin Linear Classifier Formulation: x1x1 x2x2 denotes +1 denotes -1 Margin w T x + b = 0 w T x + b = -1 w T x + b = 1 x+x+ x+x+ x-x- n such that

20 Intelligent Systems Lab. Solving the Optimization Problem s.t. Quadratic programming with linear constraints s.t. Lagrangian Function

21 Intelligent Systems Lab. Solving the Optimization Problem s.t.

22 Intelligent Systems Lab. Solving the Optimization Problem s.t., and Lagrangian Dual Problem

23 Intelligent Systems Lab. Solving the Optimization Problem The solution has the form: From KKT condition, we know: Thus, only support vectors have x1x1 x2x2 w T x + b = 0 w T x + b = -1 w T x + b = 1 x+x+ x+x+ x-x- Support Vectors

24 Intelligent Systems Lab. Solving the Optimization Problem The linear discriminant function is: Notice it relies on a dot product between the test point x and the support v ectors x i Also keep in mind that solving the optimization problem involved computing the dot products x i T x j between all pairs of training points

25 Intelligent Systems Lab. Large Margin Linear Classifier What if data is not linear separable? (noisy data, outliers, etc.) Slack variables ξ i can be added to allow miss-classification of dif ficult or noisy data points x1x1 x2x2 denotes +1 denotes -1 w T x + b = 0 w T x + b = -1 w T x + b = 1

26 Intelligent Systems Lab. Large Margin Linear Classifier Formulation: such that Parameter C can be viewed as a way to control over-fitting.

27 Intelligent Systems Lab. Large Margin Linear Classifier Formulation: (Lagrangian Dual Problem) such that

28 Intelligent Systems Lab. Datasets that are linearly separable with noise work out great: 0 x 0 x x2x2 0 x But what are we going to do if the dataset is just too hard? How about … mapping data to a higher-dimensional space: Non-linear SVMs

29 Intelligent Systems Lab. General idea: the original input space can be mapped to some higher-dime nsional feature space where the training set is separable: Φ: x → φ(x) Non-linear SVMs: Feature Space

30 Intelligent Systems Lab. With this mapping, our discriminant function is now: No need to know this mapping explicitly, because we only use the dot product o f feature vectors in both the training and test. A kernel function is defined as a function that corresponds to a dot product of two feature vectors in some expanded feature space: Non-linear SVMs: The Kernel Trick

31 Intelligent Systems Lab. Linear kernel: Examples of commonly-used kernel functions: Polynomial kernel: Gaussian (Radial-Basis Function (RBF) ) kernel: Non-linear SVMs: The Kernel Trick

32 Intelligent Systems Lab. Nonlinear SVM: Optimization Formulation: (Lagrangian Dual Problem) such that The solution of the discriminant function is The optimization technique is the same.

33 Intelligent Systems Lab. Support Vector Machine: Algorithm 1. Choose a kernel function 2. Choose a value for C 3. Solve the quadratic programming problem (many algorithms and software packages available) 4. Construct the discriminant function from the support vectors

34 Intelligent Systems Lab. Summary: Support Vector Machine 1. Large Margin Classifier Better generalization ability & less over-fitting 2. The Kernel Trick Map data points to higher dimensional space in order to make them linearly separable. Since only dot product is used, we do not need to represent the mapping explicitly.

35 Intelligent Systems Lab. Back to the proposed paper

36 Intelligent Systems Lab. Proposed algorithm parameters - Bins in histogram: 8 - Cell size: 4x4 pixels - Block size: 2x2 cells (8x8 pixels) - Image size: 64x128 pixels (8x16 blocks) - Feature vector size: 2x2x8x8x16=4096

37 Intelligent Systems Lab. LPP Algorithm Main idea: find matrix which will project original data into a space with lower dimensionality while preserving similarity between data (data which are close to each other in original space should be close after projection)

38 Intelligent Systems Lab. LPP Algorithm Is it correct? Add constraints Can be represented as a generalized eigenvalue problem Is it correct? By selecting d smallest eigenvalues and corresponding eigenvectors dimensionality reduction is achieved

39 Intelligent Systems Lab. Solving different scale problem

40 Intelligent Systems Lab. Some results Dimension d Detection rate PCA-HOG features (labeled’ *’) vs LPP-HOG features (labeled ˅ ’) Detection example

41 Intelligent Systems Lab. Conclusions - Fast human detection algorithm based on HOG features is presented - no information about computational speed is given - Proposed method is similar to PCA-HOG - feature space dimensionality decreased using LPP - why do we need to make LPP instead of finding eigenvectors from original feature space? - some equations seems to be wrong - Reference papers are very few Navneet Dalal “Finding People in Images and Videos” PhD Thesis. Institut National Polytechnique de Grenoble / INRIA Grenoble, Grenoble, July Navneet Dalal and Bill Triggs, “Histograms of Oriented Gradients for Human Detection”. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, SanDiego, USA, June Vol. II, pp Paisitkriangkrai, S., Shen, C. and Zhang, J. “Performance evaluation of local features in human classification and detection”, IET Computer Vision, vol.2, issue 4, pp ,December 2008