Low Complexity Keypoint Recognition and Pose Estimation Vincent Lepetit

Real-Time 3D Object Detection Runs at 15 Hz

3 Keypoint Recognition Pre-processing Make the actual classification easier Nearest neighbor classification One class per keypoint: the set of the keypoint’s possible appearances under various perspective, lighting, noise... The general approach [Lowe, Matas, Mikolajczyk] is a particular case of classification: Search in the Database

4 Used at run-time to recognize the keypoints Training phase Classifier

5 A New Classifier: Ferns Joint Work with Mustafa Özuysal

6 Compromise: which is proportional to but complete representation of the joint distribution infeasible. Naive Bayesian ignores the correlation: We are looking for If patch can be represented by a set of image features { f i } :

Presentation on an Example

Ferns: Training The tests compare the intensities of two pixels around the keypoint: Invariant to light change by any raising function. Posterior probabilities:

Ferns: Training

Ferns: Training

Ferns: Training Results

Ferns: Recognition

It Really Works

14 Ferns outperform Trees 500 classes. No orientation or perspective correction. FERNS TREES Number of structures Recognition rate Ferns responses are combined multiplicatively (Naive Bayesian rule) Trees responses are combined additively (average)

Optimized Locations versus Random Locations: We Can Use Random Tests Number of trees Recognition rate Information gain optimization Randomness Comparison of the recognition rates for 200 keypoints:

16 We Can Use Random Tests For a small number of classes we can try several tests, and retain the best one according to some criterion.

17 We Can Use Random Tests For a small number of classes we can try several tests, and retain the best one according to some criterion. When the number of classes is large any test does a decent job:

18 Another Graphical Interpretation

19 Another Graphical Interpretation

20 Another Graphical Interpretation

21 Another Graphical Interpretation

22 Another Graphical Interpretation

23 Another Graphical Interpretation

24 We Can Use Random Tests : Why It Is Interesting Building the ferns takes no time (except for the posterior probabilities estimation); Simplifies the classifier structure; Allows incremental learning.

25 Comparison with SIFT Recognition rate FERNS SIFT Frame Index Number of Inliers

26 Comparison with SIFT Computation time SIFT: 1 ms to compute the descriptor of a keypoint (without including convolution); FERNS: 13.5 micro-second to classify one keypoint into 200 classes.

27 1: for(int i = 0; i < H; i++) P[i ] = 0.; 2: for(int k = 0; k < M; k++) { 3: int index = 0, * d = D + k * 2 * S; 4: for(int j = 0; j < S; j++) { 5: index <<= 1; 6: if (*(K + d[0]) < *(K + d[1])) 7: index++; 8: d += 2; } 9: p = PF + k * shift2 + index * shift1; 10: for(int i = 0; i < H; i++) P[i] += p[i]; } Very simple to implement; No need for orientation nor perspective correction; (Almost) no parameters to tune; Very fast. Keypoint Recognition in Ten Lines of Code

28 Ferns Tuning The number of ferns, and The number of tests per ferns can be tuned to adapt to the hardware in terms of CPU power and memory size.

Feature Harvesting Estimate the posterior probabilities from a training video sequence:

Feature Harvesting Update Classifier Detect Object in Current Frame With the ferns, we can easily: - add a class; - remove a class; - add samples of a class to refine the classifier.  Incremental learning  No need to store image patches;  We can select the keypoints the classifier can recognize. Training examples Matches

Test Sequence

Handling Light Changes

35

Low Complexity Keypoint Recognition and Pose Estimation

37 EP n P: An Accurate Non-Iterative O(n) Solution to the P n P Problem Joint Work with Francesc Moreno-Noguer

38 The Perspective- n -Point (P n P) Problem How to take advantage of the internal parameters ? Solutions exist for the specific cases n = 3 [...], n = 4 [...], n = 5 [...], and the general case [...]. Rotation, Translation ? Internal parameters known 2D/3D correspondences known

39 A Stable Algorithm MEANMEDIAN Rotation Error (%) Number of points used to estimate pose LHM: Lu-Hager-Mjolsness, Fast and Globally Convergent Pose Estimation from Video Images. PAMI'00. (Alternatively optimize over Rotation and Translation); EPnP: Our method.

40 A Fast Algorithm Rotation Error (%) Computation Time (sec) - Logarithmic scale MEDIAN

41 General Approach Estimate the coordinates of the 3D points in the camera coordinate system. Rotation, Translation [Lu et al. PAMI00] Rotation, Translation [Lu et al. PAMI00]

42 Introducing Control Points The 3D points are expressed as a weighted sum of four control points.  12 unknowns: The coordinates of the control points in the camera coordinates system.

43 The Point Reprojections Give a Linear System For each correspondence i : Rewriting and Concatenating the Equations from all the Correspondences:

44 Mx = 0  M T Mx = 0  x belongs to the null space of M T M : with v i eigenvectors of matrix M T M associated to null eigenvalues. Computing M T M is the most costly operation — and linear in n, the number of correspondences. The Solution as Weighted Sum of Eigenvectors

45 The  i are our N new unknowns; N is the dimension of the null space of M T M ; Without noise: N = 1 (scale ambiguity). In practice: no zero eigenvalues, but several very small, and N ≥ 1 (depends on the 2D locations noise). We found that only the cases N = 1, 2, 3 and 4 must be considered. From 12 Unknowns to 1, 2, 3, or 4

46 How the Control Points Vary with the  i Reprojections in the Image Corresponding 3D points When varying the  i :

47 Imposing the Rigidity Constraint The distances between the control points must be preserved:  6 quadratic equations in the  i.

48 The Case N = 1  1 can easily be computed: Its absolute value is solution of a linear system: Its sign is chosen so that the handedness of the control points is preserved., and 6 quadratic equations:

49 We use the linearization technique. Gives 6 linear equations in  11 =  1 2,  12 =  1  2, and  22 =  2 2 : The Case N = 2, and 6 quadratic equations:

50 Same linearization technique. Gives 6 linear equations for 6 unknowns: The Case N = 3, and 6 quadratic equations:

51 Six quadratic equations in  1,  2,  3, and  4. The linearization introduces 10 products  ab =  a  b  Not enough equations anymore ! Õ Relinearization: The  ab are expressed as a linear combination of eigenvectors. The Case N = 4

52 Algorithm Summary 1.The control points coordinates are the (12) unknowns; 2.The 3D points should project on the given corresponding 2D locations: Linear system in the control points coordinates. 3.The control points coordinates can be expressed as a linear combination of the null eigenvectors of this linear system: The weights (the  i ) are the new unknowns (not more than 4). 4.Adding the rigidity constraints gives quadratic equations in the  i. 5.Solving for the  i depends on their number (linearization or relinearization).

53 Results

54 Thank you. Questions ?

55 Estimating from Samples the empirical probability constant value It is easy to prove that this expression for can be estimated as if we modelize as with the number of samples that verify and u a positive constant (in practice u = 1).

The Arborescence is not Needed when the Features are Taken at Random f1f1 f2f2 f3f3

57 Ferns outperform Trees Recognition rate Number of classes FERNS TREES

58 Linking the Two Approaches The Ferns consider while the Trees consider with

59 Linking the Two Approaches It can be proved the two methods are equivalent when the are small:

60 The Point Reprojections Give a Linear System For each correspondence i : Let's expand: Concatenating equations from all the correspondences: From point reprojection: