1. LUCAS KANADE FEATURE TRACKER a pyramidal implementation
di Stefano Pacifico – A.A. 2005/2006

2. Introduction
Feature tracking goal: Given I,J grayscaled 2D images, and a point u on I, find v = u+d on J such as u and v are similar.
Optical flow evaluation to estimate motion field (Hyp: Lambertian surfaces, infinity ranged light sources, no photometric distortion)
Feature based methods vs. Direct methods
Stefano Pacifico /2006

3. A first approach…
B.K.P. Horn and B.G. Schunck. Determining optical flow. AI Memo 572. Massachusetts Institue of Technology, 1980.
This article is the first proposal of an iterative method to compute optical flow.
Dense motion vector field, weak to noise.
Stefano Pacifico /2006

4. Optical flow constraint
Assumptions:
Pixel luminosity is the same in both images for each pair of pixels.
Motion perpendicular to local gradient direction is not discernible.
Uniform motion for near pixlel clusters (to avoid aperture problem).
Little displacemente of pixels between image I and J
Using a window of 5x5 size -> 25 equations per pixel; only u and v as unknowns.
Stefano Pacifico /2006

5. Lucas Kanade Overdetermined equation system
Equivalent to solving with least squares method
Iterate with Newton – Raphson method
A’A must be invertible.
L1 and L2 eigenvalues of A’A, L1/L2 must be not too big.
L1 and L2 not too little.
Stefano Pacifico /2006

6. Pyramidal Lucas Kanade
build pyramid representations of I and J ILm, JLm
Initialization of pyramidal guess gLm
for L = Lm to 0
uL = location of u on ILm = u/2^L
Ix = derivative of IL wrt x
Iy = derivative of IL wrt y
G = spatial gradient matrix
init iterative L-K (guess V0)
for k = 1 to K or until optical flow < treshold
Vk = iterative L-K loop estimation
end for
final optical flow at level L dL = Vk
calculate guess for next level gLm-1
final optical flow vector V = g0 + d0
return v = u +d
Relaxes iterative L-K assumption of little displacement
Algorithm complexity O(n^2*N+n^3)
n= #warp parmeters
N = integration window size
See Baker, Matthews papers.
Stefano Pacifico /2006

7. Pros, cons and alternatives
Problem and its actual solution are well studied; KLT widely used with almost optimal performances (see Barron et al. paper).
Bare KLT less reliable than affine tracker, but quite faster. Possible integration with compositional algorithm (see Baker,Matthews papers)
Feature trackers alone don’t satisfy the needs of today AI applications. KLT is integrated into more complex architectures (e.g. RANSAC, ML)
Alternatives: Mean shift tracking and Histogram tracking (see Bradski paper), Kalman Filter, Block Matching, Warping Theory (see Brax paper).
Stefano Pacifico /2006

8. Software and demo Stanford KLT implementation OpenCV implementation
Pyramidal implementation
Affine model feature checking
Can track features in non-consecutive frames
OpenCV implementation
Iterative implementation
Affine transformation estimation
Both have means to detect features, and to declare them lost (see also Shi, Tomasi paper).
Stefano Pacifico /2006

9. Bibliography
B.K.P. Horn and B.G.: Schunck. Determining optical flow. AI Memo 572. Massachusetts Institue of Technology, 1980.
Lucas, B. D.; Kanade, T.: An Iterative Image Registration Technique with an Application to Stereo Vision : Proc. of 7th International Joint Conference on Artificial Intelligence, (1981), pp
Shi, J.; Tomasi, C.: Good Features to Track. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, (1994), pp
Barron, J. L.; Fleet, D. J.: Beauchemin, S. S.; Burkitt, T. A.: Performance of optical flow techniques. In: International Journal of Computer Vision 12, Vol. 1 ( ), pp
Bouguet, J.-Y.: Pyramidal Implementation of the Lucas Kanade Feature Tracker. Part of OpenCV Documentation.
Stefano Pacifico /2006

10. Bibliography(2)
Bradski , G. R.: Computer vision face tracking for use in a perceptual user interface. Intel Technology Journal, 2nd Quarter, 1998.
Baker, S. ; Matthews, I.: Lucas-Kanade 20 years on: A unifying framework: Part 1. Technical Report CMU-RI-TR-02-16, CMU Robotics Institute, (2002).
U. Knauer, T. Dammeier, and B. Meffert.:The Structure Of Road Traffic Scenes As Revealed By Unsupervised Analysis Of The Time Averaged Optical Flow. 17th International Conference on the Applications of Computer Science and Mathematics in Architecture and Civil Engineering
Brox T., Bruhn, A. Papenberg, N. Weickert J.: High Accuracy Optical Flow Estimation Based on a Theory for Warping In Proc. 8th European Conference on Computer Vision, 2004
F. Pirri – Lecture notes A.A
Stefano Pacifico /2006