LUCAS KANADE FEATURE TRACKER a pyramidal implementation di Stefano Pacifico – A.A. 2005/2006

Introduction Feature tracking goal: Given I,J grayscaled 2D images, and a point u on I, find v = u+d on J such as u and v are similar. Optical flow evaluation to estimate motion field (Hyp: Lambertian surfaces, infinity ranged light sources, no photometric distortion) Feature based methods vs. Direct methods Stefano Pacifico - 2005/2006

A first approach… B.K.P. Horn and B.G. Schunck. Determining optical flow. AI Memo 572. Massachusetts Institue of Technology, 1980. This article is the first proposal of an iterative method to compute optical flow. Dense motion vector field, weak to noise. Stefano Pacifico - 2005/2006

Optical flow constraint Assumptions: Pixel luminosity is the same in both images for each pair of pixels. Motion perpendicular to local gradient direction is not discernible. Uniform motion for near pixlel clusters (to avoid aperture problem). Little displacemente of pixels between image I and J Using a window of 5x5 size -> 25 equations per pixel; only u and v as unknowns. Stefano Pacifico - 2005/2006

Lucas Kanade Overdetermined equation system Equivalent to solving with least squares method Iterate with Newton – Raphson method A’A must be invertible. L1 and L2 eigenvalues of A’A, L1/L2 must be not too big. L1 and L2 not too little. Stefano Pacifico - 2005/2006

Pyramidal Lucas Kanade build pyramid representations of I and J ILm, JLm Initialization of pyramidal guess gLm for L = Lm to 0 uL = location of u on ILm = u/2^L Ix = derivative of IL wrt x Iy = derivative of IL wrt y G = spatial gradient matrix init iterative L-K (guess V0) for k = 1 to K or until optical flow < treshold Vk = iterative L-K loop estimation end for final optical flow at level L dL = Vk calculate guess for next level gLm-1 final optical flow vector V = g0 + d0 return v = u +d Relaxes iterative L-K assumption of little displacement Algorithm complexity O(n^2*N+n^3) n= #warp parmeters N = integration window size See Baker, Matthews papers. Stefano Pacifico - 2005/2006

Pros, cons and alternatives Problem and its actual solution are well studied; KLT widely used with almost optimal performances (see Barron et al. paper). Bare KLT less reliable than affine tracker, but quite faster. Possible integration with compositional algorithm (see Baker,Matthews papers) Feature trackers alone don’t satisfy the needs of today AI applications. KLT is integrated into more complex architectures (e.g. RANSAC, ML) Alternatives: Mean shift tracking and Histogram tracking (see Bradski paper), Kalman Filter, Block Matching, Warping Theory (see Brax paper). Stefano Pacifico - 2005/2006

Software and demo Stanford KLT implementation OpenCV implementation Pyramidal implementation Affine model feature checking Can track features in non-consecutive frames http://www.ces.clemson.edu/~stb/klt/ OpenCV implementation Iterative implementation Affine transformation estimation http://Opencv.sourceforge.net Both have means to detect features, and to declare them lost (see also Shi, Tomasi paper). Stefano Pacifico - 2005/2006

Bibliography B.K.P. Horn and B.G.: Schunck. Determining optical flow. AI Memo 572. Massachusetts Institue of Technology, 1980. Lucas, B. D.; Kanade, T.: An Iterative Image Registration Technique with an Application to Stereo Vision : Proc. of 7th International Joint Conference on Artificial Intelligence, (1981), pp. 674-679. Shi, J.; Tomasi, C.: Good Features to Track. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, (1994), pp. 593-600. Barron, J. L.; Fleet, D. J.: Beauchemin, S. S.; Burkitt, T. A.: Performance of optical flow techniques. In: International Journal of Computer Vision 12, Vol. 1 (1992-02), pp. 43-77. Bouguet, J.-Y.: Pyramidal Implementation of the Lucas Kanade Feature Tracker. Part of OpenCV Documentation. Stefano Pacifico - 2005/2006

Bibliography(2) Bradski , G. R.: Computer vision face tracking for use in a perceptual user interface. Intel Technology Journal, 2nd Quarter, 1998. Baker, S. ; Matthews, I.: Lucas-Kanade 20 years on: A unifying framework: Part 1. Technical Report CMU-RI-TR-02-16, CMU Robotics Institute, (2002). U. Knauer, T. Dammeier, and B. Meffert.:The Structure Of Road Traffic Scenes As Revealed By Unsupervised Analysis Of The Time Averaged Optical Flow. 17th International Conference on the Applications of Computer Science and Mathematics in Architecture and Civil Engineering Brox T., Bruhn, A. Papenberg, N. Weickert J.: High Accuracy Optical Flow Estimation Based on a Theory for Warping In Proc. 8th European Conference on Computer Vision, 2004 F. Pirri – Lecture notes A.A. 2005-2006 Stefano Pacifico - 2005/2006