Pyramidal Implementation of Lucas Kanade Feature Tracker Jia Huang Xiaoyan Liu Han Xin Yizhen Tan

Abstract Introduction Tracking algorithm Lucas-Kanade algorithm Iterative implementation Tracking features analysis Feature lost Feature selection

Objective  For a given point u in image A, find its corresponding location v = u + d in image B. Image A Image B d

Residual function and Window size To find the location  Minimize residual function: : Integration window size Small integration windowHigher accuracy Larger integration windowHigher robustness Nature tradeoff:

Pyramid Implementation of LK algorithm  Calculate a set of pyramid representations of original image  Apply traditional tracking algorithm for each level  Results of current iteration is propagated to next iteration  Key point: the same window size is used for each level Top ViewSide View

Lucas-Kanade algorithm(1)  At the level L, we define images A and B:

Lucas-Kanade algorithm(2)  At the optimum, the first derivative of  After first order Taylor expansion  Components in the equation above

Lucas-Kanade algorithm(3)  Two derivative images are expressed:  With these notation, we can get:  The optimum optical flow vector is

 Pyramidal diagram  Inner loop: K-level  K initialized to 1, assume that the previous computations from iterations 1,2,...,k-1 provide an initial guess  The new translated image according to Iterative scheme of LK algorithm(1)

Iterative scheme of LK algorithm(2)  The goal: to compute the residual pixel motion vector, that minimizes the error function  Image mismatch vector, where the image difference delta I k defined as:  New pixel displacement guess is computed for the next iteration step k+1:

Iterative scheme of LK algorithm(3)  On average, 5 iterations are enough  At the 1st iteration (k=1), the initial guess is set to zero  The final solution for the optical flow vector is  Outer loop: L-level  The vector d is propagated to the next level L-1 and overall procedure is repeated L-1, L- 2, …, 0

Declaring a Feature Lost Several cases of lost feature the point falls outside of the image image patch around the tracked point varies too much between image A and image B too large displacement How to solve it combine a traditional tracking approach with an affine image matching

Feature Lost Example(1) Image A Image B

Feature Lost Example(2) Image A Image B

Feature Selection Intuitive To select the point u on image A good to track. Process steps: Compute the G matrix and λ m Call λ max the maximum value of λ m Retain the pixels that have a λ m value larger than a percentage of λ max Retain the local max. pixels Keep the subset of those pixels so that the minimum distance between pixels is larger than a threshold

Example of LK Feature Tracking Image A Image B

More Examples Image B Image A

Summary Lucas-Kanade Feature Tracker is one of the most popular versions of two-frame differential methods for motion estimation Iterative implementation of the Lucas- Kanade optical flow computation provides sufficient local tracking accuracy.

Thanks for your attention Any question?