1. Olga Zoidi, Anastasios Tefas, Member, IEEE Ioannis Pitas, Fellow, IEEE
Visual Object Tracking Based on Local Steering Kernels and Color Histograms IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY VOL. 23, NO.5, MAY 2013
Olga Zoidi, Anastasios Tefas, Member, IEEE Ioannis Pitas, Fellow, IEEE

2. Overview
Introduction
Proposed method
Experiment Result
Conclusion

3. Overview Introduction Proposed method Experiment Result Conclusion
Visual tracking
Object representation
Object position prediction
Proposed method
Experiment Result
Conclusion

4. Visual Tracking
Visual tracking is difficult to accomplish as some reason
Illumination conditions
Object may be nonrigid or articulated
Occluded
Rapid and complicated movements

5. Overview Introduction Proposed method Experiment Result Conclusion
Visual tracking
Object representation
Object position prediction
Proposed method
Experiment Result
Conclusion

6. Object Representation
Model-based
Appearance-based
Contour-based
Feature-based
Hybrid

7. Object Representation : Model-Based
Exploit a priori information about the object shape and create model [7].
Deal with the problem of object tracking under illumination variations, viewing angle, and partial occlusion.
Heavy cost.
[7]D.Roller, etc, “Model-based object tracking in monocular image sequences of road traffic scenes” Int. J. Comput. Vision, vol. 10, pp , Mar.1993.

8. Object Representation : Appearance-Based
Use the visual information for the object projection on the image plane, i.e., color, texture, and shape.
Deal with simple object transformation.
Sensitive to illumination changes.

9. Object Representation : Contour-Based
By employing shape matching or contour-evolution techniques [9]. Contour can be represented by active models, such as snakes or B-splines [10].
Deal with rigid and nonrigid objects.
Incorporate with occlusion detection and estimation techniques.
[9] A. Yilmaz, X.Li, and M. Shah, “Contour-based object tracking with occlusion handling in video acquired using mobile cameras”, IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 11, pp , Nov. 2004
[10] Y.Wang and O. Lee, “Active mesh – a feature seeking and tracking image sequence representation scheme”, IEEE Trans, Image Process, vol.3, no. 5, pp , Sep. 1994

10. Object Representation : Feature-Based
By tracking a set of feature points and these features are then grouped.
Problem is the correct distinction between the target object and background features.

11. Overview Introduction Proposed method Experiment Result Conclusion
Visual tracking
Object representation
Object position prediction
Proposed method
Experiment Result
Conclusion

12. Object Position Prediction
The position of the object in the following frame is usually predicted using a linear Kalman filter. [32]
[32] G. Welch and G. Bishop, “An introduction to the Kalman filter,” Univ. North Carolina, Chapel Hill, NC, Tech. Rep. TR95041, 2000.

13. Overview
Introduction
Proposed method
Experiment Result
Conclusion

14. Proposed Method Some tips of object tracking algorithm
Using color histogram (CH) to handle severe change in the object view.
Using decomposing target into fragments, which are tracked separately, to handle partial occlusion.
Using local steering kernel (LSK) object texture descriptor to represent the region of interest (ROI).
Proposed tracking approach is an appearance based method using both CHs and LSK descriptor.

15. Proposed Method
First, search image regions in video frame that have high color similarity to the object CH, and get candidate regions.
Next, LSK descriptors of both the target object and candidate search regions are extracted.
Discard the image regions with small CH similarity to the object CH, the new position of the object is selected as the image region, whose LSK representation has the maximum similarity to the one of the target object.
As tracking evolves, target object appearance changes and being a stack containing different instances. Stack is updated with the representation of the most recent detected object.

16. LSK Object Tracing Framework
Steps
Initialization of the object ROI in the first video frame. Initialization can be done manually.
Use CH information for color similarity search in the current search region. CH information can lead to background subtraction and reduction of the number of the candidate.
Representation of both the object and search region with LSK features.
Decision on the object ROI in the new frame, based on the similarities between candidate and: a) ROI in the previous frame, and b) top object instance in the stack.
Update the object instance in the stack.
Prediction of the object position in the following video frame and initialization of an object search region.

17. Overview Introduction Proposed method Experiment Result Conclusion
Color Similarity
Object Texture Description
Object Localization and Model Update
Search Region Extraction in The Next Frame
Experiment Result
Conclusion

18. Color Similarity
After object position prediction and search region selection, the search region of size R1*R2 is divided into candidate ROI which size is Q1*Q2.
Parameter d determines a uniform sampling of the candidate object ROIs every d pixels in the search region.
At frame t, the Bt% of the search region patches with the minimal histogram similarity to the object histogram are considered to belong to the background.
Cosine similarity : c h 1 , h 2 = cos θ < h 1 , h 2 > h 1 ∗ h 2
Normalized form S = c 2 (1− c 2 )

19. Color Similarity
Three color channels’ S of all patches comprise a matrix MCH.
The distribution of MCH takes values and sets a threshold in deciding whether the patch is a valid candidate ROI.
Finally, the binary matrix BCH, whose entry is set to 1 if entry of MCH is ≧threshold and 0, otherwise. BCH will be used in tracking in Object Location section.
Setting 𝑀 , 𝑀 𝑚𝑎𝑥 , and 𝑀 𝑚𝑖𝑛 as the mean, maximal, and minimal values of 𝑀 𝐶𝐻 entries, respectively.

20. Overview Introduction Proposed method Experiment Result Conclusion
Color Similarity
Object Texture Description
Object Localization and Model Update
Search Region Extraction in The Next Frame
Experiment Result
Conclusion

21. Object Texture Description
Introduction of LSK descriptors
LSKs are descriptors of the image salient features.
They were proven to be robust in small scale and orientation changes and deformations.
Result in successful tracking of slowly deformable objects.
LSKs descriptors are a nonlinear combination of weighted spatial distances between a pixel p of an image of size N1*N2 and its surrounding M*M pixels (pi). (M is equal to 3 pixels in this paper)
The distance K is measured using a weighted Euclidean distance, which uses as weights the covariance matrix Ci of the image gradients.

22. Object Texture Description
K i (p)= det⁡( C i ) 2𝜋 exp − p i −p T C i p i −p 2
In order to get Ci matrix in Ki(p), get gradient vectors gi and formed matrix GiM^2*2. Where G i = g x1 g y1 g x2 g y2 … … g x M 2 g y M
And Ci can be calculated via the singular value decomposition (SVD) of Gi.
𝐺 𝑖 = 𝑈 𝑖 𝑆 𝑖 𝑉 𝑖 𝑇 = 𝑈 𝑖 𝑆 1𝑖 𝑆 2𝑖 𝑣 1𝑖 𝑇 𝑣 2𝑖 𝑇 , and 𝐶 𝑖 = 𝛾 𝛼 1 2 𝑣 1𝑖 𝑣 1𝑖 𝑇 + 𝛼 2 2 𝑣 2𝑖 𝑣 2𝑖 𝑇
𝛼 1 = 𝑠 1𝑖 +1 𝑠 2𝑖 +1 , 𝛼 2 = 1 𝛼 1 ,𝛾= 𝑠 1𝑖 𝑠 2𝑖 + 10 −7 𝑀

23. Object Texture Description
For each neighboring pixel 𝑝 𝑖 , 𝑖=1,…, 𝑀 2 ,extract K(p) and normalize into 𝐾 𝑁 𝑝 = 𝐾(𝑝) 𝐾(𝑝) 1 , where ‧ 1 is the L1- norm.
Above concepts are applied. First converted ROI and search region from RGB to La*b* color space and the LSKs are computed for each channel separately through steps above.
The final representation of ROI is obtained by applying PCA [26].
Finally, the search region is divided into patches and the LSK similarity matrix, which will be used in next section, is estimated (like color similarity) by applying the cosine similarity measure.
[26] H. Seo and P. Milanfar, “Training-free, generic object detection using locally adaptive regression kernels,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 9, pp. 1688–1704, Sep

24. Overview Introduction Proposed method Experiment Result Conclusion
Color Similarity
Object Texture Description
Object Localization and Model Update
Search Region Extraction in The Next Frame
Experiment Result
Conclusion

25. Object Localization and Model Update
Object localization in the search region is performed by taking into account CH and LSK similarity of patch to the 1ROI in the previous frame and the 2object instance in the stack.
First, divide the search region into overlapping patches of size equal to the detected object. And for each patch, we extract CH and LSK features.
Then, for each patch, we construct three cosine similarity matrices
LSK similarity. Between this patch and the detected object in the previous frame.
LSK similarity. Between this patch and the last updated object instance.
CH similarity. Between this patch and the last updated object instance.

26. Object Localization and Model Update
The new ROI is decided with the final decision matrix, which is computed by M= 1−λ M LSK1 +λ M LSK2 ∗ B CH . (* denotes the element-wise matrix multiplication and λ usually takes the value 0.5)
The new candidate object position is at the patch with the maximal value maxi,j(Mij).
We compare Mij with previous frame’s maximal value Mij’. If the values drops under a threshold, it indicates a possible change in the object appearance.
Other 4 decision matrix of rotation and scaling are calculated. The final decision for the new object is the one which corresponds to the maximal value of five decision matrices.
The newly localized object is stored in a stack, which size is constant.

27. Overview Introduction Proposed method Experiment Result Conclusion
Color Similarity
Object Texture Description
Object Localization and Model Update
Search Region Extraction in The Next Frame
Experiment Result
Conclusion

28. Search Region Extraction in The Next Frame
The position of the object in the following frame is predicted using a linear Kalman filter.
The object motion state 𝑥 𝑡 = 𝑥,𝑦,𝑑𝑥,𝑑𝑦 𝑇 , and the new state is given by 𝑥 𝑡 =𝐴 𝑥 𝑡−1 + 𝑤 𝑡−1 . 𝑤 𝑡−1 denotes the process noise, with probability distribution.
After get 𝑥 𝑡 , we can then get 𝑧 𝑡 =𝐻 𝑥 𝑡 + 𝑣 𝑡 to compute the equation :
Where 𝑃 𝑡 is a covariance matrix with stochastic model. And this model is adjusted through equations below.
Among the equation above, 𝑥 𝑡 is the predicted position of a search region’s center.

29. Overview
Introduction
Proposed method
Experiment Result
Conclusion

30. Experiment Result
Quantitative evaluation comparison is performed through the frame detection accuracy (FDA) measure.
FDA calculates the overlap area between the ground truth object 𝐺 𝑖 and the detected object D at a given frame t.
𝐹𝐷𝐴 𝑡 = 1 𝑁 𝑡 𝑖=1 𝑁 𝑡 𝐺 𝑖 (𝑡)∩ 𝐷 𝑖 (𝑡) 𝐺 𝑖 (𝑡)∪ 𝐷 𝑖 (𝑡)
𝐴𝑇𝐴= 1 𝑁 𝑖=1 𝑁 𝐹𝐷𝐴(𝑡)
𝑂𝑇𝐴= 1 𝑁 𝑇 𝑖=1 𝑛 𝑁 𝑖 𝐴𝑇 𝐴 𝑖

31. Experiment Result
The performance of proposed tracker is compared with two other trackers, PF tracker and FT tracker.
Test case:

32. Experiment Result

33. Experiment Result
Test Case1 : test for variation of object scale

34. Experiment Result
Test Case2 : test for variation of object scale

35. Experiment Result
Test Case3 : test for variation of object rotation

36. Experiment Result
Test Case4 : test for variation of partial occlusion

37. Experiment Result
Test Case5 : test for variation of partial occlusion; the man walks behind the woman

38. Experiment Result
Test Case6 : test for strong change in illumination

39. Experiment Result
Test Case7 : test for human activity (orientation of glass)

40. Experiment Result
Test Case8 : test for human activity (hands are articulated objects)

41. Experiment Result
Test Case9 : test for human activity

42. Experiment Result
Test Case10 : test for face tracking

43. Experiment Result

44. Overview
Introduction
Proposed method
Experiment Result
Conclusion

45. Conclusion
The tracker extracted a representation of the target object based on LSK and CH at frame 𝑡−1 and tried to find its location in the frame 𝑡.
Proposed method is effective in object tracking under severe changes in appearance, affine transformations, and partial occlusion.
The method cannot handle the case of full occlusion. (The tracker continues tracking another object in the background)
Kalman filter cannot follow sudden changes in the object direction or speed. (Although a larger search region may solve the issue, but it would result in rapid decrease of speed)