Download presentation

Presentation is loading. Please wait.

Published byDuncan Huben Modified over 2 years ago

1
Motion Tracking Leow Wee Kheng CS4243 Computer Vision and Pattern Recognition CS4243Motion Tracking1

2
CS42432Motion Tracking Changes are everywhere!

3
Illumination change CS4243Motion Tracking3

4
Shape change CS4243Motion Tracking4

5
Object motion CS4243Motion Tracking5

6
Camera motion CS4243Motion Tracking6

7
Object & camera motion CS4243Motion Tracking7

8
Everything changes! CS4243Motion Tracking8

9
Motion analysis is tough in general! We focus on object / camera motion. CS4243Motion Tracking9

10
Change Detection Detects any change in two video frames. Straightforward method: Compute difference between corresponding pixels: : intensity / colour at (x, y) in frame t. If > threshold, has large difference. CS4243Motion Tracking10

11
Any difference? CS4243Motion Tracking11 No

12
Any difference? CS4243Motion Tracking12 Yes, illumination change

13
Any difference? CS4243Motion Tracking13 Yes, position change

14
Change Detection Can detect Illumination change Position change Illumination and position change But, cannot distinguish between them. Need to detect and measure position change. CS4243Motion Tracking14

15
Motion Tracking Two approaches Feature-based Intensity gradient-based CS4243Motion Tracking15

16
Feature-based Motion Tracking Look for distinct features that change positions. Eagle’s wing tips change positions. Tree tops don’t change positions. CS4243Motion Tracking16

17
Basic Ideas 1. Look for distinct features in current frame. 2. For each feature Search for matching feature within neighbourhood in next frame. Difference in positions displacement. Velocity = displacement / time difference. CS4243Motion Tracking17

18
Basic Ideas CS4243Motion Tracking18 feature area search area displacement

19
What feature to use? Harris corner Tomasi’s feature Feature descriptors: SIFT, SURF, GLOH, etc. Others CS4243Motion Tracking19

20
Summary Simple algorithm. Can be slow if search area is large. Can constrain search area with prior knowledge. CS4243Motion Tracking20

21
Gradient-based Motion Tracking Two basic assumptions Intensity changes smoothly with position. Intensity of object doesn’t change over time. Suppose an object is in motion. Change position (dx, dy) over time dt. Then, from 2 nd assumption: Apply Taylor’s series expansion: CS4243Motion Tracking21

22
Omit higher order terms and divide by dt Denote Then, u, v are unknown. 2 unknowns, 1 equation: can’t solve! CS4243Motion Tracking22

23
Any difference in motion? CS4243Motion Tracking23

24
Any difference in motion? CS4243Motion Tracking24

25
Any difference in motion? CS4243Motion Tracking25

26
Any difference in motion? CS4243Motion Tracking26

27
Any difference in motion? CS4243Motion Tracking27

28
Any difference in motion? CS4243Motion Tracking28

29
Aperture Problem Homogeneous region I x = I y = I t = 0. No change in local region. Cannot detect motion. CS4243Motion Tracking29

30
Aperture Problem Edge I x and I y are zero along edge Cannot measure motion tangential to edge I x and I y are non-zero normal to edge Can measure motion normal to edge So, cannot measure actual motion CS4243Motion Tracking30

31
Aperture Problem Corner I x and I y are non-zero in two perpendicular directions 2 unknowns, 2 equations Can measure actual motion CS4243Motion Tracking31

32
Lucas-Kanade Method Consider two consecutive image frames I and J: Object moves from x = (x, y) T to x + d. d = (u, v) T CS4243Motion Tracking32 I x J x + d x d

33
So, Or Due to noise, there’s an error at position x: Sum error over small window W at position x: CS4243Motion Tracking33 weight Similar to template matching

34
If E is small, patterns in I and J match well. So, find d that minimises E: Set E / d = 0, compute d that minimises E. First, expand I(x – d) by Taylor’s series expansion: Omit higher order terms: Write in matrix form: CS4243Motion Tracking34 Intensity gradient

35
Now, error E at position x is: Now, differentiate E with respect to d (exercise): Setting E / d = 0 gives CS4243Motion Tracking35 the only unknown Z b

36
So, we get 2 unknowns, 2 equations. Can solve for d = (u, v). What happen to aperture problem? Did it disappear? CS4243Motion Tracking36

37
Lucas-Kanade + Tomasi Lucas-Kanade algorithm is often used with Tomasi’s feature Apply Tomasi’s method to detect good features. Apply LK method to compute d for each pixel. Accept d only for good features. CS4243Motion Tracking37

38
Example Can you spot tracking errors? CS4243Motion Tracking38

39
Constraints Math of LK tracker assumes d is small. In implementation, W is also small. LK tracker is good only for small displacement. CS4243Motion Tracking39

40
How to handle large displacement? What if we scale down images? Displacements are smaller! CS4243Motion Tracking40

41
Image Pyramid CS4243Motion Tracking41 downsample Usually, smoothen image with Gaussian filter before scaling down. input image scaled image

42
LK Tracking with Image Pyramid CS4243Motion Tracking42

43
Construct image pyramids. Apply LK tracker to low-resolution images. Propagate results to higher-resolution images. Apply LK tracker to higher-resolution images. CS4243Motion Tracking43

44
Example Tracking results are more accurate. CS4243Motion Tracking44

45
Summary Efficient algorithm, no explicit search. Has aperture problem; track good features only LK tracker can’t track large displacement. Use LK + image pyramid for large displacement. CS4243Motion Tracking45

46
Software OpenCV supports LK and LK with pyramid. [Bir] offers LK with Tomasi’s features & pyramid. CS4243Motion Tracking46

47
Appendix Calculation of I x, I y, I t Use finite difference method Forward difference I x = I(x 1, y, t) I(x, y, t) I y = I(x, y 1, t) I(x, y, t) I t = I(x, y, t 1) I(x, y, t) Backward difference I x = I(x, y, t) I(x 1, y, t) I y = I(x, y, t) I(x, y 1, t) I t = I(x, y, t) I(x, y, t 1) CS4243Motion Tracking47

48
Further Readings Lucas-Kanade tracking with pyramid: [BK08] Chapter 10. Optical flow: [Sze10] Section 8.4. Hierarchical motion estimation (with image pyramid): [Sze10] Section 8.1.1. CS4243Motion Tracking48

49
References [Bir] S. Birchfield. KLT: An implementation of the Kanade-Lucas- Tomasi feature tracker. http://vision.stanford.edu/~birch/klt/. [BK08] Bradski and Kaehler. Learning OpenCV: Computer Vision with the OpenCV Library. O’Reilly, 2008. [LK81] B. D. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of 7th International Joint Conference on Artificial Intelligence, pages 674– 679, 1981. [ST94] J. Shi and C. Tomasi. Good features to track. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 593–600, 1994. [Sze10] R. Szeliski. Computer Vision: Algorithms and Applications. Springer, 2010. CS4243Motion Tracking49

50
References [TK91] C. Tomasi and T. Kanade. Detection and tracking of point features. Technical Report CMU-CS-91-132, School of Computer Science, Carnegie Mellon University, 1991. CS4243Motion Tracking50

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google