1. CSSE463: Image Recognition Day 30
This week
Today: motion vectors and tracking
Friday: Project workday. First status report due. Please by 11 am a brief summary of:
what you did
what you didn’t and the technical challenges (lack of time isn’t an excuse)
what you will do next week.
Questions?

2. What is image flow?
Notice that we can take partial derivatives with respect to x, y, and time.

3. Image flow equations
Goal: to find where each pixel in frame t moves in frame t+Dt
E.g. for 2 adjacent frames, Dt = 1
That is, Dx, Dy are unknown
Assume:
Illumination of object doesn’t change
Distances of object from camera or lighting don’t change
Each small intensity neighborhood can be observed in consecutive frames: f(x,y,t)f(x+Dx, y+Dy, t+Dt) for some Dx, Dy (the correct motion vector).
Compute a Taylor-series expansion around a point in (x,y,t) coordinates.
Gives edge gradient and temporal gradient
Solve for (Dx, Dy)
See answers to first quiz question now

4. Limitations Assumptions don’t always hold in real-world images.
Doesn’t give a unique solution for flow
Sometimes motion is ambiguous
Look at last question on last quiz.
“Live demo”

5. Aperture problem
When we only see a small part of the image, motion can be ambiguous
Lesson: By constraining ourselves to a small neighborhood, we lose the “big picture”
Solutions:
Assume pixels belong to same object and have same motion
Propagate constraints between pixels
Can still take many frames to converge
Doesn‘t work at occlusions (hidden parts of object)
Sensitive to outliers
Only track motion of “interesting points” Q1

6. Interest point tracking
Idea: corners and other easily-identifiable points can be tracked, and their motion is unambiguous.
Technique for calculating a set of interest points:
Loop over image:
Calculate an interest score for a small neighborhood around each pixel
If the score is above threshold, then add it to a point set to be returned. Q2

7. Interest scores One solution: (3x3 shown, but would use bigger window)
Take minimum variance of those found in 4 directions:
(3x3 shown, but would use bigger window)
Another related texture-based operator is based on directions with significant edge gradient.
Could also detect corners using Kirsch operators (similar to other edge operators).

8. Tracking interest points
Search for them directly using a template.
Choose one with highest correlation
Just need to search in small neighborhood if frame rate is high enough
Shapiro’s Alg. 9.3

9. Review: Correlation Dot product with template at each point.
Normalize each to avoid false matches with bright regions

10. Matlab libraries exist to extract frames (still images) from video for processing.
Demo would go here of simple tracking app

11. Computing trajectories
Problem: what if unlabeled moving points belong to multiple trajectories?
Idea: want to maximize smoothness, defined as change in direction and change of speed.
Given two points pt, pt+1, thought to be related, calculate vt = pt+1 – pt. Then:
direction
Average speed compared to geom. mean

12. Computing trajectories
Then the “Greedy-Exchange” algorithm is used to choose which points belong to which trajectory.
Operates by creating trajectories and exchanging points if it would increase total smoothness.
Local optimum.

13. Another approach
The Kalman filter is a probabilistic model that combines noisy measurements with the expected trajectory of the object. It works even with occlusion.
See for a start.