CSSE463: Image Recognition Day 30 This week Today: motion vectors and tracking Friday: Project workday. First status report due. Please email by 11 am a brief summary of: what you did what you didn’t and the technical challenges (lack of time isn’t an excuse) what you will do next week. Questions?

What is image flow? Notice that we can take partial derivatives with respect to x, y, and time.

Image flow equations Goal: to find where each pixel in frame t moves in frame t+Dt E.g. for 2 adjacent frames, Dt = 1 That is, Dx, Dy are unknown Assume: Illumination of object doesn’t change Distances of object from camera or lighting don’t change Each small intensity neighborhood can be observed in consecutive frames: f(x,y,t)f(x+Dx, y+Dy, t+Dt) for some Dx, Dy (the correct motion vector). Compute a Taylor-series expansion around a point in (x,y,t) coordinates. Gives edge gradient and temporal gradient Solve for (Dx, Dy) See answers to first quiz question now

Limitations Assumptions don’t always hold in real-world images. Doesn’t give a unique solution for flow Sometimes motion is ambiguous Look at last question on last quiz. “Live demo” http://www.cs.mcgill.ca/~chundt/coherence/

Aperture problem When we only see a small part of the image, motion can be ambiguous Lesson: By constraining ourselves to a small neighborhood, we lose the “big picture” Solutions: Assume pixels belong to same object and have same motion Propagate constraints between pixels Can still take many frames to converge Doesn‘t work at occlusions (hidden parts of object) Sensitive to outliers Only track motion of “interesting points” Q1

Interest point tracking Idea: corners and other easily-identifiable points can be tracked, and their motion is unambiguous. Technique for calculating a set of interest points: Loop over image: Calculate an interest score for a small neighborhood around each pixel If the score is above threshold, then add it to a point set to be returned. Q2

Interest scores One solution: (3x3 shown, but would use bigger window) Take minimum variance of those found in 4 directions: (3x3 shown, but would use bigger window) Another related texture-based operator is based on directions with significant edge gradient. Could also detect corners using Kirsch operators (similar to other edge operators).

Tracking interest points Search for them directly using a template. Choose one with highest correlation Just need to search in small neighborhood if frame rate is high enough Shapiro’s Alg. 9.3

Review: Correlation Dot product with template at each point. Normalize each to avoid false matches with bright regions

Matlab libraries exist to extract frames (still images) from video for processing. Demo would go here of simple tracking app

Computing trajectories Problem: what if unlabeled moving points belong to multiple trajectories? Idea: want to maximize smoothness, defined as change in direction and change of speed. Given two points pt, pt+1, thought to be related, calculate vt = pt+1 – pt. Then: direction Average speed compared to geom. mean

Computing trajectories Then the “Greedy-Exchange” algorithm is used to choose which points belong to which trajectory. Operates by creating trajectories and exchanging points if it would increase total smoothness. Local optimum.

Another approach The Kalman filter is a probabilistic model that combines noisy measurements with the expected trajectory of the object. It works even with occlusion. See http://www.cs.unc.edu/~welch/kalman/ for a start.