Computing motion between images The basics

Matching images We will deal with two similar tasks: A. Given a point in one image, find its matching point in the other image - feature matching/optical flow. B. Find the transformation mapping one image to the other - Image alignment/ Image Registration. Feature matching/ optical flow Alignment: Scale by 3/4

Why is it useful Camera stabilization Tracking moving objects Finding the camera motion and 3D reconstruction. Image processing by combining several images (mosaicing, super-resolution…) Compression. …..

Matching by sliding window A basic method to find matching points between images. (correspondence) a neighborhood of the point (Window) in image I1 is compared to several Windows in I2. The window with the best score wins. Various scores can be used.

Matching by sliding window The winning window The window in image 1 A sliding window in I2 I2 I2 I2

Matching Criteria - Difference Common matching criteria: SSD - Sum of Squared Differences k - the size of the window. p1 = (x1, y1) is the center of the window in I1. p2 = (x2, y2) is the center of the window in I2. SAD - Sum of Absolute Differences

Matching Criteria - Correlation Cross Correlation is similar to SSD, but can be implemented more efficiently: The window to search The image to search (I2) The maximum cross correlation score The cross correlation score

Handling illumination changes SSD, SAD and Cross Correlation assume constant brightness. On order to handle illumination changes, Normalized cross-correlation can be used. I1 I2

Normalized Cross Correlation When ordering the pixels in the windows in vectors v1, v2: The Normalized Cross Correlation is: Squared Vector norm Inner product

The aperture problem ? ? ?

The aperture problem (cont’) Easy to track in both directions. Hard to track vertically.

The aperture problem (cont’) T junction normal flow real motion

And what about smooth areas ? Use bigger windows: Less noise Smaller aperture problem Might include different motions Smooth the image ! ? ?

Using a Pyramid for optical flow Smoothing the images: we get an estimation of the motion in uniform regions. Handling large motions: Even if the motion in the original image is big, the motion in the small level is small. Once the solution was find in level k, there is only small motion to fix in the level k-1: The optical flow calculations become much faster ! Pyramid[1] Pyramid[2]

Auto-Correlation (and how does the SSD handle a combination of motions) The AC meaning: How similar does the image looks to it selves in different shifts. It has a strong connection with the aperture problem. The SSD should find the best motion – depending on the auto-correlation function.

Deformations The window matching assumes a pure image translation in small regions. Possible solutions for deformations: Invariant features Iterations of motion computations + warping

Limitations of Window Matching Accuracy: A pixel is always matched to integer location on the grid. The image motion is usually not integer. Neighborhood/Scene constraints: High level knowledge about the scene/camera may help in limiting the search, and reducing errors. (for example, the scene is planar)