1. Computing motion between images
The basics

2. Matching images We will deal with two similar tasks:
A. Given a point in one image, find its matching point in the other image - feature matching/optical flow.
B. Find the transformation mapping one image to the other - Image alignment/ Image Registration.
Feature matching/
optical flow
Alignment: Scale by 3/4

3. Why is it useful Camera stabilization Tracking moving objects
Finding the camera motion and 3D reconstruction.
Image processing by combining several images (mosaicing, super-resolution…)
Compression.
…..

4. Matching by sliding window
A basic method to find matching points between images. (correspondence)
a neighborhood of the point (Window) in image I1 is compared to several Windows in I2.
The window with the best score wins.
Various scores can be used.

5. Matching by sliding window
The winning window
The window in image 1
A sliding window in I2 I2 I2 I2

6. Matching Criteria - Difference
Common matching criteria:
SSD - Sum of Squared Differences
k - the size of the window.
p1 = (x1, y1) is the center of the window in I1.
p2 = (x2, y2) is the center of the window in I2.
SAD - Sum of Absolute Differences

7. Matching Criteria - Correlation
Cross Correlation is similar to SSD, but can be implemented more efficiently:
The window to search
The image to search (I2)
The maximum cross correlation score
The cross correlation score

8. Handling illumination changes
SSD, SAD and Cross Correlation assume constant brightness.
On order to handle illumination changes, Normalized cross-correlation can be used. I1 I2

9. Normalized Cross Correlation
When ordering the pixels in the windows in vectors v1, v2:
The Normalized Cross Correlation is:
Squared
Vector norm
Inner product

10. The aperture problem ? ? ?

11. The aperture problem (cont’)
Easy to track
in both directions.
Hard to track
vertically.

12. The aperture problem (cont’)
T junction
normal
flow
real
motion

13. And what about smooth areas ?
Use bigger windows:
Less noise
Smaller aperture problem
Might include different motions
Smooth the image ! ? ?

14. Using a Pyramid for optical flow
Smoothing the images: we get an estimation of the motion in uniform regions.
Handling large motions: Even if the motion in the original image is big, the motion in the small level is small.
Once the solution was find in level k, there is only small motion to fix in the level k-1: The optical flow calculations become much faster !
Pyramid[1]
Pyramid[2]

15. Auto-Correlation (and how does the SSD handle a combination of motions)
The AC meaning: How similar does the image looks to it selves in different shifts.
It has a strong connection with the aperture problem.
The SSD should find the best motion – depending on the auto-correlation function.

16. Deformations
The window matching assumes a pure image translation in small regions.
Possible solutions for deformations:
Invariant features
Iterations of motion computations + warping

17. Limitations of Window Matching
Accuracy:
A pixel is always matched to integer location on the grid. The image motion is usually not integer.
Neighborhood/Scene constraints:
High level knowledge about the scene/camera may help in limiting the search, and reducing errors. (for example, the scene is planar)