1. The Brightness Constraint
Brightness Constancy Equation:
Linearizing (assuming small (u,v)):
Where: ) , ( y x J I t - =
Each pixel provides 1 equation in 2 unknowns (u,v).
Insufficient info.
Another constraint: Global Motion Model Constraint

2. Global Motion Models 2D Models: Affine Quadratic
Homography (Planar projective transform)
3D Models:
Rotation, Translation, 1/Depth
Instantaneous camera motion models
Plane+Parallax

3. Example: Affine Motion
Substituting into the B.C. Equation:
Each pixel provides 1 linear constraint in 6 global unknowns
Least Square Minimization (over all pixels):
(minimum 6 pixels necessary)
Every pixel contributes  Confidence-weighted regression

4. Example: Affine Motion
Differentiating w.r.t. a1 , …, a6 and equating to zero
 6 linear equations in 6 unknowns:

5. Coarse-to-Fine Estimation
Parameter propagation:
Pyramid of image J
Pyramid of image I
image I
image J Jw
warp
refine +
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
==> small u and v ...
image J
image I

6. Other 2D Motion Models
Quadratic – instantaneous approximation to planar motion
Projective – exact planar motion
(Homography H)

7. Panoramic Mosaic Image
Alignment accuracy (between a pair of frames): error < 0.1 pixel
Original video clip
Generated Mosaic image

8. Video Removal
Original
Original
Outliers
Synthesized

9. Video Enhancement
ORIGINAL
ENHANCED

10. Direct Methods:
Methods for motion and/or shape estimation, which recover the unknown parameters
directly from measurable image quantities
at each pixel in the image.
Minimization step:
Direct methods:
Error measure based on dense measurable image quantities (Confidence-weighted regression; Exploits all available information)
Feature-based methods:
Error measure based on distances of a sparse set of distinct feature matches.

11. Benefits of Direct Methods
High subpixel accuracy.
Do not need distinct features.
Locking property.

12. Limitations Limited search range (up to ~10% of the image size).
Brightness constancy assumption.

13. Video Indexing and Editing

14. Ex#4: Image Alignment (2D Translation)
Differentiating w.r.t. a1 and a2 and equating to zero
 2 linear equations in 2 unknowns:

15. Camera induced motion =
The 2D/3D Dichotomy
Camera motion +
Scene structure
Independent motions
Camera induced motion = +
Independent motions =
Image motion =
2D techniques
3D techniques
Do not model “3D scenes”
Singularities in “2D scenes”

16. The Plane+Parallax Decomposition
Original Sequence
Plane-Stabilized Sequence
The residual parallax lies on a radial (epipolar) field:
epipole

17. Benefits of the P+P Decomposition
Eliminates effects of rotation
Eliminates changes in camera parameters / zoom
1. Reduces the search space:
Camera parameters: Need to estimate only epipole.
(gauge ambiguity: unknown scale of epipole)
Image displacements: Constrained to lie on radial lines
(1-D search problem)
A result of aligning an existing structure in the image.

18. Benefits of the P+P Decomposition
2. Scene-Centered Representation:
Translation or pure rotation ???
Focus on relevant portion of info
Remove global component which dilutes information !

19. Benefits of the P+P Decomposition
2. Scene-Centered Representation:
Shape = Fluctuations relative to a planar surface in the scene
STAB_RUG SEQ

20. Benefits of the P+P Decomposition
2. Scene-Centered Representation:
Shape = Fluctuations relative to a planar surface in the scene
Height vs. Depth (e.g., obstacle avoidance)
Appropriate units for shape
A compact representation
- fewer bits, progressive encoding
total distance [ ]
camera
center
scene
global (100)
component
local [-3..+3]
component

21. Benefits of the P+P Decomposition
3. Stratified 2D-3D Representation:
Start with 2D estimation (homography).
3D info builds on top of 2D info.
Avoids a-priori model selection.

22. Dense 3D Reconstruction (Plane+Parallax)
Epipolar geometry in this case reduces to estimating the epipoles. Everything else is captured by the homography.
Original sequence
Plane-aligned sequence
Recovered shape

23. Dense 3D Reconstruction (Plane+Parallax)
Original
sequence
Plane-aligned
sequence
Recovered shape

24. Dense 3D Reconstruction (Plane+Parallax)
Original sequence
Plane-aligned
sequence
Epipolar geometry in this case reduces to estimating the epipoles. Everything else is captured by the homography.
Recovered shape