1. Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in 2 unknowns (u,v). Insufficient info. Another constraint: Global Motion Model Constraint

2. Global Motion Models 2D Models: Affine Quadratic Planar projective transform (Homography) 3D Models: Rotation, Translation, 1/Depth Instantaneous camera motion models Plane+Parallax

3. Example: Affine Motion Substituting into the B.C. Equation: Each pixel provides 1 linear constraint in 6 global unknowns (minimum 6 pixels necessary) Least Square Minimization (over all pixels):

4. image I image J JwJw warp refine + Pyramid of image JPyramid of image I image I image J Coarse-to-Fine Estimation u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels ==> small u and v...

5. Quadratic – instantaneous approximation to planar motion Other 2D Motion Models Projective – exact planar motion (Homography)

6. Panoramic Mosaic Image Original video clip Generated Mosaic image Alignment accuracy (between a pair of frames): error < 0.1 pixel

7. Original Outliers Original Synthesized Video Removal

8. ORIGINAL ENHANCED Video Enhancement

9. Direct Methods: Methods for motion and/or shape estimation, which recover the unknown parameters directly from measurable image quantities at each pixel in the image. Minimization step: Direct methods: Error measure based on dense measurable image quantities (Confidence-weighted regression; Exploits all available information) Feature-based methods: Error measure based on distances of a sparse set of distinct features.

10. Benefits of Direct Methods Subpixel accuracy. Does not need distinct features. Locking property.

11. Limitations Limited search range (up to 10% of the image size). Brightness constancy.

12. Handling Varying Brightness Preprocessing: Mean and contrast normalization. Laplacian pyramids. Measurable image quantities: brightness values correlation surfaces [Irani-Anandan:iccv98, Mandelbaum-et-al:iccv99] mutual information [Viola-et-al]

13. Video Indexing and Editing

14. The 2D/3D Dichotomy Image motion = Camera induced motion = + Independent motions = Camera motion + Scene structure + Independent motions 2D techniques 3D techniques Singularities in “2D scenes” Do not model “3D scenes”

15. The Plane+Parallax Decomposition Original SequencePlane-Stabilized Sequence The residual parallax lies on a radial (epipolar) field: epipole

16. Benefits of the P+P Decomposition Eliminates effects of rotation Eliminates changes in camera parameters / zoom Camera parameters: Need to estimate only epipole. (gauge ambiguity: unknown scale of epipole) Image displacements: Constrained to lie on radial lines (1-D search problem) A result of aligning an existing structure in the image. 1. Reduces the search space:

17. Remove global component which dilutes information ! Translation or pure rotation ??? Benefits of the P+P Decomposition 2. Scene-Centered Representation: Focus on relevant portion of info

18. Benefits of the P+P Decomposition 2. Scene-Centered Representation: Shape = Fluctuations relative to a planar surface in the scene STAB_RUG SEQ

19. - fewer bits, progressive encoding Benefits of the P+P Decomposition 2. Scene-Centered Representation: Shape = Fluctuations relative to a planar surface in the scene Height vs. Depth (e.g., obstacle avoidance) A compact representation global (100) component local [-3..+3] component total distance [97..103] camera center scene Appropriate units for shape

20. Start with 2D estimation (homography). 3D info builds on top of 2D info. 3. Stratified 2D-3D Representation: Avoids a-priori model selection. Benefits of the P+P Decomposition

21. Original sequencePlane-aligned sequenceRecovered shape Dense 3D Reconstruction (Plane+Parallax)

22. Original sequence Plane-aligned sequence Recovered shape

23. Results Original sequence Plane-aligned sequence Recovered shape

24. Brightness Constancy constraint Multi-Frame vs. 2-Frame Estimation The intersection of the two line constraints uniquely defines the displacement. 1. Eliminating Aperture Problem Epipolar line epipole p

25. other epipolar line Epipolar line Multi-Frame vs. 2-Frame Estimation The two line constraints are parallel ==> do NOT intersect 1. Eliminating Aperture Problem p another epipole Brightness Constancy constraint The other epipole resolves the ambiguity !

26. 3D Motion Models Local Parameter: Instantaneous camera motion: Global parameters: Residual Planar Parallax Motion Global parameters: Local Parameter: