1. Segmentation of Dynamic Scenes from Image Intensities
René Vidal Shankar Sastry
Department of EECS, UC Berkeley

2. Motivation and problem statement
A static scene: multiple 2D motion models
A dynamic scene: multiple 3D motion models
Given an image sequence, determine
Number of motion models (affine, Euclidean, etc.)
Motion model: affine (2D) or Euclidean (3D)
Segmentation: model to which each pixel belongs
3-D Motion Segmentation (Vidal-Soatto-Ma-Sastry, ECCV’02)
Generalization of the 8-point algorithm
Multibody fundamental matrix
This Talk: 2-D Motion Segmentation

3. Previous work
Probabilistic approaches (Jepson-Black’93, Ayer-Sawhney ’95, Darrel-Pentland’95, Weiss-Adelson’96, Weiss’97, Torr-Szeliski-Anandan ’99)
Generative model: data membership + motion model
Obtain motion models using Expectation Maximization
E-step: Given motion models, segment image data
M-step: Given data segmentation, estimate motion models
How to initialize iterative algorithms?
Spectral clustering: normalized cuts (Shi-Malik ‘98)
Similarity matrix based on motion profile
Local methods (Wang-Adelson ’94)
Estimate one model per pixel using a data in a window
Global methods (Irani-Peleg ‘92)
Dominant motion: fit one motion model to all pixels
Look for misaligned pixels & fit a new model to them

4. Our Approach to Motion Segmentation
Can we estimate ALL motion models simultaneously using ALL the image measurements?
When can we do so analytically? In closed form?
Is there a formula for the number of models?
We propose an algebraic geometric approach to affine motion segmentation
Number of models = degree of a polynomial
Groups ≈ roots of a polynomial
= polynomial factorization
In the absence of noise
Derive a constraint that is independent on the segmentation
There exists a unique solution which is closed form iff n<5
The exact solution can be computed using linear algebra
In the presence of noise
Derive a maximum likelihood algorithm for zero-mean Gaussian noise in which the E-step is algebraically eliminated

5. One-dimensional Segmentation
with a. ames
Number of models?

6. One-dimensional Segmentation
with a. ames
For n groups
Number of groups
Groups

7. Motion segmentation: the affine model
Constant brightness constraint
Affine motion model for the optical flow
Bilinear affine constraint
Mixture of n affine motion models

8. The multibody affine constraint
1-dimensional case
Multibody affine constraint
Veronese map
Affine segmentation case

9. The multibody affine matrix
Multibody affine constraint
Multibody affine matrix
Embedding
Lifting
Embedding

10. Affine motion segmentation algorithm
1-dimensional case
Affine segmentation case
Estimate all models: coefficients of a polynomial
Estimate number models: rank of a matrix
Chance a by c
Change \ell by \b and transpose
Np ng
Estimate individual models: roots/factors of the polynomial

11. Estimation of individual affine models
Can be reduced to scalar case!!
Factorization of affine motion models
Factorization of bilinear forms can be reduced to factorization of linear forms
Factorization of linear forms corresponds to segmentation of mixtures of subspaces
Generalized PCA: mixture of subspaces
Find roots of polynomial of degree n in one variable
Solve one linear systems in n variables

12. Optimal affine motion segmentation
Zero-mean Gaussian noise
Minimize distance error in image intensities subject to multibody affine constraints
Using Langrange optimization
After some algebra

13. Experimental results: flower sequence

14. Experimental results Two motions Camera panning to the right
Car translating to the right

15. Conclusions
There is an analytic solution to affine motion segmentation based on
Multibody affine constraint: segmentation independent
Polynomial factorization: linear algebra
Solution is closed form iff n<5
A similar technique also applies to
Eigenvector segmentation: from similarity matrices
Generalized PCA: mixtures of subspaces
3-D motion segmentation: of fundamental matrices
Future work
Reduce data complexity, sensitivity analysis, robustness