1. A Closed Form Solution to Direct Motion Segmentation
René Vidal and Dheeraj Singaraju
Center for Imaging Science
Johns Hopkins University

2. Direct 2-D motion segmentation problem
Given the image intensities of a dynamic scene containing multiple motions, determine
Number of motion models (translational, affine, etc.)
Motion model: translational, or affine
Segmentation: model to which each pixel belongs
Copyright © JHU Vision Lab

3. Prior work on 2-D motion segmentation
Probabilistic approaches (Jepson-Black’93, Ayer-Sawhney ’95, Darrel-Pentland’95, Weiss-Adelson’96, Weiss’97, Torr-Szeliski-Anandan ’99)
Generative model: mixture of 2-D motion models
Estimate model using Expectation Maximization
E-step: Given motion models, segment image
M-step: Given grouping, estimate motion models
Local methods (Wang-Adelson ’93)
Estimate one model per pixel using a data in a window
Cluster models with K-means
Aperture problem
Motion across boundaries
Global methods (Irani-Peleg ’92)
Dominant motion: fit one motion model to all pixels
Look for misaligned pixels & fit a new model to them
Motion competition (Cremers’02)
Piece-wise parametric model
Minimize Mumford-Shah cost functional
Copyright © JHU Vision Lab

4. Paper contributions
Propose a global algebraic solution to direct 2-D motion segmentation
Multibody Brightness Constancy Constraint (MBCC)
2-D translational motions and 2-D affine motions
Algebraic solution to direct motion segmentation
Fit MBCC linearly to image partial derivatives
Compute optical flow as the derivative of the MBCC
Compute motion model parameters from the cross products of the derivatives of the MBCC
Main features
Requires no feature tracking or correspondences
Fits a mixture of motion models globally
Can be used to initialize nonlinear/probabilistic methods
Needs improvement
Copyright © JHU Vision Lab

5. Brightness constancy constraint (BCC)
Linear motion model: 2-D translational
Bilinear motion model: 2-D affine
Copyright © JHU Vision Lab

6. Multibody brightness constancy constraint
Mixture of n 2-D motion models
Multibody brightness constancy constraint
Optical flow from derivative of MBCC
Copyright © JHU Vision Lab

7. Segmentation of 2-D translational motions
MBCC is homogeneous polynomial of degree n
Linear on embedded data!
Number of motions
Veronese map
Motion models
Copyright © JHU Vision Lab

8. Segmentation of 2-D affine motions
MBCC bi-homogeneous polynomials of degree n
Lifting
Embedding
Bilinear on embedded data!
Multibody motion matrix
Copyright © JHU Vision Lab

9. Segmentation of 2-D affine motions
Multibody motion can be computed linearly from
Number of motion models can be computed from the rank of embedded data matrix
Individual motion models can be computed from the derivatives of
Copyright © JHU Vision Lab

10. Segmentation of 2-D affine motions
If belongs to ith motion, partials of MBC at give linear combination of the rows of
As 3rd row of is , 1st and 2nd rows can be obtained from cross product of partials
Copyright © JHU Vision Lab

11. Calculation of number of motions
Copyright © JHU Vision Lab

12. Segmentation of 2-D translational motions
Copyright © JHU Vision Lab

13. Segmentation of 2-D translational motions
Good segmentation (71% of frames)
Bad segmentation (29% of frames)
Copyright © JHU Vision Lab

14. Segmentation of 2-D affine motions
Copyright © JHU Vision Lab

15. Segmentation of 2-D affine motions
Good segmentation (78% of frames)
Bad segmentation (22% of frames)
Copyright © JHU Vision Lab

16. Optical flow for 2-D affine motions
Copyright © JHU Vision Lab

17. Conclusions and open problems
Algebraic solution to direct motion segmentation
Multibody Brightness Constancy Constraint (MBCC)
Optical flow & affine models: cross product of MBCC derivatives
Main features
Requires no feature tracking or correspondences
Fits a mixture of motion models globally
Can be used to initialize nonlinear/probabilistic methods
Open problems
How to deal with outliers?
How to incorporate smoothing in space and time?
How to incorporate multiple resolutions?
How to segment motion models of different type?
Copyright © JHU Vision Lab

18. Thanks

19. Optical flow for 2-D translational motions
Copyright © JHU Vision Lab

20. Best segmentation results
Copyright © JHU Vision Lab

21. Worst segmentation results
Copyright © JHU Vision Lab

22. Best segmentation results
Copyright © JHU Vision Lab

23. Worst segmentation results
Copyright © JHU Vision Lab