1. 1 Structure from Motion with Unknown Correspondence (aka: How to combine EM, MCMC, Nonlinear LS!) Frank Dellaert, Steve Seitz, Chuck Thorpe, and Sebastian Thrun slides courtesy of Frank Dellaert, adapted by Steve Seitz

2. 2 Traditionally: 2 Problems 1.Correspondence 2.Compute 3D positions, camera motion

3. 3 The Correspondence Problem

4. 4 The Structure from Motion Problem Find the most likely structure and motion 

5. 5 j ik = Structure From Motion u ik mimi m i’ xjxj Image i Image i’ Non-linear Least-Squares (Rick Szeliski’s lecture)

6. 6 Question How to recover structure and motion with unknown correspondence? Aside: related subproblems –known correspondence, unknown structure, motion structure from motion –known motion, unknown correspondence, structure stereo –known structure, unknown correspondence, motion 3D registration, ICP

7. 7 Without Correspondence: Try Tracking Features Lucas-Kanade Tracker (figures from Tommasini et. al. 98)

8. 8 New Idea: try all correspondences Likelihood: P(  ; U, J) structure + motion image featurescorrespondence Marginalize over J: P(  ; U) =  J P(  ; U, J)

9. 9 Combinatorial Explosion 3 images, 4 features: 4! 3 =13,824 5 images, 30 features: 30! 5 =1.3131e+162 (number of stars:1e+20, atoms: 1e+79) Computing P(  ; U, J) is intractable !

10. 10 EM for Correspondence

11. 11 Expectation Maximization

12. 12 E-Step: Soft Correspondences View 1View 3View 2 P(J | U,  t )

13. 13 M-Step: SFM Compute expected SFM solution Problem: this requires solving an exponential number of SFM problems If we assume correspondences j ik are independent, we can simplify...

14. 14 Virtual Measurements View 2 v A = p 1A u 1 + p 2A u 2 + p 3A u 3 + p 4A u 4

15. 15 j ik M-Step: SFM v ij mimi m i’ xjxj Image i Image i’ one SFM problem on virtual measurements

16. 16 Structure from Motion without Correspondence via EM: In each image, calculate the n 2 “soft correspondences”

17. 17 Key Issue: Calculating Weights

18. 18 Calculating Weights Weights = Marginal Probabilities

19. 19 Mutual Exclusion

20. 20 Monte Carlo EM

21. 21 Monte Carlo Approximation

22. 22 Sampling Assignments using the Metropolis Algorithm (1953) P( ) = -------------- P( )

23. 23 Swap Proposals (inefficient) a b c d e f

24. 24 ‘Chain Flipping’ Proposals

25. 25 Efficient Sampling (Example)

26. 26 a b c d e f MCMC to deal with mutex

27. 27 Input Images Manual measurements Random order

28. 28 Results: Cube

29. 29 Soft Correspondences scene feature x j uiui EM iterations