1. Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004

2. Stereo reconstruction problem: Input Input  Set of images of a scene I={I 1,…,I K }  Camera matrices P 1,…,P K Output Output  Surface model

3. Shape parametrisation Disparity-map parametrisation   MRF formulation – good optimisation techniques exist (Graph-cuts, Loopy BP)   MRF smoothness is viewpoint dependent   Disparity is unique per pixel – only functions represented

4. Shape parametrisation Volumetric parametrisation – e.g. Level- sets, Space carving etc.   Able to cope with non-functions   Convergence properties not well understood, Local minima   Memory intensive   For Space carving, no simple way to impose surface smoothness

5. Solution ? Cast volumetric methods in MRF framework Cast volumetric methods in MRF framework Key assumption: Approximate scene geometry given Key assumption: Approximate scene geometry given Benefits: Benefits:  General surfaces can be represented  Optimisation is tractable (MRF solvers)  Occlusions are approximately modelled  Smoothness is viewpoint independent

6. MRFs The labelling problem: The labelling problem:

7. MRFs A set of random variables h 1,…,h M A set of random variables h 1,…,h M A binary neighbourhood relation N defined on the variables A binary neighbourhood relation N defined on the variables Each can take a label out of a set H 1,…,H L Each can take a label out of a set H 1,…,H L C i (h i ) (Labelling cost) C i (h i ) (Labelling cost) C i,j (h i,h j ) for (i,j)  N (Compatibility cost) C i,j (h i,h j ) for (i,j)  N (Compatibility cost) -log P(h 1,…,h M ) =  C i (h i ) +  C i,j (h i,h j )

8. MRF inference Minimise  C i (h i ) +  C i,j (h i,h j ) Minimise  C i (h i ) +  C i,j (h i,h j ) Not in polynomial time in general case Not in polynomial time in general case Special cases (e.g. no loops or 2 label MRF) solved exactly Special cases (e.g. no loops or 2 label MRF) solved exactly General cases solved approximately via Graph-cuts or Loopy Belief Propagation. Approx. 10-15mins for MRF with 250,000 nodes. General cases solved approximately via Graph-cuts or Loopy Belief Propagation. Approx. 10-15mins for MRF with 250,000 nodes.

9. Relief Surfaces Approximate base surface Approximate base surface  Triangulated feature matches  Visual hull from silhouettes  Initialised by hand

10. Relief Surfaces labels :

11. Relief Surfaces labelling cost : Low cost High cost XiXi nini X i +h i n i C i (h i )=photoconsistency(X i +h i n i )

12. Relief Surfaces Compatibility cost : Low cost XiXi nini njnj XjXj X i +h i n i X j +h j n j

13. Relief Surfaces Neighbour cost : C i,j (h i, h j )= ||(X i +h i n i )-(X j +h j n j )|| XiXi nini X i +h i n i X j +h j n j High cost

14. Relief Surfaces Base surface is the occluding volume Base surface is the occluding volume If base surface ‘contains’ true surface (e.g. visual hull) then If base surface ‘contains’ true surface (e.g. visual hull) then  Points on the base surface X i are not visible by cameras they shouldn’t be [Kutulakos, Seitz 2000] Approximation: Approximation:  Visibility is propagated from X i to X i +h i n i

15. Loopy Belief Propagation Iterative message passing algorithm Iterative message passing algorithm m (t) i,j (h j ) is the message passed from i to j at time step t m (t) i,j (h j ) is the message passed from i to j at time step t It is a L-dimensional vector It is a L-dimensional vector Represents what node i ‘believes’ about the true state of node j. Represents what node i ‘believes’ about the true state of node j. min  C i (h i ) +  C i,j (h i,h j ) i j m i,j

16. Loopy Belief Propagation Message passing rule: Message passing rule: After convergence, optimal state is given by After convergence, optimal state is given by m (t+1) i,j (h j )= min { C ij (h i,h j ) +C i (h i ) +  m (t) k,i (h i ) } k  N(i) hihi h i * = min { C i (h i ) +  m (  ) k,i (h i ) } k  N(i) hihi i j m i,j

17. Loopy Belief Propagation O(L 2 ) to compute a message (L is number of allowable heights) O(L 2 ) to compute a message (L is number of allowable heights) Message passing schedule can be asynchronous which can accelerate convergence [Tappen & Freeman ICCV 03] Message passing schedule can be asynchronous which can accelerate convergence [Tappen & Freeman ICCV 03]

18. Iterative Scheme BP is memory intensive. BP is memory intensive. Can consider few possible labels at a time Can consider few possible labels at a time After convergence we ‘zoom in’ to heights close to the optimal After convergence we ‘zoom in’ to heights close to the optimal

19. Evaluation Artificial deformed sphere Artificial deformed sphere Textured with random patern Textured with random patern 20 images 20 images 40,000 sample points on sphere base surface 40,000 sample points on sphere base surface True surfaceTexture-mappedReconstruction

20. Evaluation Benchmark: 2-view, disparity based Loopy Belief Propagation [Sun et al ECCV02] Benchmark: 2-view, disparity based Loopy Belief Propagation [Sun et al ECCV02] BP run on 10 pairs of nearby views BP run on 10 pairs of nearby views Compare Disparity Maps given by Compare Disparity Maps given by  2-view BP  Relief surfaces  Ground truth

21. Evaluation 2-view BPRelief surface Ground truth 2-view BP Relief surf. MSE 1.466 pixels 0.499 pixels % of correct disparities 75.9%79.2%

22. Results Sarcophagus Sarcophagus

23. Results Sarcophagus Sarcophagus

24. Results Sarcophagus Sarcophagus

25. Results Building facade Building facade

26. Results Building facade Building facade

27. Results Base surfaceRelief surface with texture Stone carving Stone carving

28. Summary MRF methods can be extended in the volumetric domain MRF methods can be extended in the volumetric domain Advantages Advantages  General surfaces can be represented  Optimisation is tractable (MRF solvers)  Smoothness is viewpoint independent

29. Future work Photoconsistency beyond Lambertian surface models. (Optimise both height and surface normal fields) Photoconsistency beyond Lambertian surface models. (Optimise both height and surface normal fields) Change in topology Change in topology In cases where C mn (h m,h n )=|| h m -h n || or || h m - h n || 2 we can compute messages in O(L) time instead of O(L 2 ) (Felzenszwalb & Huttenlocher CVPR 04). In cases where C mn (h m,h n )=|| h m -h n || or || h m - h n || 2 we can compute messages in O(L) time instead of O(L 2 ) (Felzenszwalb & Huttenlocher CVPR 04).