Reconstructing Relief Surfaces George Vogiatzis, Philip Torr, Steven Seitz and Roberto Cipolla BMVC 2004

Stereo reconstruction problem: Input Input  Set of images of a scene I={I 1,…,I K }  Camera matrices P 1,…,P K Output Output  Surface model

Shape parametrisation Disparity-map parametrisation   MRF formulation – good optimisation techniques exist (Graph-cuts, Loopy BP)   MRF smoothness is viewpoint dependent   Disparity is unique per pixel – only functions represented

Shape parametrisation Volumetric parametrisation – e.g. Level- sets, Space carving etc.   Able to cope with non-functions   Convergence properties not well understood, Local minima   Memory intensive   For Space carving, no simple way to impose surface smoothness

Solution ? Cast volumetric methods in MRF framework Cast volumetric methods in MRF framework Key assumption: Approximate scene geometry given Key assumption: Approximate scene geometry given Benefits: Benefits:  General surfaces can be represented  Optimisation is tractable (MRF solvers)  Occlusions are approximately modelled  Smoothness is viewpoint independent

MRFs The labelling problem: The labelling problem:

MRFs A set of random variables h 1,…,h M A set of random variables h 1,…,h M A binary neighbourhood relation N defined on the variables A binary neighbourhood relation N defined on the variables Each can take a label out of a set H 1,…,H L Each can take a label out of a set H 1,…,H L C i (h i ) (Labelling cost) C i (h i ) (Labelling cost) C i,j (h i,h j ) for (i,j)  N (Compatibility cost) C i,j (h i,h j ) for (i,j)  N (Compatibility cost) -log P(h 1,…,h M ) =  C i (h i ) +  C i,j (h i,h j )

MRF inference Minimise  C i (h i ) +  C i,j (h i,h j ) Minimise  C i (h i ) +  C i,j (h i,h j ) Not in polynomial time in general case Not in polynomial time in general case Special cases (e.g. no loops or 2 label MRF) solved exactly Special cases (e.g. no loops or 2 label MRF) solved exactly General cases solved approximately via Graph-cuts or Loopy Belief Propagation. Approx mins for MRF with 250,000 nodes. General cases solved approximately via Graph-cuts or Loopy Belief Propagation. Approx mins for MRF with 250,000 nodes.

Relief Surfaces Approximate base surface Approximate base surface  Triangulated feature matches  Visual hull from silhouettes  Initialised by hand

Relief Surfaces labels :

Relief Surfaces labelling cost : Low cost High cost XiXi nini X i +h i n i C i (h i )=photoconsistency(X i +h i n i )

Relief Surfaces Compatibility cost : Low cost XiXi nini njnj XjXj X i +h i n i X j +h j n j

Relief Surfaces Neighbour cost : C i,j (h i, h j )= ||(X i +h i n i )-(X j +h j n j )|| XiXi nini X i +h i n i X j +h j n j High cost

Relief Surfaces Base surface is the occluding volume Base surface is the occluding volume If base surface ‘contains’ true surface (e.g. visual hull) then If base surface ‘contains’ true surface (e.g. visual hull) then  Points on the base surface X i are not visible by cameras they shouldn’t be [Kutulakos, Seitz 2000] Approximation: Approximation:  Visibility is propagated from X i to X i +h i n i

Loopy Belief Propagation Iterative message passing algorithm Iterative message passing algorithm m (t) i,j (h j ) is the message passed from i to j at time step t m (t) i,j (h j ) is the message passed from i to j at time step t It is a L-dimensional vector It is a L-dimensional vector Represents what node i ‘believes’ about the true state of node j. Represents what node i ‘believes’ about the true state of node j. min  C i (h i ) +  C i,j (h i,h j ) i j m i,j

Loopy Belief Propagation Message passing rule: Message passing rule: After convergence, optimal state is given by After convergence, optimal state is given by m (t+1) i,j (h j )= min { C ij (h i,h j ) +C i (h i ) +  m (t) k,i (h i ) } k  N(i) hihi h i * = min { C i (h i ) +  m (  ) k,i (h i ) } k  N(i) hihi i j m i,j

Loopy Belief Propagation O(L 2 ) to compute a message (L is number of allowable heights) O(L 2 ) to compute a message (L is number of allowable heights) Message passing schedule can be asynchronous which can accelerate convergence [Tappen & Freeman ICCV 03] Message passing schedule can be asynchronous which can accelerate convergence [Tappen & Freeman ICCV 03]

Iterative Scheme BP is memory intensive. BP is memory intensive. Can consider few possible labels at a time Can consider few possible labels at a time After convergence we ‘zoom in’ to heights close to the optimal After convergence we ‘zoom in’ to heights close to the optimal

Evaluation Artificial deformed sphere Artificial deformed sphere Textured with random patern Textured with random patern 20 images 20 images 40,000 sample points on sphere base surface 40,000 sample points on sphere base surface True surfaceTexture-mappedReconstruction

Evaluation Benchmark: 2-view, disparity based Loopy Belief Propagation [Sun et al ECCV02] Benchmark: 2-view, disparity based Loopy Belief Propagation [Sun et al ECCV02] BP run on 10 pairs of nearby views BP run on 10 pairs of nearby views Compare Disparity Maps given by Compare Disparity Maps given by  2-view BP  Relief surfaces  Ground truth

Evaluation 2-view BPRelief surface Ground truth 2-view BP Relief surf. MSE pixels pixels % of correct disparities 75.9%79.2%

Results Sarcophagus Sarcophagus

Results Sarcophagus Sarcophagus

Results Sarcophagus Sarcophagus

Results Building facade Building facade

Results Building facade Building facade

Results Base surfaceRelief surface with texture Stone carving Stone carving

Summary MRF methods can be extended in the volumetric domain MRF methods can be extended in the volumetric domain Advantages Advantages  General surfaces can be represented  Optimisation is tractable (MRF solvers)  Smoothness is viewpoint independent

Future work Photoconsistency beyond Lambertian surface models. (Optimise both height and surface normal fields) Photoconsistency beyond Lambertian surface models. (Optimise both height and surface normal fields) Change in topology Change in topology In cases where C mn (h m,h n )=|| h m -h n || or || h m - h n || 2 we can compute messages in O(L) time instead of O(L 2 ) (Felzenszwalb & Huttenlocher CVPR 04). In cases where C mn (h m,h n )=|| h m -h n || or || h m - h n || 2 we can compute messages in O(L) time instead of O(L 2 ) (Felzenszwalb & Huttenlocher CVPR 04).