1. Multiview stereo

2. Volumetric stereo Scene Volume V Input Images (Calibrated) Goal: Determine occupancy, “color” of points in V

3. Discrete formulation: Voxel coloring Discretized Scene Volume Input Images (Calibrated) Goal: Assign RGBA values to voxels in V photo-consistent with images

4. Complexity and computability Discretized Scene Volume N voxels C colors 3 All Scenes ( C N 3 ) Photo-Consistent Scenes True Scene

5. Voxel coloring 1. Choose voxel 2. Project and correlate 3.Color if consistent (standard deviation of pixel colors below threshold) Visibility Problem: in which images is each voxel visible?

6. Depth ordering: occluders first! Layers SceneTraversal Condition: depth order is the same for all input views

7. Panoramic Depth Ordering Cameras oriented in many different directions Planar depth ordering does not apply

8. Layers radiate outwards from cameras

11. Compatible Camera Configurations Outward-Looking cameras inside scene Inward-Looking cameras above scene

12. Voxel Coloring Results Dinosaur Reconstruction 72 K voxels colored 7.6 M voxels tested 7 min. to compute on a 250MHz SGI Flower Reconstruction 70 K voxels colored 7.6 M voxels tested 7 min. to compute on a 250MHz SGI

13. Limitations of Depth Ordering A view-independent depth order may not exist pq Need more powerful general-case algorithms Unconstrained camera positions Unconstrained scene geometry/topology

14. Space Carving Algorithm Image 1 Image N …... Initialize to a volume V containing the true scene Repeat until convergence Choose a voxel on the current surface Carve if not photo-consistent Project to visible input images

15. Convergence Consistency Property The resulting shape is photo-consistent all inconsistent points are removed Convergence Property Carving converges to a non-empty shape a point on the true scene is never removed p

16. Which shape do you get? The Photo Hull is the UNION of all photo-consistent scenes in V It is a photo-consistent scene reconstruction Tightest possible bound on the true scene True Scene V Photo Hull V

17. Space Carving Results: African Violet Input Image (1 of 45)Reconstruction

18. Space Carving Results: Hand Input Image (1 of 100) Views of Reconstruction

19. Multi-Camera Scene Reconstruction via Graph Cuts

20. Comparison with stereo Much harder problem than stereo In stereo, most scene elements are visible in both cameras It is common to ignore occlusions Here, almost no scene elements are visible in all cameras Visibility reasoning is vital

21. Key issues Visibility reasoning Incorporating spatial smoothness Computational tractability Only certain energy functions can be minimized using graph cuts! Handle a large class of camera configurations Treat input images symmetrically

22. Approach Problem formulation Discrete labels, not voxels Carefully constructed energy function Minimizing the energy via graph cuts Local minimum in a strong sense Use the regularity construction Experimental results Strong preliminary results

23. Problem formulation Discrete set of labels corresponding to different depths For example, from a single camera Camera pixel plus label = 3D point Goal: find the best configuration Labeling for each pixel in each camera Minimize an energy function over configurations Finding the exact minimum is NP-hard

24. Sample configuration C1 p r q C2 l = 2 l = 3 Depth labels

25. Energy function has 3 terms: smoothness, data, visibility Neighborhood systems involve 3D points Smoothness: spatial coherence (within camera) Data: photoconsistency (between cameras) Two pixels looking at the same scene point should see similar intensities Visibility: prohibit certain configurations (between cameras) A pixel in one camera can have its view blocked by a scene element visible from another camera

26. Smoothness neighborhood r C1 p l = 2 l = 3 Depth labels Smoothness neighbors

27. Smoothness term Smoothness neighborhood involves pairs of 3D points from the same camera We’ll assume it only depends on a pair of labels for neighboring pixels Usual 4- or 8-connected system among pixels Smoothness penalty for configuration f is V must be a metric, i.e. robustified L 1 (regularity)

28. Photoconsistency constraint C2 q C1 p l = 2 l = 3 Depth labels If this 3D point is visible in both cameras, pixels p and q should have similar intensities

29. Photoconsistency neighborhood C1 p C2 q l = 2 l = 3 Depth labels Photoconsistency neighbors

30. Data (photoconsistency) term Photoconsistency neighborhood N photo Arbitrary set of pairs of 3D points (same depth) Current implementation: if the projection of on C2 is nearest to q Our data penalty for configuration f is Negative for technical reasons (regularity)

31. Visibility constraint C1 p C2 q l = 2 l = 3 Depth labels is an impossible configuration

32. Visibility neighborhood C1 p C2 q l = 2 l = 3 Depth labels Visibility neighbors

33. Visibility term Visibility neighborhood N vis is all pairs of 3D points that violate the visibility constraint Arbitrary set of pairs of points at different depths Needed for regularity The pair of points come from different cameras Current implementation: based on the photoconsistency neighborhood A configuration containing any pair of 3D points in the visibility neighborhood has infinite cost

34. C1C2 Input (non-binary) f Red expansion move Energy minimization via expansion move algorithm We must solve the binary energy minimization problem of finding the  -expansion move that most reduces E We only need to show that all the terms in E are regular!

35. Smoothness term is regular True because V is a metric

36. Visibility term is regular Consider a pair of pixels p,q Input configuration has finite cost Therefore A =0 3D points at the same depth are not in visibility neighborhood N vis Therefore D =0 B,C can be 0 or , hence non- negative

37. Data term is regular

38. Tsukuba images Our results, 4 interactions

39. Comparison Our results, 10 interactionsBest results [SS ’02]