Binocular Stereo

Topics Principle basic equation epipolar line features and strategies for matching Case study Block matching Relaxation DP stereo

Basic principles

Binocular stereo single image is ambiguous A another image taken from a different direction gives the unique 3D point a’ a”

Epipolar line Epipolar plane Epipolar line constraints Corresponding points lie on the Epipolar lines Epipolar line constratints Base line One image point Possible line of sight

Epipoles C1C1 C2C2 e1e1 e2e2 intersections of baseline with image planes projection of the optical center in another image the vanishing points of camera motion direction

Examples of epipolar lines

Rectification rectification

Terminology A physical point focal length right image point z left image point base line length right image plane left image plane World coordinate system left image center right image center

Pinhole Camera

Perspective Projection (X, Y, Z) Image plane X Y -Z u v (u, v) f : focal length View point (Optical center)

Basic binocular stereo equation z=-2df/(x”-x’) x”-x’: disparity 2d : base line length x” x’ -z f d d z d + xd - x

Features for matching a. brightness b. edges c. edge intervals d. interest points

a. relaxation b. coarse to fine c. dynamic programming local optimam Strategies for matching global optimam

Classification of stereo methods Features for matching brightness value point edge region Strategies for matching brute-force coarse-to-fine relaxation dynamic programming Constraints for matching epipolar lines disparity limit continuity uniqueness

Case study

Block-Matching Stereo 1. method b c bcbc 2. problem a. trade-off of window size and resolution b. dull peak b c

Cost Function (b) SSD (sum. of squared difference) (a) SAD (sum. of absolute difference) (c) Correlation Near Object left d Background Near Object right Background

Moravec Stereo(`79) navigation Moravec “Visual mapping by a robot rover” Proc 6th IJCAI,pp (1979)

Moravec’s cart Slide stereo Motion stereo

Slider stereo (9 eyes stereo) u 9 C 2 = 36 stereo pairs!!! u each stereo has an uncertainty measure u uncertainty = 1 / base-line u each stereo has a confidence measure long base line large uncertainty

Coarse to fine expand matching

σ estimated distance σ:uncertainty measure area:confidence measure 9 C 2 = 36 curves Interest point

1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force (not a strategy ???) b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. Uniqueness Purpose: navigation (Stanford) Moravec Stereo(`81) interest point

Near Object Recent Progress left disparity Background How to estimate the disparities? disparity Near Object right Background Minimize some cost function along the epipolar line H. Hirschnuller, "Improvements in Real-Time Correlation-Based Stereo Vision", IEEE Workshop on Stereo and Multi-Baseline Vision, 2001

Fatting Effect on Object Boundary Near Object left Background Near Object right Background Background Correspondence Foreground Correspondence u No single window fits at the discontinuity ⇒ Fatting effect of the object

Near Object Background Accurate Estimation on Object Boundary u Shiftable Window left Near Object right Background disparity Min{c1,c2,c3,c4} Min{{c1,c2,c3,c4}-c’} c0 c1c2 c3c4 d c0 c1 c3 are used.

Consistency Checking u Check if two independent disparity estimation coincide –Left ⇒ Right search –Right ⇒ Left search u Inconsistent disparities are considered as a false match Epipolar line left right Epipolar line 1 st search 2 nd search Check if they coincide

Result u SAD with 11x11 window u Shiftable window + consistency checking Disparity mapLeft image

Cooperative stereo: Marr-Poggio Stereo(`76) Simulating human visual system (random dot stereo gram) Marr,Poggio “Cooperative computation of stereo disparity” Science 194,

Input : random dot stereo left image random dot shift the catch pat right image we can see the height different between the central and peripheral area

Constraints –Epipolar line constraint –Uniqueness constraint »each point in a image has only one depth value O.K. No. –Continuity constraint »each point is almost sure to have a depth value near the values of neighbors O.K. No.

Uniqueness constraint prohibits two or more matching points on one horizontal or vertical lines continuity constraint attracts more matching on a diagonal line ABCABC D E F A B C ABCABC (E-A) (E-B) (E-C) prohibit attract (D-A) (E-B) (F-C) Same depth

nn+1 Relaxation

1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness Purpose: simulate the human visual system (MIT) Marr-Poggio Stereo (`76)

Recent progress: Graph-cut u Solve graph partition problem in globally optimal way 1.Formulate the problem in energy minimization framework 2.Design a graph such that the sum of cut edges equals to the total energy 3.Find a “Cut” that minimizes the energy i j Vij Cut

Example of Graph-cut u Image segmentation Graph={N,e} Ni: Graph node e ij : Edge connecting nodes C: Cut Image={pixel} Vij: Similarity between neighboring pixels Foreground/Background boundary Ni Nj Vij Graph partition Segmentation

Solution to a Graph-cut Problem u Min-Cut/Max-Flow algorithm 1.Given source (s) and sink nodes (t) 2.Define capacity on each edge 3.Find the maximum flow from s ⇒ t, satisfying capacity constraints, and cut the bottleneck Yuri Boykov, Vladimir Kolmogorov, "An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision", PAMI, 2004 i j SourceSink Min-Cut = Max-Flow Bottleneck Flow

Multi-label Problem u Find the labeling f that minimizes the energy measures the extent to which f is not piece wise smooth measures the disagreement between f and the observed data Measures how well label fp fits pixel p given the observed data Smoothness penalty between adjacent (N) pixels Yuri Boykov and Olga Veksler and Ramin Zabih, “Fast Approximate Energy Minimization via Graph Cuts,” ICCV, 2001

u Iterative graph-cut approach u 2 types of move algorithm are proposed –αβ-swap –α-expansion Minimize E under cond. is preserved Minimize E under cond. can be changed to α Multi-label Solution via Graph-cut α β γ α β γ Graph-cut

αβ-swap Algorithm 1.Start with an arbitrary labeling f 2.Success:= 0 3.For each pair of labels a.Find f’=arg min E(f’) among f’ within one αβ-swap of f b.If E(f’) < E(f) then f’:=f and success:=1 4.If success=1 goto 2 5.Return f α β γ

αβ-swap Graph Structure pqrs α β edgeweight for should be a semi metric

αβ-swap Cut u 3 possible cases pq α β pq α β pq α β Cut ααβββα

α-expansion Algorithm 1.Start with an arbitrary labeling f 2.Success:= 0 3.For each label a.Find f’=arg min E(f’) among f’ within one α-expansion of f b.If E(f’) < E(f) then f’:=f and success:=1 4.If success=1 goto 2 5.Return f α β γ

α-expansion Graph Structure pqrs α edgeweight for should be a metric Auxiliary nodes are added at the boundary of sets P where a b

α-expansion Cut u 3 possible cases pq α Cut a pq α a pq α a Because V(a,b) is a metric V(a,b) < V(a,c)+V(c,b) Never happens! ααα

Pixel-based Stereo Matching via Graph-cut leftright Image 12L minimizing the cost function by iterative graph cut (α-expansion) Label: Labeling f means disparity assignment p

Graph-cut with Occlusions u Occluded pixels are handled explicitly in the graph u Find a subset of A u Find a configuration f such that V. Kolmogorov and R. Zabih, “Computing visual correspondence with occlusions via graph cuts,” ICCV, 2001 leftright p q if the pixels (p,q) correspond otherwise p is an occluded pixel

Energy Function to Minimize Occlusion penalty Number of pixels paired with p T(a) is 1 if a=true, otherwise 0

Minimization Flow pqrs wxyz left right disparity 0 disparity 1 disparity α (=2) Current assignment Possible assignment after α-expansion α Choose α Partition via Graph-cut (α-expansion) Construct a Graph Assign weight to each edge

Result Left ImageGround Truth Graph-cut with Occ. Graph-cut without Occ. L1 correlation

DP stereo Ohta-Kanade Stereo(`85) Map making Ohta,Kanade “Stereo by intra- and inter-scanline search using dynamic programming”,IEEE Trans.,Vol. PAMI-7,No.2,pp

now matching become 1D to 1D yet, N line * M L * M R (512 * 100 * 100 * 10 m sec = 15 hours) L1 L2 L3 L4 L5 L6 R1 R2 R3 R4 R5 R6 L R disparity

Path Search u Matching problem can be considered as a path search problem u define a cost at each candidate of path segment based some ad-hoc function

Dynamic programming We can formalize the path finding problem as the following iterative formula optimum cost to K cost between M and K Optimum costs are known

stereo pair edges

pathdisparity depth

stereo pair edges depth

1. Features for matching a. brightness value b. point c. edge d. region 2. Strategies for matching a. brute-force b. coarse-to-fine c. relaxation d. dynamic programming 3. Constraints for matching a. epipolar lines b. disparity limit c. continuity d. uniqueness aerial image analysis (CMU) Ohta-Kanade Stereo(`85) Brightness of interval

Recent progress: 4-move, 4-plane DP u Occluded pixels are handled explicitly in 4-move, 4- plane representation u Disparity map is calculated under DP Matching (global energy minimization) A. Criminisi; J. Shotton; A. Blake; C. Rother; P.H.S. Torr, “Efficient Dense-Stereo and Novel-view Synthesis for Gaze Manipulation in One-to-one Teleconferencing,” MSR-TR , 2003

Occluded path and visible path cannot be distinguished in this representation occluded move Conventional 3-move DP Right Left occluded move Matched move Left scan-line Right scan-line True matching path Approximated matching path Problem Visible only from right Visible from both left and right

4-move DP Occluded path and visible path are handled separately Occluded move (r) Matched move(l) Left scanline Right scanline True matching path Approximated matching path Occluded move(l) Matched move (r)

Design of Move Transition Lm Lo Ro Rm Normalized Sum of Squared Difference Move Transition Matching Cost Lm Left Matched Move Right Occluded Move Ro Rm Lo Left Occluded Move Right Matched Move

4-move, 4-plane DP u Each node should hold 4 accumulation costs separately for each move ⇒ 4-plane model

Inter-scanline Consistency u Propagate information across scan-lines ⇒ Gaussian filter is applied on Matching cost array Without Gaussian filterWith Gaussian filter Left Occluded Pixels Right Occluded Pixels

Result Left Occluded Pixels Right Occluded Pixels Input Images 3-move DP 4-move, 4-plane DP

Comparison in Severe Situation Graph-cut with occlusions 4-move, 4-plane DPBlock Matching leftright Input Result Occluded Pixels Texture-less region

Summary 1.Two images from two different positions give depth information 2. Epipolar line and plane 3. Basic equation Z=-2df/(x”-x’) x”-x’: disparity 2d : base line length 4. case study Block-matching-based stereo Cooperative stereo DP based stereo

Recent directions 1. Block Matching (Local optimization) –Fast for real-time applications (parallel processing, SIMD) –Not accurate in texture-less region 2. Graph-cut variants (Global optimization) uglobally consistent disparity map is obtained u texture-less region is interpolated nicely move, 4-plane, DP Matching (Global optimization) uglobally consistent disparity map is obtained utexture-less region is interpolated nicely uInter scan-line inconsistency is reduced, but yet to be seen 4. Belief Propagation