Discontinuity Preserving Stereo with Small Baseline Multi-Flash Illumination Rogerio Feris 1, Ramesh Raskar 2, Longbin Chen 1, Karhan Tan 3 and Matthew Turk 1 1 University of California, Santa Barbara 2 Mitsubishi Electric Research Labs 3 Epson Palo Alto Lab

Introduction Correspondence Problem Stereo Near Depth Discontinuities: - Occlusion Problem - Perspective Distortions - Violation of Smoothness Constraints Passive Versus Active Methods

Introduction Our Approach: Small Baseline Multi-Flash Illumination - Simple, Inexpensive - Compact, Self-Contained - Discontinuity Preserving

Depth Edges with Multi-Flash Raskar, Tan, Feris, Yu, Turk – ACM SIGGRAPH 2004

Bottom Flash Top Flash Left Flash Right Flash Ratio images and directions of epipolar traversal Shadow-Free Depth Edges Shadow-FreeDepth Edges

Qualitative Depth Map

Qualitative Depth Sign of Depth Edge - Indicates which side is the foreground and which side is the background Shadow Width - Encodes object relative distances

Sign of Depth Edge (+) Foreground (-) Background Original Ratio Left Ratio Right Signed Edges

Shadow Width Bottom Flash Image Ratio Image Plot Along Scanline

Shadow Width Bottom Flash Image Ratio Image Shadow Width Estimation: Meanshift Segmentation algorithm applied on the ratio image

Imaging Geometry Object Flash Shadow Camera B z1z1 z2z2 f Shadow Width d

Qualitative Depth Working on this Equation … Log Depth Difference Shadow Width Gradient-Domain Problem!

Qualitative Depth 1) Compute Sharp Depth Gradient G = (G h,G v ) Log Depth Difference Sign of depth edge 2) Compute Q by integrating G (Poisson Equation) 3) Qualitative depth map Q = exp(Q)

Qualitative Depth Useful Prior Information for Stereo !

Occlusion Map

Partial Occlusion Problem Object Camera Occlusion AB (Seen by A but not by B)

Occlusion Bounded by Shadows Object CameraAB Flash Occlusion (Seen by A but not by B)

Occlusion Bounded by Shadows Object CameraAB Flash Lower Bound Shadow

Occlusion Bounded by Shadows Object CameraAB Flash Upper Bound Shadow

Occlusion Bounded by Shadows Object Camera Occlusion AB Average of Upper/Lower Shadow widths Flash

Occlusion Bounded by Shadows Occlusion Map Left ViewRight View

Discontinuity Preserving Stereo Matching

Local Stereo Problem: Shape and size of correlation window - Small Window Ambiguities / Noise - Large Window Problems at Depth Discontinuities Depth Edge Preserving Local Stereo

Local Stereo Smooth Disparity Delimited by depth edges + Occlusions Correlation Window Problem: Shape and size of correlation window - Small Window Ambiguities / Noise - Large Window Problems at Depth Discontinuities Depth Edge Preserving Local Stereo

Local Stereo Left View Depth Edges + OcclusionGround Truth Challenging Scene: - Ambiguous patterns, textureless regions, geometrically complex object, thin structures

Local Stereo Conventional 9x9 Conventional 31x31 Our Approach 31x Conventional Stereo Our Approach

Global Stereo Global Optimization – Markov Random Field (MAP-MRF) X = {x s } Disparity of each pixel (Hidden) Y = {y s } Matching cost at each disparity (Observed) X3X3 X1X1 X2X2 X7X7 X4X4 X6X6 X5X5 X8X8 y1y1 y2y2

Global Stereo Global Optimization – Markov Random Field (MAP-MRF) X = {x s } Disparity of each pixel (Hidden) Y = {y s } Matching cost at each disparity (Observed) Data Term Smoothness Term Inference by Belief Propagation [Jian Sun et al, 2003]

Global Stereo Qualitative Depth Map as Evidence - Used to set the smoothness term - Information propagation is stopped at depth edges - Encourage disparities for neighboring pixels according to depth difference in qualitative map Occlusion Penalty

Global Stereo Conventional Belief Propagation Our Approach RMS: RMS:

Conclusions Contributions - Stereo with small baseline illumination - Useful Feature Maps (Qualitative Depth + Occlusion Map) - Enhanced Local and Global Stereo Algorithms Pros / Cons - Robust, Simple, Inexpensive and Compact - Limited to handle outdoor scenes and motion Website (datasets, source code) -

Thank you ! Multi-Flash Stereo Webpage Four Eyes Lab, UCSB

Occlusion Bounded by Shadows Occlusion Detection by averaging length of shadows Images taken with light sources surrounding the other camera