Depth and Motion Discontinuities Stan Birchfield Ph.D. oral defense Stanford University January 1999

PHOTOMETRIC GEOMETRIC Discontinuities intensity, color, texture,... DepthMotion camera surface camera surface

Fundamental to image understanding Object boundaries (well-defined) Assumption of continuity Occlusion Importance of Discontinuities camera 1camera 2 object

Motivation segmentation classification tracking camera control video retrieval detection IMAGESLOW-LEVEL FILTERSHIGH-LEVEL TASKS

Evidence for Discontinuities LOCAL –T-junctions [Parida et al. 1997, Ruzon & Tomasi 1999] –Occlusion [Toh & Forrest 1990, Wixson 1993] –Multi-modal depth or motion [Spoerri & Ullman 1987, Little & Gillett 1990] –Changes in intensity, color, texture, … [Canny 1986, Wang & Binford 1994] GLOBAL –“Continuity of discontinuities” [Marr & Poggio 1979] –Changes in depth or motion field [ Thompson et al. 1982, Birchfield & Tomasi 1998]

Complications from Sampling Amount of discontinuity Curvature ?

Finding and Using Discontinuities Depth discontinuities from stereo Maximum flow for stereo and motion Motion discontinuities from a sequence Using discontinuities to track heads

Depth Discontinuities from a Stereo Pair Use dynamic programming to minimize: intensity L R disparity lamp wall pixel dissimilarity discontinuity penalty epipolar constraint

Discontinuity Penalty u(l ) i penalty occlusion length u(3) = 3u(1) u(3) < 3u(1) BAD GOOD l i L R L R BAD disparity pixel intensity

Stereo Algorithm Matches pixels directly with convex penalty function (no preprocessing or windows) Handles large untextured regions Solves sampling problem Handles slanted surfaces Is fast

Handling Untextured Regions Our constraint: Depth discontinuities lie to a particular side of intensity edge Discontinuities in left (right) scanline lie to the left (right) of intensity edge Assumption: Depth discontinuities are accompanied by intensity edge Naïve constraint: Depth discontinuities lie near intensity edge Intensity edges (x-derivative with low threshold) With naïve constraint With our constraint

Discontinuity Lies to the Left of an Edge near object far object left camera right camera Obvious matches Physical setup Matches violate assumption Matches are consistent with assumption edge depth discontinuity

The Problem of Image Sampling Absolute difference Our measure Left scanlineRight scanline

A Dissimilarity Measure That is Insensitive to Image Sampling d(x L,x R ) xLxL xRxR d(x L,x R ) = min{d(x L,x R ),d(x R,x L )}Our dissimilarity measure:

Analysis of Dissimilarity Measure Concave/convex regions: measure is guaranteed to work Inflection points: measure works in practice when lens is defocused to remove aliasing (inflection points  linear {convex, concave}) intensity function

Analysis of Dissimilarity Measure our measure absolute difference maximum average T c = 1 fcfc cutoff frequency As a function of lens defocus At a hypothesized disparity of 10 pixels aliasing { defocus

Speeding Computation D isparity map occlusion depth discontinuity Pruning bad nodes Standard Ours maximum disparity time Fast postprocessing TWO STEPS: 1. Reliable overruns unreliable 2. Background overruns foreground RIGHT LEFT

Handling Slanted Surfaces Independent scanlines: no coherence Postprocessing: forbid propagation if disparity changes by just one level disparity column

Results

More Results [Images from JISCT data set]

Finding and Using Discontinuities Depth discontinuities from stereo Maximum flow for stereo and motion Motion discontinuities from a sequence Using discontinuities to track heads

Motion Discontinuities from a Sequence Similar to stereo (matching pixels to fit piecewise-smooth function) But different (2D search, non-rigid transformation, many images) How far can we get using sparse features?

Motion Video

Difficulties of Image Sequences Instantaneous velocity from sampled positions Accumulation of evidence Frame of reference, motion model time position ? feature position time feature background threshold position time feature 2 feature 1 background threshold

Image Strain

Motion Algorithm Select and track features Group Trace boundaries between groups

Finding and Using Discontinuities Depth discontinuities from stereo Maximum flow for stereo and motion Motion discontinuities from a sequence Using discontinuities to track heads

Why Maximum Flow? Our stereo algorithm (Dynamic programming) –Good results on difficult images –Fast –But, suboptimal (processes rows, then columns independently) Newer algorithms (Maximum flow) –Set up graph, find maximum flow --- minimum cut yields disparity or motion map –Able to look at whole image

Minimizing a 2D Cost Function Minimize: disparity 2D: GLOBAL disparity pixel ? 1D: Global u(l ) p,q Discontinuity penalty: l p,q minimum cut = disparity surface solves LOCAL Local (GOOD) (BAD) 

Maximum Flow for Stereo and Motion Global algorithm [Roy & Cox 1998, Ishikawa & Geiger 1998] Local algorithm [Boykov et al. 1998] penalty BAD discontinuity amount GOOD

Challenges for Maximum Flow … and slant of table With our intensity variation constraint and large penalty With small penaltyWith large penalty … and slant of surfaces

Finding and Using Discontinuities Depth discontinuities from stereo Maximum flow for stereo and motion Motion discontinuities from a sequence Using discontinuities to track heads

Problem TILT PAN ZOOM CHALLENGES: * rotation * multiple people * zoom APPLICATIONS: * video conferencing * distance learning

Previous Methods 3D RotationMoving people in background Skin color: N? Background subtraction: YN Template: NY Contour: YN

Head Tracking Algorithm MODEL (x,y)  Ellipse: vertical aspect ratio = 1.2  state s = (x,y,  ) ellipse in frame t OUTPUT color module Model Current Intersection SkinHair LOCAL SEARCH gradient module gradient ellipse normal ellipse in frame t INPUT frame t-1

Evaluation of Head Tracker 1. Tracks head in real time on standard hardware 2. Insensitive to - full 360-degree out-of-plane rotation - arbitrary camera movement (including zoom) - multiple moving people - severe but brief occlusion - hair/skin color, hair length, facial hair, glasses

Headtracker Video

Finding and Using Discontinuities Depth discontinuities from stereo Maximum flow for stereo and motion Motion discontinuities from a sequence Using discontinuities to track heads

Contributions Precise, unified definitions and relationship Pixel-to-pixel stereo algorithm [ICCV 1998] –good results on difficult images intensity variation constraint dissimilarity measure [PAMI 1998] preserving slant –fast (pruning search nodes, postprocessor) Motion discontinuities (sparse features, image strain, grouping algorithm) Maximum flow techniques (edge weights for global algorithm, smoothness, left border ; applied to motion) Elliptical head tracker [Asilomar 1997, CVPR 1998] –advanced state of the art by tracking people undergoing 3D rotation in front of a dynamic background

Appendix: Extra slides

Comparison of Our Algorithm with Maxflow Our algorithm Local maxflow algorithm

Depth Discontinuity Proposition 1: A depth discontinuity is a point in the image plane whose projection ray grazes a surface in the world. object image plane focal point Depth discontinuities

Definition of Jump The jump of a discontinuity is the smallest  >0 such that there is a  >0 for which, if ||x-x 0 ||  x f(x) x0x0   jump

Comparison with Intensity Edges Original imageIntensity edgesDepth discontinuities More closely tied to actual object boundaries Not distracted by change in albedo (reflectance) change in lighting (e.g., shadows)

Comparison with Layers Original imageMotion discontinuities

Comparison with Segmentation A plausible segmentation Depth discontinuities Original image Well-defined (not task-dependent) Curves are not necessarily closed

Epipolar Constraint Left cameraRight camera world point center of projection epipolar plane epipolar line

Comparison of Dissimilarity Measure with Other Methods absolute difference our measure spurious matches subpixel resolution Computing time

Propagating Information Between Scanlines Calculate reliabilities & threshold Disparities: [ ] T Reliabilities: [ ] T unreliable < reliable < very reliable Propagate very reliable when –neighbor unreliable, OR –disparity(neighbor) > disparity(pixel) + 2 Repeat in x-direction Fast (30% increase instead of 800%)

Features on Freethrow

Maximum Flow Finds greatest shipping rate from source to sink w/o violating capacity constraints Flow network G=(V,E) is directed graph with edge capacities c(u,v) > 0, source s, and sink t Flow f:E R Value of flow |f| =  f(s,v) Find f* = arg max |f| Cut (S,T) is partition of V s.t. s in S and t in T capacity of cut c(S,T) Max-flow min-cut theorem: There exists a cut (S,T) s.t. |f*| = c(S,T)

Blowup of Meter Global algorithm Local algorithm

Maximum Flow and Occlusion [from University of Tsukuba, Japan]

Maximum Flow and Left Image Border