Recent progress in optical flow Presented by: Darya Frolova and Denis Simakov

Optical Flow is not in favor Very popular slide: Often not using Optical Flow is stated as one of the main advantages of a method Optical Flow methods have a reputation of either unreliable or slow Recent works claim: Optical Flow can be computed fast and accurately

Optical Flow Research: Timeline Horn&Schunck many methods more methods Lucas&Kanade 1981 1992 1998 now Benchmark: Barron et.al. Benchmark: Galvin et.al. Seminal papers A slow and not very consistent improvement in results, but a lot of useful ingredients were developed

In This Lecture We will describe : Ingredients for an accurate and robust optical flow How people combine these ingredients Fast algorithms Papers: Combining the advantages of local and global optic flow methods (“Lucas/Kanade meets Horn/Schunck”) A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005 High accuracy optical flow estimation based on a theory for warping T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005 Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 - 2005 Towards ultimate motion estimation: Combining highest accuracy with real-time performance A. Bruhn, J. Weickert, 2005 Bilateral filtering-based optical flow estimation with occlusion detection J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006

What is Optical Flow? explain: arrows and dots of flow field …

Definitions The optical flow is a velocity field in the image which transforms one image into the next image in a sequence [ Horn&Schunck ] + = frame #1 flow field frame #2 The motion field … is the projection into the image of three-dimensional motion vectors [ Horn&Schunck ]

Ambiguity of optical flow flow (1): true motion flow (2) Frame 1

Applications optical flow video compression 3D reconstruction segmentation object detection activity detection key frame extraction interpolation in time motion field We are usually interested in actual motion

Outline Ingredients for an accurate and robust optical flow Local image constraints on motion Robust statistics Spatial coherence How people combine these ingredients Fast algorithms

Local image constraints

Brightness Constancy u frame t+1 v frame t

Linearized brightness constancy Deviation from brightness constancy (we want it to be zero) Complex dependence on Linearize:

Linearized brightness constancy Let us square the difference: J – “motion tensor”, or “structure tensor”

Averaged linearized constraint J is a function of x, y (a matrix for every point) Combine over small neighborhoods (more robust to noise): J = *

Method of Lucas&Kanade Solve independently for each point [ Lucas&Kanade 1981 ] linear system Can be solved for every point where matrix is not degenerate

Lukas&Kanade - Results Rubik cube Hamburg taxi flow field flow field

Brightness is not always constant Rotating cylinder Brightness constancy does not always hold Gradient constancy holds intensity intensity derivative position position

Local constraints - Summary We have seen linearized brightness constancy averaged linearized averaged linearized gradient constancy

Local constraints are not enough!

Local constraints work poorly Optical flow direction using only local constraints input video color encodes direction as marked on the boundary

Where local constraints fail Uniform regions Motion is not observable in the image (locally)

Where local constraints fail “Aperture problem” We can estimate only one flow component (normal)

Where local constraints fail Occlusions We have not seen where some points moved Occluded regions are marked in red

Obtaining support from neighbors Two main problems with local constraints: information about motion is missing in some points => need spatial coherency constraints do not hold everywhere => need methods to combine them robustly good missing wrong

Robust combination of partially reliable data or: How to hold elections

Toy example xi xi → xi + ∆ L2: L1: Find “best” representative for the set of numbers xi L2: L1: Influence of xi on E: xi → xi + ∆ proportional to equal for all xi Outliers influence the most Majority decides

Elections and robust statistics many ordinary people a very rich man wealth Oligarchy Democracy ordinary people – 1 vote a rich man – many votes If you know what to do – choose oligarchy, if don’t know - democracy Votes proportional to the wealth One vote per person like in L2 norm minimization like in L1 norm minimization

Combination of two flow constraints usual: L2 easy to analyze and minimize sensitive to outliers robust: L1 robust in presence of outliers non-smooth: hard to analyze robust regularized smooth: easy to analyze robust in presence of outliers ε [A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance

Spatial Propagation

Obtaining support from neighbors Two main problems with local constraints: information about motion is missing in some points => need spatial coherency constraints do not hold everywhere => need methods to combine them robustly good missing wrong

Homogeneous propagation - flow in the x direction flow in the y direction gradient This constraint is not correct on motion boundaries => over-smoothing of the resulting flow [Horn&Schunck 1981]

Robustness to flow discontinuities ε (also known as isotropic flow-driven regularization) [T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004] High accuracy optical flow estimation based on a theory for warping

Selective flow filtering We want to propagate information without crossing image and flow discontinuities from “good” points only (not occluded) Solution: use “bilateral” filter in space, intensity, flow; taking into account occlusions [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006] Bilateral filtering-based optical flow estimation with occlusion detection

Bilateral filter Unilateral (usual) Bilateral x I I x Preserves discontinuities! x I x I [C. Tomasi, R. Manduchi, 1998] Bilateral filtering for gray and color images.

Using of bilateral filter - Example occluded regions cyan rectangle moves to the right and occludes background region marked by red [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006] Bilateral filtering-based optical flow estimation with occlusion detection

Learning of spatial coherence Come to the next lecture …

Spatial coherence: Summary Homogeneous propagation - oversmoothing Robust statistics with homogeneous propagation - preserves flow discontinuities Bilateral filtering - combines information from regions with similar flow and similar intensities Handles occlusions

Two more useful ingredients in brief – one slide each

2D vs. 3D Several frames allow more accurate optical flow estimation

Multiscale Optical Flow Linearization: valid only for small flow pyramid for frame 1 pyramid for frame 2 frame 1 warped ? + upsample + (other names: “warping”, “coarse-to-fine”, “multiresolution”)

How to make tasty soup with these ingredients: several recipes Methods How to make tasty soup with these ingredients: several recipes

Outline Ingredients for an accurate and robust optical flow How people combine these ingredients Lukas & Kanade meet Horn & Schunck The more ingredients – the better Bilateral filtering and occlusions Fast algorithms

Combining ingredients Spatial coherency Homogeneous Flow-driven Bilateral filtering + occlusions Local constraints Brightness constancy Image gradient constancy Energy = ∫ϕ (Data) + ∫ϕ (“Smoothness”) Combined using robust statistics Computed coarse-to-fine Use several frames

Combining Local and Global Remember: Lucas&Kanade Horn&Schunk Basic “Combining local and global” [A. Bruhn, J. Weickert, C. Schnörr, 2002 ]

Sensitivity to noise – quantitative results frame t+1 Error measure: angle between true and computed flow in (x,y,t) space frame t ground truth flow

The more ingredients - the better brightness constancy spatial coherence gradient constancy [Bruhn, Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance

Combining local and global Quantitative results Average Standard Deviation Lucas&Kanade 4.3 (density 35%) Horn&Schunk 9.8 16.2 Combining local and global 4.2 7.7 “Towards ultimate …” 2.4 6.7 Angular error Method Yosemite sequence with clouds Average error decreases, but standard deviation is still high….

Influence of each ingredient For Yosemite sequence with clouds

Handling occlusions bilateral filtering of flow: preserve intensity and flow discontinuities; model occlusions [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, ECCV 2006] Bilateral filtering-based optical flow estimation with occlusion detection

Qualitative results

Combining local and global Bilateral+occlusions Quantitative results Average Standard Deviation Lucas&Kanade 4.3 (density 35%) Horn&Schunk 9.8 16.2 Combining local and global 4.2 7.7 “Towards ultimate …” 2.4 6.7 Bilateral+occlusions 2.6 6.1 Angular error Method Yosemite sequence with clouds

Outline Ingredients for an accurate and robust optical flow How people combine these ingredients Fast algorithms Energy functional => discrete equation Multigrid solver: nearly real-time

Euler-Lagrange equation How to minimize energy Analogy: Necessary condition: Necessary condition Euler-Lagrange equation

An example Let us see how to derive discretized equation for 1D Horn & Schuhck Horn&Schunk 1D version (simplified):

Iterative minimization (simple example) Euler-Lagrange Linear system of equation for u Discretized: Local iterations:

Life is not a picnic Linear discretized system Non-linear in u, non-linear discretized system Even more complicated

Optimization algorithms Simple iterative minimization Multigrid: much faster convergence

Solving the system How to solve? Start with some initial guess and apply some iterative method fast convergence good initial guess 2 components of success:

Relaxation smoothes the error Relaxation schemes have smoothing property: . . . . . . . . . . . . It may take thousands of iterations to propagate information to large distance oscillatory modes of the error are eliminated effectively, but smooth modes are damped slowly Only neighboring pixels are coupled in relaxation scheme

Relaxation smoothes the error Examples 1D case: 2D case: Error of initial guess Error after 5 relaxation Error after 15 relaxations

Idea: coarser grid On a coarser grid low frequencies become higher initial grid – fine grid On a coarser grid low frequencies become higher Hence, relaxations can be more effective coarse grid – we take every second point

Multigrid 2-Level V-Cycle 5. Correct the previous solution 6. Iterate ⇒ remove interpolation artifacts 1. Iterate ⇒ error becomes smooth 2. Transfer error equation to the coarse level ⇒ low frequencies become high 4. Transfer error to the fine level 3. Solve for the error on the coarse level ⇒ good error estimation

Coarse grid - advantages Coarsening allows: make iteration process faster (on the coarse grid we can effectively minimize the error) obtain better initial guess (solve directly on the coarsest grid) go to the coarsest grid interpolate to the finer grid solve here the equation to find

Multigrid approach – Full scheme

Non-linear: Full Approximation Scheme A(·) non-linear Linear Non-linear fine level: coarse level: Difference from the linear case: Equation for error involves current solution u0: ⇒ Need to transfer current solution to the coarser level

Multigrid: Summary Used to solve linear or non-linear equations Method: combine two techniques Basic iterative solver: quickly removes high frequencies of the error Coarsening: makes low frequencies high Contribution: fast minimization of loosely coupled equations

Fast Optimization: Results Time [sec] frames/sec Gauss-Seidel 1.15 0.87 Full multigrid 0.016 62.8 9.52 0.11 FAS-multigrid 0.087 11.5 34.5 0.03 0.396 2.5 Horn&Schunck CLG “Towards ultimate…” image size: 160 x 120

Summary of the Talk 25 years of Optical Flow : a lot of useful ingredients were developed: local constraints: brightness constancy gradient constancy smoothing techniques: homogeneous flow-driven (preserving discontinuities) bilateral filters handling of occlusions robust functions multiscale All ingredients are combined an a global Energy Minimization approach This difficult global optimization can be done very fast using Multigrid

Thank you!