CISC 489/689 Spring 2009 University of Delaware

Presentation on theme: "CISC 489/689 Spring 2009 University of Delaware"— Presentation transcript:

CISC 489/689 Spring 2009 University of Delaware
Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware

Outline Review of Optical Flow Constraint, Lucas-Kanade, Horn and Schunck Methods Lucas-Kanade Meets Horn and Schunck 3D Regularization Techniques for solving optical flow Confidence Measures in Optical Flow

Optical Flow Constraint

Interpretation Constraint Line

Lucas-Kanade Method Local Method, window based
Cannot solve for optical flow everywhere Robust to noise Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005

Dense optical Flow Lacks Smoothness
Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005

Horn and Schunck Method
Euler-Lagrange Equations

Horn and Schunck Method
Global Method Estimates flow everywhere Sensitive to noise Oversmooths the edges Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005

Why combine them? Need dense flow estimate Robust to noise
Preserve discontinuities

Combining the two…

Combined Local Global Method
Euler-Lagrange Equations Average Error Standard Deviation Lucas&Kanade 4.3 (density 35%) Horn&Schunk 9.8 16.2 Combining local and global 4.2 7.7 Table: Courtesy - Darya Frolova, Recent progress in optical flow

Preserving discontinuities
Gaussian Window does not preserve discontinuities Solutions Use bilateral filtering Add gradient constancy

Bilateral support window
Images: Courtesy, Darya Frolova, Recent progress in optical flow

Robust statistics – simple example
Find “best” representative for the set of numbers xi L2: L1: Influence of xi on E: xi → xi + ∆ proportional to equal for all xi Outliers influence the most Majority decides Slide: Courtesy - Darya Frolova, Recent progress in optical flow

Robust statistics many ordinary people a very rich man wealth
Oligarchy Democracy ordinary people – 1 vote a rich man – many votes If you know what to do – choose oligarchy, if don’t know - democracy Votes proportional to the wealth One vote per person like in L2 norm minimization like in L1 norm minimization Slide: Courtesy - Darya Frolova, Recent progress in optical flow

Combination of two flow constraints
usual: L2 easy to analyze and minimize sensitive to outliers robust: L1 robust in presence of outliers non-smooth: hard to analyze robust regularized smooth: easy to analyze robust in presence of outliers ε [A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance Slide: Courtesy - Darya Frolova, Recent progress in optical flow

Robust statistics

3D Regularization accounted for spatial regularization
If velocities do not change suddenly with time, can we regularize in time as well?

3D Regularization

Multiresolution estimation
Gaussian pyramid of image 1 Gaussian pyramid of image 2 Image 2 image 1 run iterative estimation warp & upsample run iterative estimation . image J image I

Compute Iterative LK at highest level For Each Level i Take flow u(i-1), v(i-1) from level i-1 Upsample the flow to create u*(i), v*(i) matrices of twice resolution for level i. Multiply u*(i), v*(i) by 2 Compute It from a block displaced by u*(i), v*(i) Apply LK to get u’(i), v’(i) (the correction in flow) Add corrections u’(i), v’(i) to obtain the flow u(i), v(i) at the ith level, i.e., u(i)=u*(i)+u’(i), v(i)=v*(i)+v’(i)

Comparison of errors For Yosemite sequence with clouds
Table: Courtesy - Darya Frolova, Recent progress in optical flow

and apply some iterative method fast convergence good initial guess 2 components of success:

Relaxation smoothes the error
Relaxation schemes have smoothing property: It may take thousands of iterations to propagate information to large distance oscillatory modes of the error are eliminated effectively, but smooth modes are damped slowly Only neighboring pixels are coupled in relaxation scheme

Relaxation smoothes the error Examples
1D case: 2D case: Error of initial guess Error after 5 relaxation Error after 15 relaxations

Idea: coarser grid initial grid – fine grid
On a coarser grid low frequencies become higher Hence, relaxations can be more effective coarse grid – we take every second point

Multigrid 2-Level V-Cycle
5. Correct the previous solution 6. Iterate ⇒ remove interpolation artifacts 1. Iterate ⇒ error becomes smooth 2. Transfer error equation to the coarse level ⇒ low frequencies become high 4. Transfer error to the fine level 3. Solve for the error on the coarse level ⇒ good error estimation

Coarsening allows: make iteration process faster (on the coarse grid we can effectively minimize the error) obtain better initial guess (solve directly on the coarsest grid) go to the coarsest grid interpolate to the finer grid solve here the equation to find

Multigrid approach – Full scheme

Confidence Metric Intrinsic in Local Methods
How to evaluate for global methods? Edge strength? Doesn’t work (Barron et al.,1994)

Confidence Metric Histogram of error contribution Number of pixels

Confidence Metric

More Results

More Results

Further Reading Combining the advantages of local and global optic flow methods (“Lucas/Kanade meets Horn/Schunck”) A. Bruhn, J. Weickert, C. Schnörr, High accuracy optical flow estimation based on a theory for warping T. Brox, A. Bruhn, N. Papenberg, J. Weickert, Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, Towards ultimate motion estimation: Combining highest accuracy with real-time performance. A. Bruhn, J. Weickert, 2005 Bilateral filtering-based optical flow estimation with occlusion detection. J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006