1. CISC 489/689 Spring 2009 University of Delaware
Optical Flow Methods
CISC 489/689
Spring 2009
University of Delaware

2. Outline
Review of Optical Flow Constraint, Lucas-Kanade, Horn and Schunck Methods
Lucas-Kanade Meets Horn and Schunck
3D Regularization
Techniques for solving optical flow
Confidence Measures in Optical Flow

3. Optical Flow Constraint

4. Interpretation
Constraint Line

5. Lucas-Kanade Method * =

6. Lucas-Kanade Method Local Method, window based
Cannot solve for optical flow everywhere
Robust to noise
Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic
Flow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005

7. Dense optical Flow Lacks Smoothness
Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic
Flow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005

8. Horn and Schunck Method
Euler-Lagrange Equations

9. Horn and Schunck Method
Global Method
Estimates flow everywhere
Sensitive to noise
Oversmooths the edges
Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic
Flow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005

10. Why combine them? Need dense flow estimate Robust to noise
Preserve discontinuities

11. Combining the two…

12. Combined Local Global Method
Euler-Lagrange Equations
Average
Error
Standard Deviation
Lucas&Kanade
4.3
(density 35%)
Horn&Schunk
9.8
16.2
Combining local and global
4.2
7.7
Table: Courtesy - Darya Frolova, Recent progress in optical flow

13. Preserving discontinuities
Gaussian Window does not preserve discontinuities
Solutions
Use bilateral filtering
Add gradient constancy

14. Bilateral support window
Images: Courtesy, Darya Frolova, Recent progress in optical flow

15. Robust statistics – simple example
Find “best” representative for the set of numbers xi
L2:
L1:
Influence of xi on E:
xi → xi + ∆
proportional to
equal for all xi
Outliers influence the most
Majority decides
Slide: Courtesy - Darya Frolova, Recent progress in optical flow

16. Robust statistics many ordinary people a very rich man wealth
Oligarchy
Democracy
ordinary people – 1 vote
a rich man – many votes
If you know what to do – choose oligarchy, if don’t know - democracy
Votes proportional to the wealth
One vote per person
like in L2 norm minimization
like in L1 norm minimization
Slide: Courtesy - Darya Frolova, Recent progress in optical flow

17. Combination of two flow constraints
usual: L2
easy to analyze and minimize
sensitive to outliers
robust: L1
robust in presence of outliers
non-smooth: hard to analyze
robust regularized
smooth: easy to analyze
robust in presence of outliers ε
[A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance
Slide: Courtesy - Darya Frolova, Recent progress in optical flow

18. Robust statistics

19. 3D Regularization accounted for spatial regularization
If velocities do not change suddenly with time, can we regularize in time as well?

20. 3D Regularization

21. Multiresolution estimation
Gaussian pyramid of image 1
Gaussian pyramid of image 2
Image 2
image 1
run iterative estimation
warp & upsample
run iterative estimation .
image J
image I

22. Multi-resolution Lucas Kanade Algorithm
Compute Iterative LK at highest level
For Each Level i
Take flow u(i-1), v(i-1) from level i-1
Upsample the flow to create u*(i), v*(i) matrices of twice resolution for level i.
Multiply u*(i), v*(i) by 2
Compute It from a block displaced by u*(i), v*(i)
Apply LK to get u’(i), v’(i) (the correction in flow)
Add corrections u’(i), v’(i) to obtain the flow u(i), v(i) at the ith level, i.e., u(i)=u*(i)+u’(i), v(i)=v*(i)+v’(i)

23. Comparison of errors For Yosemite sequence with clouds
Table: Courtesy - Darya Frolova, Recent progress in optical flow

24. Solving the system How to solve? Start with some initial guess
and apply some iterative method
fast convergence
good initial guess
2 components of success:

25. Relaxation smoothes the error
Relaxation schemes have smoothing property:
It may take thousands of iterations to propagate information to large distance
oscillatory modes of the error are eliminated effectively, but smooth modes are damped slowly
Only neighboring pixels are coupled in relaxation scheme

26. Relaxation smoothes the error Examples
1D case:
2D case:
Error of initial guess
Error after 5 relaxation
Error after 15 relaxations

27. Idea: coarser grid initial grid – fine grid
On a coarser grid low frequencies become higher
Hence, relaxations can be more effective
coarse grid – we take every second point

28. Multigrid 2-Level V-Cycle
5. Correct the previous solution
6. Iterate ⇒ remove interpolation artifacts
1. Iterate ⇒ error becomes smooth
2. Transfer error equation to the coarse level ⇒ low frequencies become high
4. Transfer error to the fine level
3. Solve for the error on the coarse level ⇒ good error estimation

29. Coarse grid - advantages
Coarsening allows:
make iteration process faster (on the coarse grid we can effectively minimize the error)
obtain better initial guess (solve directly on the coarsest grid)
go to the coarsest grid
interpolate to the finer grid
solve here the equation
to find

30. Multigrid approach – Full scheme

31. Confidence Metric Intrinsic in Local Methods
How to evaluate for global methods?
Edge strength?
Doesn’t work (Barron et al.,1994)

32. Confidence Metric Histogram of error contribution Number of pixels

33. Confidence Metric

34. More Results

35. More Results

36. Further Reading
Combining the advantages of local and global optic flow methods (“Lucas/Kanade meets Horn/Schunck”) A. Bruhn, J. Weickert, C. Schnörr,
High accuracy optical flow estimation based on a theory for warping
T. Brox, A. Bruhn, N. Papenberg, J. Weickert,
Real-Time Optic Flow Computation with Variational Methods
A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr,
Towards ultimate motion estimation: Combining highest accuracy with real-time performance. A. Bruhn, J. Weickert, 2005
Bilateral filtering-based optical flow estimation with occlusion detection. J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006