1. Professor Horst Cerjak, 19.12.2005 1 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG A Duality Based Approach for Realtime TV-L 1 Optical Flow Christopher Zach 1, Thomas Pock 2, and Horst Bischof 2 1 VRVis Research Center, Graz 2 Institute for Computer Graphics and Vision, TU Graz E-mail: {zach, pock, bischof}@icg.tugraz.at

2. Professor Horst Cerjak, 19.12.2005 2 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG time Motivation Discontinuity preserving regularization of the flow field Robustness to occulsions Handle large displacements Realtime (> 30 fps) for large images (512x512)

3. Professor Horst Cerjak, 19.12.2005 3 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Outline (I) Variational Optical Flow (II) TV-L 1 optical flow (III) Duality Based Approach (IV) Acceleration using the GPU (V) Performance Evaluation (VI) Conclusion & Demo

4. Professor Horst Cerjak, 19.12.2005 4 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Optical Flow Optical Flow (OF) is a major task of biological and artificial visual systems Relates the motion of pixel intensities between consecutive image frames Optical Flow Constraint: Gives only the normal flow No OFC in untextured areas u1u1 u2u2 u

5. Professor Horst Cerjak, 19.12.2005 5 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Variational Optical Flow First studied by Horn and Schunck in 1981 [1] Quadratic regularization does not allow for discontinuities and occlusions Modifying the Horn and Schunck functional was pioneered by Black and Rangarajan [2] [1] B.K. Horn and B.G. Schunck. Determinig Optical Flow. Artificial Intelligence, 1981 [2] M.J. Black and P. Rangarajan. On the Unification of Line Processes, Outlier Rejection and Robust Statistics with Applications in Early Vision, IJCV, 1996

6. Professor Horst Cerjak, 19.12.2005 6 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG TV-L 1 Optical Flow We use a robust variant of the Horn-Schunck formulation –Total Variation (TV) of Rudin Osher and Fatemi (ROF) [3] –L 1 penalization of the OF constraint TV-L 1 has been used in many approaches Allows for discontinuities in the flow field and outliers in the optical flow constraint Sophisticated optimization techniques are needed This is the major goal of this paper [3] L. Rudin and S. Osher and E. Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms, Physica D, 1992

7. Professor Horst Cerjak, 19.12.2005 7 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG An Approximative Formulation E θ E as Θ 0 Main difficulty is induced by the TV term –> ROF model [3]Simple pointwise optimization problem -> Thresholding [3] L. Rudin and S. Osher and E. Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms, Physica D, 1992

8. Professor Horst Cerjak, 19.12.2005 8 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Primal Formulation Study of the ROF model Primal Euler Lagrange equations Degenerated if gradient vanishes Simple solution: Replace by Disadvantage: Large ε smoothes edges!

9. Professor Horst Cerjak, 19.12.2005 9 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Dual Formulation Studied by Chan [4], and later by Chambolle [5] p One arrives at two new equations [4] T. Chan and G. Golub and P. Mulet, A Nonlinear Primal Dual Method for TV-based Image Restoration, 1999 [5] A. Chambolle, An Algorithm for Total Variation Minimization and Applications, 2004 Advantage: No regularization is needed! Dual Euler Lagrange Equations |p| 1

10. Professor Horst Cerjak, 19.12.2005 10 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Primal vs. Dual Convergence of the Primal and Dual formulation –Primal: fixed-point scheme of Vogel & Oman [6] –Dual: fixed-point scheme of Chambolle [5] ε=10 -15 ε=10 -1 ε=1 ε=10 E ROF iterations [5] A. Chambolle, An Algorithm for Total Variation Minimization and Applications, 2004 [6] C. R. Vogel and M. E. Oman. Iterative Methods For Total Variation Denoising. 1996

11. Professor Horst Cerjak, 19.12.2005 11 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Final Algorithm 1.Fix v, minimize wrt. u (Chambolles algorithm) 2.Fix u, minimize wrt. v (Thresholding) 3.Goto 1 until convergence Energy minimization is embedded into a coarse-to- fine approach to handle large displacements Solved via alternating optimization

12. Professor Horst Cerjak, 19.12.2005 12 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Implementation on Graphics Hardware Particularly well suited to compute variational methods –High degree of parallelism –High performance processing units All features can be accessed via C-like languages Performance of graphics cards is steadily increasing G92 Nov 2007

13. Professor Horst Cerjak, 19.12.2005 13 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Performance Evaluation Image resolution50 Iterations100 Iterations200 Iterations Graphics card7900GTX8800GTX7900GTX8800GTX7900GTX8800GTX 128x12895125.957.665.1230.932.82 256x25634.183.1117.543.298.921.85 512x5129.345.074.722.932.311.69 Error evaluation on the well known Yosemite without clouds sequence HS [1]TV-L 1 (50 it.) Nir et al. [7] AAE32.43°2.85°0.85° Frames per second [1] B.K. Horn and B.G. Schunck. Determinig Optical Flow. Artificial Intelligence, 1981 [7] T. Nir and A.M. Bruckstein and R. Kimmel, Over-Parameterized Variational Optical Flow, IJCV 2007

14. Professor Horst Cerjak, 19.12.2005 14 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Conclusion & Future work We have developed a duality based algorithm for TV- L 1 optical flow computation We have implemented this algorithm on state-of-the- art graphics hardware In summary, we obtained an optical flow algorithm having a realtime performance of ~45 fps for 512x512 images Implementation in CUDA should give an additional speedup More sophisticated data terms for illumination changes Multigrid techniques for the dual formulation

15. Professor Horst Cerjak, 19.12.2005 15 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Demo

16. Professor Horst Cerjak, 19.12.2005 16 Thomas Pock A Duality Based Approach for Realtime TV-L 1 Optical Flow ICG Solution of the ROF model Compute the minimizer of ROF model –Solution of a huge sytem of non-linear equations –Leads to iterative algorithms Primal formulation: –Fixed-point scheme of Vogel and Oman [6] Dual formulation –Fixed-point scheme of Chambolle [5] [5] A. Chambolle, An Algorithm for Total Variation Minimization and Applications, 2004 [6] C. R. Vogel and M. E. Oman. Iterative Methods For Total Variation Denoising. 1996