1. Contours and Optical Flow: Cues for Capturing Human Motion in Videos Thomas Brox Computer Vision and Pattern Recognition Group University of Bonn Research partially funded by the German Research Foundation (DFG)

2. Thomas Brox University of Bonn 2 Human pose tracking from video 1.Tracking of markers attached to the body + Designed to be easy to track  Reliable and fast tracking –Accuracy limited by number of markers –People may feel uncomfortable 2.Tracking features that naturally appear in the images Patches (e.g. KLT, SIFT, etc.) Contour/Silhouette Optic flow How to extract these features reliably from the images Introduction Segmentation Optic Flow Summary

3. Thomas Brox University of Bonn 3 Contour and optic flow based human tracking Joint work with Bodo Rosenhahn Introduction Segmentation Optic Flow Summary

4. Part I Object Contour Extraction

5. Thomas Brox University of Bonn 5 Object contour extraction Find two regions: object & background Often: Static background  background subtraction Optimality criteria here: –Strong similarity within regions –Small boundary Bayesian approach: Introduction Segmentation Optic Flow Summary

6. Thomas Brox University of Bonn 6 Level set representation of contours (Dervieux-Thomasset 1979, Osher-Sethian 1988 ) Introduce embedding function Contour C represented as zero-level line of Courtesy of Daniel Cremers Introduction Segmentation Optic Flow Summary

7. Thomas Brox University of Bonn 7 Region-based active contours (Chan-Vese 2001, Paragios-Deriche 2002 ) Minimize negative logarithm: Gradient descent: plus update of p 1 and p 2 H(x)H(x) H’(x) Introduction Segmentation Optic Flow Summary

8. Thomas Brox University of Bonn 8 Region statistics 7 channels: –3 color channels (CIELAB) –4 texture channels Channels assumed to be independent Probability densities p ij approximated by Gaussians Introduction Segmentation Optic Flow Summary

9. Thomas Brox University of Bonn 9 Texture Usually modeled by Gabor filters ( Gabor 1946) Includes 1.Magnitude 2.Orientation 3.Scale High redundancy Sparse alternative representation feasible Nonlinear structure tensor (Brox et al. 2006 ) Region based local scale measure (Brox-Weickert 2004 ) Introduction Segmentation Optic Flow Summary

10. Thomas Brox University of Bonn 10 Sparse texture features  Gabor filter bank Sparse representation Introduction Segmentation Optic Flow Summary

11. Thomas Brox University of Bonn 11 Examples for contour extraction Introduction Segmentation Optic Flow Summary

12. Thomas Brox University of Bonn 12 Local region statistics Object and background usually not homogeneous Idea: assume them to be locally homogeneous Probability densities estimated by local Gaussians Introduction Segmentation Optic Flow Summary

13. Thomas Brox University of Bonn 13 Introducing a shape prior Idea: object model can serve as 3-D shape prior  Constrains the segmentation, unwanted solutions discarded Bayesian formula: Pose parameters of model unknown  Two variables: contour and pose Introduction Segmentation Optic Flow Summary

14. Thomas Brox University of Bonn 14 Simultaneously optimize contour and pose: Iterative alternating scheme: –Update contour –Update pose parameters Related works: 2-D shape priors (Leventon et al. 2000, Cremers et al. 2002, Rousson-Paragios 2002 ) Joint optimization shape+pose constraint conventional segmentation part Introduction Segmentation Optic Flow Summary

15. Part II Optic Flow

16. Thomas Brox University of Bonn 16 Optic flow based tracking Image 1 and 2, estimate flow in betweenGiven pose at Image 1 Estimated pose at Image 2 Pose change due to optic flow Introduction Segmentation Optic Flow Summary

17. Thomas Brox University of Bonn 17 Tracking example Introduction Segmentation Optic Flow Summary

18. Thomas Brox University of Bonn 18 How to compute the optic flow? Given: two images I(x,y,t) and I(x,y,t+1) in a sequence Goal: displacement vector field (u,v) between these images Variational approach: (Horn-Schunck 1981 ) Introduction Segmentation Optic Flow Summary

19. Thomas Brox University of Bonn 19 Enhanced model (Brox et al. 2004, Papenberg et al. 2006 ) Robust smoothness term (Cohen 1993, Schnörr 1994 ) Robust data term (Black-Anandan 1996, Mémin-Pérez 1996 ) Spatiotemporal smoothness (Nagel 1990 ) Gradient constancy (Brox et al. 2004 ) Original Horn-Schunck: Final optic flow model: Non-linearized constancy (Nagel-Enkelmann 1986, Alvarez et al. 2000 ) Introduction Segmentation Optic Flow Summary

20. Thomas Brox University of Bonn 20 Horn-Schunck Robust smoothness Impact of each improvement Correct result Robust data termGradient constancyNonlinear constancySpatiotemporal smoothness Introduction Segmentation Optic Flow Summary

21. Thomas Brox University of Bonn 21 Accurate and robust optic flow computation TechniqueAAE Nagel10.22° Uras et al.8.94° Alvarez et al.5.53° Mémin-Pérez4.69° Brox et al. (Noisy)4.49° Bruhn et al.4.17° Brox et al.1.78° Introduction Segmentation Optic Flow Summary

22. Thomas Brox University of Bonn 22 Contour and optic flow based human tracking Joint work with Bodo Rosenhahn Introduction Segmentation Optic Flow Summary

23. Thomas Brox University of Bonn 23 Summary Contours and optic flow can be reliable features for pose tracking Texture, local statistics, and a shape prior are important for general contour based human motion tracking High-end optic flow helps in case of fast motion What’s next? Real-time performance Automatic pose initialization Prior knowledge about joint angle configurations Introduction Segmentation Optic Flow Summary

24. Thomas Brox University of Bonn 24 Outlook Joint work with Bodo Rosenhahn Introduction Segmentation Optic Flow Summary

25. Thomas Brox University of Bonn 25 Backup: nonlinear structure tensor Texture orientation can be measured with the structure tensor (second moment matrix) (Förstner-Gülch 1987, Rao-Schunck 1991, Bigün et al. 1991 ) Gaussian smoothing  nonlinear diffusion Input image Linear structure tensor Nonlinear structure tensor Introduction Segmentation Optic Flow Summary

26. Thomas Brox University of Bonn 26 Backup: region based local scale measure Estimate regions, measure their size Nonlinear diffusion: TV flow (Andreu et al. 2001 ) Tends to yield piecewise constant images  regions Local evolution speed inversely proportional to size of region (Steidl et al. 2004 )  local scale measure Introduction Segmentation Optic Flow Summary Input image Local scale