1. Video: savior, or “more of the same”? 16-721: Learning-Based Methods in Vision A. Efros, CMU, Spring 2009 © A.A. Efros

2. Why do video? Disambiguation: what is difficult in a single image may become easier in a sequence Improve segmentation spatial reasoning, parallax, occlusions Improve object recognition (e.g. multiview) etc. Understanding temporal phenomena actions, events, activities, object interactions, affordances, etc.

3. Disambiguation Jackson Pollock Number 21 (detail)

4. The (alleged) benefits of video Video as space-time volume Object correspondence via tracking “Background subtraction”, time-lapse data 3D from parallax, occlusion reasoning More data (never hurts, right?)

5. Space-time volume x y t

6. Sliding XYT windows

7. 3D volume segmentation C. Fowlkes, S. Belongie, F. Chung, J. Malik. "Spectral Grouping Using The Nyström Method", TPAMI 26 (2)TPAMI Erdos # = 1

8. Problem with space-time volumes Time t is hugely undersampled compared to x,y

9. Tracking: making t special Optical Flow Feature Tracking (Lucas-Kanade) CHALLENGE: Find me an object tracking paper with more than 15 seconds of resulting video…Tracking eventually fails!

10. Tracking by repeated recognition

11. Hybrid approach Ramanan, Forsyth, Zisserman, 2004

12. Build Model, then Detect Ramanan, Forsyth, Zisserman, 2004

13. Very impressive result http://www.ics.uci.edu/~dramanan/videos/kwanPostSmoothGreen_divx.avi

14. Stationary Camera assumption What can we do with this?

15. Background Subtraction - =

16. A largely unsolved problem… Estimated background Difference Image Thresholded Foreground on blue One video frame

17. Even when crazy stuff happens…

18. …averaging can often handle it

19. Scene-specific Motion Priors Robert Pless er al

20. Sometimes works for moving cameras Irani and Anandan

21. Time-lapse data (Webcams) Claude Monet, Haystacks studies

22. “Smoke” (1996), the “photo album scene” http://www.youtube.com/watch?v=MpCpsExvD4Q&feature=channel_page

23. Removing Shadows (Weiss, 2001) How does one detect (subtract away) shadows?

24. Averaging Derivatives

25. Recovering Shadows

26. Compositing with Shadows

27. Motion for parallax, occlusions Stein, Hoiem, et alHoiem, Stein, et al Parallax tricky for forward movement

28. Lots of Data is good! YouTube-style Data … Flickr Data

29. Temporal Phenomena So, what’s the big deal about motion/video? Probably, temporal phenomena: Actions Activities Functional categories Object interactions Affordances (probably as priors) Etc.