1. Light Field Video Stabilization ICCV 2009, Kyoto Presentation for CS 534: Computational Photography Friday, April 22, 2011 Brandon M. Smith Li Zhang University of Wisconsin, Madison Hailin Jin Aseem Agarwala Adobe Systems Incorporated

2. Motivation Feng Liu et al., SIGGRAPH 2009 © 2011 Brandon M. Smith

3. Video Stabilization: Professional Solutions Use special hardware to avoid camera shake steadicamcamera dollycamera crane Expensive Cumbersome © 2011 Brandon M. Smith

4. Video Stabilization: Software Solutions 2D-transformation, software based methods Burt & Anandan, Image Stabilization by Registration to a Reference Mosaic. DARPA Image Understanding Workshop, 1994 Hansen et al., Real-Time Scene Stabilization and Mosaic Construction. DARPA Image Understanding Workshop, 1994 Lee et al., Video Stabilization Using Robust Feature Trajectories. ICCV 2009 Distant, planar scenes Rotational camera motion … © 2011 Brandon M. Smith

5. Video Stabilization: Very Simple Algorithm © 2011 Brandon M. Smith Time

6. Video Stabilization: Camera Solutions sensor stabilization Vibration Reduction (VR) optical stabilization Limited DOF Small baseline © 2011 Brandon M. Smith

7. Video Stabilization: State-of-the-Art Demos taken from Feng Liu et al., ACM Transactions on Graphics 2011 http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/index.htm

8. Video Stabilization: State-of-the-Art Demos taken from Feng Liu et al., ACM Transactions on Graphics 2011 http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/index.htm Works well unless Background has few“trackable” visual features - demodemo There are large dynamic targets - demodemo Too much rolling shutter and motion blur – demo 1, demo 2demo 1demo 2 Parallax is too significant

9. Video Stabilization Challenges Nearby dynamic targets Few “trackable” background visual features Violent shake © 2011 Brandon M. Smith

10. New Approach Viewplus Profusion 25C Panasonic HD Stereo Camcorder LOREO 3D Lens in a Cap 9005 © 2011 Brandon M. Smith

11. New Approach Viewplus Profusion 25C New-view synthesis [Levoy & Hanrahan SIGGRAPH ‘96, Gortler et al. SIGGRAPH ‘96] Synthetic aperture [Wilburn et al., SIGGRAPH 2005] - demodemo Noise Removal [Zhang et al., CVPR 2009] Existing applications Video Stabilization New application Panasonic HD Stereo Camcorder LOREO 3D Lens in a Cap 9005 © 2011 Brandon M. Smith Left Input Right Input Synthetic Viewpoint

12. Why Does a Camera Array Help? Stabilization as image based rendering [Buehler et al. CVPR 2001] More input views at each time instant Easier to work with dynamic scenes Better handling of parallax Actual Camera Path Synthesize a video along a virtual smooth camera path Virtual Camera Path © 2011 Brandon M. Smith

13. How to Avoid Structure from Motion? Advantage: Do not need to compute 3D input camera path Spacetime optimization: Maximize smoothness of virtual video as function of {R f, t f } f=1…F Actual Camera Path Virtual Camera Path Insight: Only Need Relative Transformation R f, t f © 2011 Brandon M. Smith

14. How to Define the Smoothness of a Video? Original Camera  f=2 F-1  E reg Virtual Camera w f,j {R,t} f=1,…,F   E, Z f-1, Z f, Z f+1 p f-1 p f+1 pfpf pfpf  2 q f,j  q f-1,j prev  q f+1,j next 1    j f || 2 © 2011 Brandon M. Smith

15. Original Camera Salient Features Canny Edge Maps Aside: you can generate these in Matlab: >> img = imread(‘image.png’); >> edgemap = edge(img, ‘canny’); © 2011 Brandon M. Smith

16. Matching Features Canny Edge Maps Optical Flow [Bruhn et al. 2005] © 2011 Brandon M. Smith

17. Algorithm Outline 2.Compute optical flow for each time instant [Bruhn et al. 2005] 1.Compute depth map for each time instant [Smith et al. 2009] 4.Run spacetime optimization to find {R,t} f=1,…,F  f=2 F-1  E reg w f,j {R,t} f=1,…,F   E  2 1    j f || 2 q f,j  q f-1,j prev  q f+1,j next 5.New view synthesis © 2011 Brandon M. Smith

18. New View Synthesis R,t Input Images Depth Maps Synthesized Image Relative Transformation New View Synthesis New View Synthesis © 2011 Brandon M. Smith

19. Results © 2011 Brandon M. Smith http://pages.cs.wisc.edu/~lizhang/projects/lfstable/

20. Summary of Contributions Use an array for stabilization Stabilization without structure from motion Can handle challenging cases: –Nearby, dynamic targets –Large scene depth variation –Violent camera shake © 2011 Brandon M. Smith http://pages.cs.wisc.edu/~lizhang/projects/lfstable/

21. Limitations and Future Work Increase algorithm efficiency Use fewer cameras (two instead of five) Motion deblurring with camera arrays Better handle image periphery problems Evaluate a range of camera baselines © 2011 Brandon M. Smith http://pages.cs.wisc.edu/~lizhang/projects/lfstable/

22. Results and Paper http://pages.cs.wisc.edu/~lizhang/projects/lfstable/