1. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman 3 Wolfgang Heidrich 1 1 University of British Columbia 2 Saarland University 3 MIT Media Lab Eurographics 2011 – State of the Art Report VI. Multiplexing Time

2. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging2 History – Eadweard Muybridge 1830-1904 http://en.wikipedia.org/wiki/Eadweard_Muybridge

3. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging3 History – Étienne-Jules Marey 1830-1904 www.wikipedia.org

4. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging4 VI.I Time Lapse Photography

5. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging5 BBC Time Lapse – Look It! ND 3.0 filter, f22, 1 minute exposure Long exposures to avoid temporal aliasing

6. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging6 VI.II High-Speed Imaging

7. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging7 High-Speed Cameras Vision Research Phantom Flex (CMOS) 2570 fps at HD resolution Photron FASTCAM SA5 (CMOS) 7500 fps at megapixel resolution one million fps at 64x64 pixels Casio Exilim Series (consumer cam) 1000 fps at reduced resolution Shimadzu HyperVision HPV-2 (CCD) one million fps at 312x260 pixels

8. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging8 Assorted Pixels [Narasimhan & Nayar 05]

9. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging9 Temporal Mosaic with DMD [Bub et al. 10] DMD aligned with CCD in microscope

10. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging10 Non-Destructive Sensor Readout & Pixim Cypress Semiconductor LUPA 3000 3 megapixels, 485 fps www.pixim.com

11. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging11 Coded Rolling Shutter [Gu et al. 10]

12. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging12 Reinterpretable Imager Moving pinhole over time in aperture Capture with light field camera [Agrawal et al. 10]

13. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging13 Bullet Time Effect from ‘The Matrix’

14. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging14 Stanford Multi-Camera Array [Wilburn et al. 04]

15. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging15 Coded Temporal Sampling [Agrawal et al. 10]

16. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging16 High-Speed Illumination – Electronic Strobes Harold ‘Doc’ Edgerton 1903-1990

17. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging17 Temporal Dithering with DLP Illumination [Narasimhan et al. 08]

18. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging18 Coded Strobing Photography [Reddy et al. 11]

19. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging19 Streak Cameras www.hamamatsu.com C5680 $200K

20. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging20 VI.I Motion Deblurring

21. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging21 Motion Deblurring Overview Motion blur is velocity-dependent Can be described as convolution, where –Kernel shape is motion trajectory –Trajectory is modulated by exposure function http://en.wikipedia.org/wiki/Motion_blur

22. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging22 Deconvolution is Still Hard Again – problems: –Camera noise –Spatially varying kernel (velocity-dependent) –Unknown motion trajectory –Ill-posed problem, kernel of box integration function is not invertible (optical cancellation of image frequencies)

23. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging23 Approaches to Improve Motion Deblurring Make PSF invertible  coded exposure Make PSF velocity-invariant  shift- invariant deconvolution Automatize PSF estimation

24. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging24 Flutter Shutter [Raskar et al. 06]

25. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging25 Optimal Motion PSFs Optimality criteria PSF invertibility & estimation [Agrawal & Xu 07]

26. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging26 Motion Invariant Photography Engineer PSF to be motion invariant Only for 1D motion [Levin 08]

27. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging27 Hybrid Cameras Combined high-speed low-quality & low- speed high-quality camera Input images Computed PSF Deblurred Result Ground Truth [Ben-Ezra & Nayar 04]

28. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging28 Motion Blur in Video Coded exposure & super-resolution in successive video frames [Agrawal et al. 09]

29. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging29 Next: Further Light Properties

30. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging30 Flexible Voxels Flexible space-time resolution as post- processing [Gupta et al. 10]

31. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging31 Synthetic Shutter Speed Imaging Combine multiple short exposures to reduce noise Align with optical flow [Telleen 07]

32. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging32 Hybrid Cameras Motion deblurring & super-resolution [Tai et al. 08]

33. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging33 Hybrid Cameras Motion deblurring & depth from two low- resolution high-speed camers [Li et al. 08] Input images Deblurred resultRecovered Depth

34. EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging34 Analysis Analysis of optimal coded, single image deblurring MIP becomes worse when velocities exceed expectations [Agrawal & Raskar 09] Coded Exposure Motion Invariant Photography