1. Technische Universität Berlin Communication Systems Group Director: Prof. Thomas Sikora Sebastian Knorr knorr@nue.tu-berlin.de 21/08/2007 Super-Resolution Stereo- and Multi-View Synthesis from Monocular Video Sequences S. Knorr, M. Kunter, and T. Sikora

2. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Outline  Motivation  Overview of the conversion system  Stereo- and multi-view synthesis  Super-resolution approach  Simulation results  Subjective Evaluation  Limitations, conclusion and future work 2

3. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Motivation  Conversion of existing 2D video material automaticaly into 3D (stereoscopic video)  High quality stereoscopic video for 3D devices (e.g. shutter glasses, auto-stereoscopic displays, etc.)  Multi-View video for multi-user 3D displays 3 2D TV-display auto-stereoscopic display shutter-glasses anaglyph

4. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video 4 Overview of the Conversion System

5. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video 5 Stereo- and Multi-view Synthesis Multi-view camera setup (red cameras) original camera path (grey cameras) 3D point cloud current camera (blue camera)

6. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video 6

7. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Principles of Stereo- and Multi-view Synthesis  Define a virtual camera for each frame of the the sequence (at least one for stereo)  Project 3D points into the virtual camera:  Estimate global homography between the virtual view and the closest view of the original camera path 7

8. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Stereo- and Multi-view Synthesis  Estimate global homography between the virtual view and the closest view of the original camera path (e.g. minimizing a cost function of the transfer errors)  Warp complete image into the virtual view 8 original left view virtual right view

9. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Super-Resolution Stereo View Synthesis 99

10. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Super-Resolution Stereo View Results 10

11. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Simulation Results 11 Original view and 8 virtual views for multi-user auto-stereoscopic displays Anaglyph stereo pair of "Statue" Anaglyph stereo pair of "Medusa"

12. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video 12 Simulation Results [Source: BBC-documentation "Planet Earth", Series "Ice Worlds"]

13. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Subjective Evaluation (MOS) 13

14. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Subjective Evaluation (DMOS) 14

15. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video  Scene has to be (almost) static  Restrictions on the camera movement  just forward- or backward camera movement is not allowed  up- and down movement can be handled by transposing the frames by 90°  Virtual cameras too far from the camera path result in large errors when estimating homographies Limitations 15

16. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Conclusion:  the presented algorithm is highly attractive as a tool for user-assisted 2D-3D conversion and 3D production systems  software tool is highly attractive for amateur film-maker Future work:  applying local homographies  extension for dynamic scenes Conclusion and Future Work 16

17. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video Homepage: http://www.nue.tu-berlin.de/ Contact: Sebastian Knorr knorr@nue.tu-berlin.de 0049 - 30 - 314 28501 Thank you for your attention! 17

18. Communication Systems GroupS. Knorr: From 2D- to Stereo- to Multi-View Video  Determine the positions of the virtual stereo cameras  Normalize the coordinates of the cameras  Define positions of the virtual cameras (parallax is a multiple of 64 mm)  Determine projection matrices of the virtual stereo cameras 18 Stereo- and Multi-view Synthesis user interaction!