1. Overview of Multi-view Video Coding Yo-Sung Ho; Kwan-Jung Oh; Systems, Signals and Image Processing, 2007 and 6th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services. 14th International Workshop on Systems, Signals and Image Processing, 2007 and 6th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services. 14th International Workshop on

2. Outline Introduction Applications of Multi-view Video Requirements for Multi-view Video Coding Test Data Sets and Test Conditions Joint Multi-view Video Model (JMVM) Conclusion

3. Introduction

4. Multimedia Demands holography two-view stereoscopic system with special glasses multi-view video Multi-view Video: FVV, FVT, 3DTV What is multi-view video? Why we need multi-view video coding (MVC)? MVC has been studied in the past. MVP, MCP, DCP, MPEG4 MAC, H.263/4

5. Introduction MPEG and ad hoc group on 3DAV 4 exploration experiments on 3DAV EE1 on omni-directional video EE2 on FTV EE3 on coding of stereoscopic video using multiple auxiliary components (MAC) EE4 on depth/disparity coding for 3DTV and intermediate view interpolation Companies have interests for FTV & 3DTV JVT = MVC group of MPEG + VCEG JMVM as an extension of the H.264/MPEG4 AVC

6. Application of MVC

7. Free Viewpoint Television (FTV) Three-dimensional TV (3DTV) Immersive Teleconference

8. FTV What is FTV (Free Viewpoint Television)? Application of FTV: Entertainment Education Sightseeing Surveillance Archive

9. FTV

10. 3DTV What is 3DTV? Interaction may not be required To broadcast on 3DTV

11. 3DTV

12. Capture by various types of multiple cameras 1D parallel 2D parallel 1D arc …etc. Intermediate view reconstruction (IVR)

13. Immersive Teleconference What is immersive teleconference? Interaction

14. Requirements for Multi-view Video Coding

15. Requirements for multi-view video coding: Compression related requirements System support related requirements

16. Compression Related Requirements Compression efficiency View scalability Free viewpoint scalability Spatial/Temporal/SNR scalability Backward compatibility Resource consumption Low delay Robustness

17. Compression Related Requirements Resolution, bit depth, chroma sampling format Picture quality among views Temporal random access View random access Spatial random access Resource management Parallel processing

18. System Support Related Requirements Synchronization View generation Non-planar imaging and display systems Camera parameters

19. Test Data Sets and Test Conditions

22. Joint Multi-view Video Model

24. Random Access GGOP contains frames. For accessing any frame within a GGOP, we have to decode maximum number of frames. b4(S5/T7), following 18 referencing frames.

25. Time-first coding order

26. Encoder Complexity Minimum decoded picture buffer (DPB) EX: GOP_length=16, number_of_views=8, the DPB size = 42 MVC codec will have the same coding delay as single view video coding since time-first coding is mandated.

27. GOP structures and view prediction structure

30. Illumination compensation ICA MC: illumination change-adaptive motion compensation Macroblocks(MB) mode in h.264/MPEG-4 AVC: Inter 16*16 mode, Direct 16*16 mode (include B_Skip), and P_Skip mode DVIC: difference value of illumination change ICA ME: illumination change-adaptive motion estimation

32. SAD calculation for the motion estimation of S*T blocks: In order to compensate the illumination change

33. 1 Illumination compensated residual signal

34. Other Technical Issues View-temporal prediction structure Single video v.s. Multi-view video Three main coding structures Encode multiple video sequences separately Utilizes inter-view correlation only Utilizes both temporal and inter-view correlation View interpolation prediction Decoder side disparity estimation Computing depth at encoder side and transmitting this to the decoder Motion/Disparity vector coding Highly correlated each other

35. Conclusion

36. The multi-view video includes multi-viewpoint video sequence captured by several cameras at the same time. Compress multi-view video efficiently MPEG and JVT are leading the standardization of MVC.