2010/10/13VCLAB, National Tsing Hua University, Taiwan1 Multiview Video Kai-Chao Yang

2010/10/13VCLAB, National Tsing Hua University, Taiwan2 Outline History of Video Coding Standards Definition of Multiview Video Applications of Multiview Video Concept of Stereo Video Multiview TV/Video System  Multiview Content  Capture  Correction  Coding  Display

2010/10/13VCLAB, National Tsing Hua University, Taiwan3 History of Video Coding Standards H.264/MPEG4 AVCH.262/MPEG H.261H.263H.263+H MPEG-1MPEG-4 v2/visual SVCMVC ISO MPEG JVT ITU-T VCEG FRExt

2010/10/13VCLAB, National Tsing Hua University, Taiwan4 Definition of Multiview Video Multiview video  Multiple cameras are used to simultaneously acquire various viewpoints of a scene. Multiview video coding  Encoding of sequences captured simultaneously from multiple cameras using a single video stream.

2010/10/13VCLAB, National Tsing Hua University, Taiwan5 Applications of Multiview Videos Free viewpoint TV  The viewers can experience the free viewpoint navigation within the range covered by the cameras  Three-dimensional TV Immersive teleconference

2010/10/13VCLAB, National Tsing Hua University, Taiwan6 Examples (1/4) Entertainments  3D game, FVV, … Medicine  3D microscope, 3D endoscope, … Education  3D model, 3D classroom, … Surveillance  Nurse, object recognition and tracking, … Simulation  Flight simulation, …

2010/10/13VCLAB, National Tsing Hua University, Taiwan7 Examples (2/4) 3D microscope and 3D endoscope -jerseys-first-3d-endoscopic-surgery-performed-by-ninj- faculty.html

2010/10/13VCLAB, National Tsing Hua University, Taiwan8 Examples (3/4) Pedestrian Tracking Kyungnam Kim and Larry S. Davis, "Multi-Camera Tracking and Segmentation of Occluded People on Ground Plane using Search-Guided Particle Filtering", European Conference on Computer Vision (ECCV), LNCS, 2006.

2010/10/13VCLAB, National Tsing Hua University, Taiwan9 Examples (4/4) Flight Simulation

2010/10/13VCLAB, National Tsing Hua University, Taiwan10 Concept of Stereo Image/Video 3D information: Determination of relative depths in a perceived scene  Stereopsis  Accommodation of the eyeball  Occlusion  Linear perspective  Vertical position  … 2D knowledge

2010/10/13VCLAB, National Tsing Hua University, Taiwan11 Concept of Stereo Video (3D Video) Stereopsis  visual perception leading to the sensation of depth from the two slightly different projections of the world onto the retinas of the two eyes

2010/10/13VCLAB, National Tsing Hua University, Taiwan12 Concept of Stereo Video

2010/10/13VCLAB, National Tsing Hua University, Taiwan13 Multiview TV System ContentDeliveryDisplayEncodingDecoding

2010/10/13VCLAB, National Tsing Hua University, Taiwan14 Multiview Content

2010/10/13VCLAB, National Tsing Hua University, Taiwan15 Multiview Content Camera capture  Stereo cameras  Camera array Computer generated  Objects is defined  Depth is inherent Conversion from 2D video  Detecting objects  Assigning depth  Filling occluded parts

2010/10/13VCLAB, National Tsing Hua University, Taiwan16 Camera Capture Stanford multi-camera array (128 video cameras) Google street view camera (Image only) Panasonic Full-HD 3D camera

2010/10/13VCLAB, National Tsing Hua University, Taiwan17 2D to 3D Usage of 2D depth information

2010/10/13VCLAB, National Tsing Hua University, Taiwan18 Correction Rectification of misalignment Normalization of colors

2010/10/13VCLAB, National Tsing Hua University, Taiwan19 Multiview Video Coding

2010/10/13VCLAB, National Tsing Hua University, Taiwan20 Representation Formats (Full-Resolution) Full-Resolution Stereo and Multiview  Double data rate for stereo videos  N-fold data rate for N-view videos Efficient compression is the key issue  MVC extension of H.264/AVC

2010/10/13VCLAB, National Tsing Hua University, Taiwan21 Representation Formats (Stereo Interleaving) Stereo Interleaving  A multiplex of the two views into a single sequence Spatial multiplexing  The left and right views are sub-sampled and interleaved into a single frame Temporal multiplexing  The left and right views are interleaved as alternating frames Advantages  Compatible with existing codecs and delivery infrastructure Drawbacks  Loss of spatial or temporal resolution R L R L

2010/10/13VCLAB, National Tsing Hua University, Taiwan22 Representation Formats (Depth- based) Depth-based Format  2D video + depth map Advantages  Backward compatibility with the older coding standards  Supporting both stereo and multiview displays  Depth is adjustable Drawbacks  Limited depth range  Occlusions  Other views have to be synthesized 2D view Synthesized view

2010/10/13VCLAB, National Tsing Hua University, Taiwan23 Multiview Video Coding Single view Multi-view  Independent encoding Low coding complexity but also low coding efficiency  Inter-view prediction Exploiting both spatial and temporal redundancy IPBBBBBBP IPBBBBBBP IPBBBBBBP

2010/10/13VCLAB, National Tsing Hua University, Taiwan24 Multiview Video Coding Multiview camera P B B B P B B B I P B B B P B B B B P B B B P B B B P P B B B P B B B I P B B B P B B B B P B B B P B B B P P B B B P B B B I P B B B P B B B B P B B B P B B B P B IBP P PB P P

2010/10/13VCLAB, National Tsing Hua University, Taiwan25 View Generation View 1View 2synthesis

2010/10/13VCLAB, National Tsing Hua University, Taiwan26 View Generation View synthesis  Disparity-based e.g. view1  view3/2  view2 object(0,0)  object(3,0)  object(6,0)  Depth-based 2D + depth map DEPTH MAPS EXTRACTION FROM MULTI-VIEW VIDEOS

2010/10/13VCLAB, National Tsing Hua University, Taiwan27 View Generation

2010/10/13VCLAB, National Tsing Hua University, Taiwan28 Display (3D space) Volumetric video Holographic video

2010/10/13VCLAB, National Tsing Hua University, Taiwan29 Display Free viewpoint video   E E  XA XA

2010/10/13VCLAB, National Tsing Hua University, Taiwan30 Display with 3D Glasses Stereo video with glasses  Left view and right view Passive and active

2010/10/13VCLAB, National Tsing Hua University, Taiwan31 Display with 3D Glasses Anaglyph, polarized, and shutter glasses

2010/10/13VCLAB, National Tsing Hua University, Taiwan32 Display Lenticular panel and Parallax barrier

2010/10/13VCLAB, National Tsing Hua University, Taiwan33 Display Stereo view display without glasses A1 A2 Pixel 2 Pixel 1

2010/10/13VCLAB, National Tsing Hua University, Taiwan34 Display Multiview display without glasses A2 A1 B1 C1 B2 C2 Pixel 2 Pixel 1

2010/10/13VCLAB, National Tsing Hua University, Taiwan35 Conclusions An overview of multiview video from capture to display. Issues  Display Reduction of brightness Reduction of refresh rate Reduction of temporal of spatial resolution  Coding efficiency  Decoding delay  2D  3D  View synthesis  …

2010/10/13VCLAB, National Tsing Hua University, Taiwan36 Reference A. Vetro, “Representation and Coding Formats for Stereo and Multiview Video,” Tech. Rep. TR , April, S. Gaël, “Depth Map Estimation and Use for 3DTV,” Tech. Rep. n  0379, February, Y.-S. Ho and K.-J. Oh, “Overview of Multi-view Video Coding,” IWSSIP, 許精益 and 黃乙白 “3D 立體顯示技術之發展與研 究,” 光學工程第 98 期, 96 年 6 月.