D EPTH I MAGE -B ASED T EMPORAL E RROR C ONCEALMENT FOR 3-D V IDEO T RANSMISSION Yunqiang Liu, Jin Wang, and Huanhuan Zhang IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 20, NO. 4, APRIL 2010 Professor: Jar - Ferr Yang Presenter: Jen - Hung Yeh

O UTLINE Introduction Depth Image-Based Temporal Error Concealment A. Error Concealment for Depth Stream B. Error Concealment for 2-D Video Stream Homogeneous MB Boundary MB Simulation Results Conclusion

I NTRODUCTION (1/2) 3-D television (3DTV) system has recently received increasing attention and is believed to be the next logical development. In DIBR-based 3-D video application, it requires transmitting 2-D video and its corresponding depth information. In the video transmission over unreliable channels such as wireless or internet, the transmission errors, such as packet losses and bit errors, are inevitable.

I NTRODUCTION (2/2) Therefore, there is a need for good error concealment (EC) algorithms to compensate the large drop in received video quality. It explores the correlation between the 2-D video stream and the depth stream and proposes a novel temporal error concealment approach by taking advantage of the depth information.

D EPTH I MAGE -B ASED T EMPORAL E RROR C ONCEALMENT A. Error Concealment for Depth Stream B. Error Concealment for 2-D Video Stream

E RROR C ONCEALMENT FOR D EPTH S TREAM (1/2) The MV from the corresponding 2-D frame can be taken as the recovered MV for the corrupted MB in the depth stream. In case of the error occurring at the same location in the depth stream and the video stream, the DMVE algorithm is performed in the depth sequence to recover the lost MV.

E RROR C ONCEALMENT FOR D EPTH S TREAM (2/2) The depth map does not contain any texture information; moreover, it cannot distinguish the different objects at the same distance with the camera, even if they have relative motion. It is not appropriate to take directly the MV of depth map as the recovered MV for 2-D video.

E RROR C ONCEALMENT FOR 2-D V IDEO S TREAM In the following steps, we will only consider the situation that the error occurs in the 2-D video. Belong to two categories & The difference between their average depth values>T1 8-category histogram for the depth value The lost MBs YESNO Boundary MBHomogeneous MB

<T2 H OMOGENEOUS MB Find MV candidates of the MBs around the The MV of the corresponding MB in depth map Filter these MVs candidate The MV with minimum discard YES NO MVi The referenced depth block should coincide with that of the.

H OMOGENEOUS MB (SAD represents sum of absolute differences) is calculated to measure the similarity between the two depth MBs For 3-D video, we introduce a new matching criterion with consideration of the corresponding depth, which is defined as

B OUNDARY MB It concludes four steps as follows: Step 1: MB Segmentation: Step 2: MV Candidates’ Initialization: Step 3: MV Selection: Step 4: Motion Compensation:

S TEP 1: MB S EGMENTATION : The lost MB The pixels with the large depth value Background Foreground We take the foreground part as an example to describe the approach. YESNO

S TEP 2: MV C ANDIDATES ’ I NITIALIZATION : If the depth value of the grid is close to the average depth of the foreground area, both the MV of the grid in 2-D video and depth map are chosen as the MV candidate for the EC of foreground area. And the collocated MV in the reference frame is also taken as the MV candidate. Background Foreground The neighbors of the foreground area Strong correlation with the lost foreground part

Similar with the recovery of homogeneous MB, the matching criterion considers both the neighbor information and the corresponding depth. The matching criterion is defined as: S TEP 3: MV S ELECTION :

measures the temporal correlation of the 2-D video, which is defined as follows: measures the similarity between the current depth MB and its referenced MB, which is defined as: S TEP 3: MV S ELECTION :

S TEP 4: M OTION C OMPENSATION : The foreground area in the lost MB is recovered using the corresponding area in the reference frame according the recovered MV. The background area is recovered with a similar process.

S IMULATION R ESULTS (1/4) depth map 2-D video

S IMULATION R ESULTS (2/4) Fig. 2. Comparison of PSNR for each frame with 5% packet loss. (a) Little Girl. (b) Mobile Phone. (c) Orbi.

S IMULATION R ESULTS (3/4) Homogeneous MB Boundary MB

S IMULATION R ESULTS (4/4) Homogeneous MB Boundary MB

C ONCLUSION A depth image-based error concealment approach exploits the correlations between the 2-D video and its corresponding depth map to recover the lost blocks. This strategy makes it preserve the temporal and spatial consistency on both 2-D video and depth information.