{ Fast Disparity Estimation Using Spatio- temporal Correlation of Disparity Field for Multiview Video Coding Wei Zhu, Xiang Tian, Fan Zhou and Yaowu Chen IEEE TCE 2010

 Introduction  Analysis of Disparity Field in MVC  Proposed Fast Disparity Estimation Algorithm  Experimental Results  Conclusion Outline 2

Fig. 1. MVC coding schemes among views for different sequences. Fig. 1. MVC coding schemes among views for different sequences. (a) For “Ballroom” sequence with general 1D camera setup. (b) For “Akko&Kayo” sequence with 2D-array camera setup. (c) For “Flamenco2" sequence with 2D-cross camera setup. Introduction 3

 Motion Estimation  Temporal motion is dependent on the movement of objects.  Only moving objects have motion displacements, objects in background often have no motion displacement.  Disparity Estimation  Inter-view disparity is dependent on the depth of object and the camera setup.  Objects with no motion displacements may also have large disparity displacements. Introduction 4

Analysis of Disparity Field in MVC 5 [17] X. M. Li, D. B. Zhao, S. W. Ma and W. Gao, “Fast disparity and motion estimation based on correlations for multiview video coding,” IEEE Trans. Consumer Electron., vol. 54, no. 4, pp , Nov object depth camera parameters

 Disparity is related to the depth of objects  Shallow depths  large disparity  Distant depths  small disparity  Objects in MVC have large range of depths, the range of disparity is also large.  DE requires a large search range to find optimal disparity vector(DV). Analysis of Disparity Field in MVC 6

7 Fig. 2. Histogram of horizontal disparity Fig. 2. Histogram of horizontal disparity vectors between view S1 and view S0 for vectors between view S1 and view S0 for “Ballroom” sequence. “Ballroom” sequence.  Size of DV vary a lot:  Most of the DV are in [0, 24], background objects.  Some DV are in [24, 72], foreground objects.  Most of DV are in the positive.  The disparity direction is only determined by the location of views.

 Spatial direction correlation Fig. 3. A field of original disparity vectors for the 123th frame of view S1 in Fig. 3. A field of original disparity vectors for the 123th frame of view S1 in “Ballroom” sequence. “Ballroom” sequence. [14] Y. Kim, J. Kim, and K. Sohn, “Fast disparity and motion estimation for multi-view video [14] Y. Kim, J. Kim, and K. Sohn, “Fast disparity and motion estimation for multi-view video coding,” IEEE Trans. Consumer Electron., vol. 53, no.2, pp , May coding,” IEEE Trans. Consumer Electron., vol. 53, no.2, pp , May Analysis of Disparity Field in MVC 8 DVs are highly correlated with neighboring vectors in spatial direction.[14] Some irregular DVs are obvious different with their neighbors, especially in homogenous regions.

 Temporal direction correlation Fig. 4. Histogram of temporal difference for horizontal disparity vectors Fig. 4. Histogram of temporal difference for horizontal disparity vectors between view S1 and view S0 in “Ballroom” sequence. between view S1 and view S0 in “Ballroom” sequence. Analysis of Disparity Field in MVC 9 DVs in the temporal direction are also highly correlated.

 The characteristics of disparity field in MVC:  The size of DVs could vary a lot because of different depths of objects. Due to the fixed camera setup, most of DVs have a consistent direction with the real disparities.  Some DVs deviate from the real disparities, DVs of previous coded should be filtered to be consistent with real disparities.  Since DVs have a high correlation with neighboring vectors in the temporal and spatial direction, the search center of DVs and preliminary DV can be predicted. Analysis of Disparity Field in MVC 10

 Basic idea: Select the search center by using the spatio-temporal correlation of disparity field, and to predict the search range adaptively according to the temporal variation of disparity field.  Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame.  Part B: Selection of Search Center Based on the Spatial-temporal Correlation of Disparity Field.  Part C: Prediction of Search Range Based on the Temporal Variation of Disparity Field.  Part D: Overall Proposed Algorithm. Proposed Fast Disparity Estimation Algorithm 11

 Basic idea: There is a high correlation of DVs in temporal direction. Use previous coded frames to predict current frame.  Problem: Some noisy DVs are not consistent with their real disparities, these vectors should be eliminated to obtain a smooth disparity field.  Solution: Because noisy DVs have irregular directions, check every DV direction first.  Requirement: GDV is calculated by averaging all DVs of block 16x16 in the previous coded frame, and its direction is used as the reference of the real disparity direction. Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame 12

 Step1: Using GDV, the DV of block 16x16 for each MB is regularized. Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame 13

Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame 14

Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame 15 encoding order of current frame 0.5 co-located (Zero Disparity Vector)

 Basic idea: The disparity vectors are highly correlated in spatio- temporal directions, so the neighboring disparity vectors in spatio- temporal directions are used to determine the search center for the current block.  Select candidates:  The neighboring disparity vectors are selected as candidates of the search center.  Non-anchor frame: 5-9 are selected from the forward and backward temporal reference frame.  Anchor frame: 5-9 are selected from the previous coded anchor frame. Part B: Selection of Search Center Based on the Spatial-temporal Correlation of Disparity Field 16

Part B: Selection of Search Center Based on the Spatial-temporal Correlation of Disparity Field 17

Part B: Selection of Search Center Based on the Spatial-temporal Correlation of Disparity Field 18 It’s the smallest, only a smaller search range is needed.

 Temporal variation(distance) of DV: |CDV – TDV|  CDV: An approximation of the DV in current frame.  TDV: A temporal prediction of the DV in previous coded frame.  Basic idea: the distance is related the DV consistency between current frame and previous coded frame.  Small distance  small search range  Large distance  large search range Part C: Prediction of Search Range Based on the Temporal Variation of Disparity Field. 19

 Search range: Part C: Prediction of Search Range Based on the Temporal Variation of Disparity Field. 20 (larger)  Calculate search range

Part D: Overall Proposed Algorithm 21

 JMVC4.0  Testing Configuration  Only Inter16x16 mode for inter mode  Use first two or three views of the sequences.  View S1 was chosen for proposed algorithm, and disable ME  S0 was chosen as the reference view for Flamenco2(2D-cross), S0 and S2 were chosen for others.  Comparing with full search algorithm and fast search algorithm. Experimental Results 22

 Rate-distortion performance Experimental Results 23

 Performance comparison Experimental Results 24

 In this paper, a fast DE algorithm is proposed to save the computational load of MVC.  Take into account the spatial-temporal correlation and the temporal variation of disparity.  Perform well on all test sequences.  Compare with full search algorithm, achieve an average 96% reduction of computational complexity, while RD performance remain the same.  Compare with fast search algorithm, achieve an average 43% reduction of computational complexity, while RD performance is improved. Conclusion 25