Applying 3-D Methods to Video for Compression Salih Burak Gokturk Anne Margot Fernandez Aaron March 13, 2002 EE 392J Project Presentation

OUTLINE MOTIVATION PREVIOUS WORK ALGORITHMS RESULTS CONCLUSION

BACKGROUND Most common video compression scheme Blockwise transform (i.e. 2D-DCT) for spatial compression Blockwise Motion Estimation and Compensation for temporal redundancy Used in MPEG Motion Compensated Predictive Coding is better than Image based coding since it uses a good prediction from the previous frame.

OBJECTIVE How? Use 3-D Transformations and Methods. Exploit Temporal Redundancies MORE! Why?

3-D Transform Methods Good for ideal case – smooth motion, global motion Bad for non-uniform motion Need large number of frame stores

PREVIOUS WORK 3-D DCT for Video –Bauer and Sayood, Chan and Sui, Servais and de Jager 3-D Wavelets for Video – Pearlman et al., Taubman et al., Motion Compensated Coding

ALGORITHMS IN OUR EXPERIMENTS 1. 2D DCT 2. 3D DCT 3. (Block) Motion Compensated Coding 4. 3D (Volume) Motion Compensated Coding 5. Other Transformation as a prediction scheme (PCA, Wavelet)

2-D versus 3-D DCT 2-D Blocks versus 3-D Blocks 2-D DCT versus 3-D DCT When would 3-D DCT not work? Markov-1 chain P(X n =X n-1 ) = p X n uniform [0, 255]

3D Volume Motion Compensated Coding Current stack of frames Previous stack of frames Divide stack into volume blocks Find 3D motion vectors For each volume block… Calculate residual 3D DCT on residual Q depth

3D Volume Motion Compensated Coding Finding the motion vectors –Full Search –Measurement criterion MAD standard deviation of residual When it does not work –No smooth motion (acceleration present) –Cases when 2D MC does not work (occlusion, illumination, non-uniform motion) –Block size (depth) along time axis is too small or too big Centroid Effect : Center of the block usually has a better prediction than edges of the block.

Initial Prediction Results 2D MC Prediction3D MC Prediction

3D Principal Component Analysis Current stack of frames Divide each stack into volume blocks Find PCA Coefs For each volume block… Calculate residual 3D DCT on residual Q 1 – Apply PCA on a large collection of blocks 2 – Each Block is represented by a linear combination of principal components: = 11 + 22 + 33 + … = A = U  V T [ ] … U= [ ] …

Experimental Setup Techniques 1.2D DCT by 8x8 Blocks 2.3D DCT by 8x8x8 Blocks 3.Block Motion Compensated Coding (8x8 and 16x16 followed by 2-D DCT) 4.Volume Motion Compensated Coding (8x8x4, 8x8x6 and 8x8x8) followed by 3-D DCT. 5.Principal Component Analysis (use principal components of 8x8x8 blocks as prediction followed by 3-D DCT) Bit Rate vs PSNR Comparison DCT coefficients were quantized using a uniform quantizer PSNR was computed over the same support region. Bit rate was calculated by finding the theoretical entropy of the quantized DCT coefficients and motion vectors.

Miss Am

2D-DCT 2D-MC3D-DCT 3D-PCA3D-MC 0.14 bit/pixel

Salesman – 2D-MC vs 3D-MC 2D-MC 3D-MC 0.11 bit/pixel

Salesman – 3D-DCT vs 3D-MC 3D-DCT 3D-MC 0.16 bit/pixel

Foreman 2D-MC 3D-DCT 3D-MC 0.56 bit/pixel

Bus

CONCLUSION There is redundancy across temporal domain. There is a lot of room for improvement if we use 3-D compression methods! 2-D motion compensated coding is good for movies with A LOT of motion. When movements are smooth, 3-D methods will perform better than 2-D motion compensated coding. FUTURE WORK Find Better Residual Coding Schemes for 3D MC. (That deals with centroid effect better) Other 3D Methods ?