1. Investigation of Motion-Compensated Lifted Wavelet Transforms Information Systems Laboratory Department of Electrical Engineering Stanford University Markus Flierl and Bernd Girod

2. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 2 Outline Motion-compensated wavelet coding scheme Experimental results for temporal Haar and 5/3 wavelets Mathematical model and performance bounds Comparison to predictive coding Can motion-compensated wavelet coding really do better than motion-compensated predictive coding? Why? Can motion-compensated wavelet coding really do better than motion-compensated predictive coding? Why?

3. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 3 Motion-Compensated Wavelet Coder Temporal Decomposition Intraframe Coder DWT/ MC-Lifting Haar and 5/3 wavelet 16x16 block motion compensation ½ pixel accuracy 8x8 DCT coder Run-level entropy coding Same quantizer step-size in all frames 7 6 5 4 3 2 1 0 H H H H LLL LLH LH Original frames

4. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 4 Motion-Compensated Haar Wavelet Update step uses negative motion vector of corresponding prediction step Even frames Odd frames Low High

5. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 5 Motion-Compensated 5/3 Wavelet Update steps uses negative motion vectors of corresponding prediction steps Frame 0 Frame 1 Frame 2 Low High Low

6. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 6 R-D Performance of M.C.Wavelet Coder 0100200300400500600700800 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 R [kbit/s] PSNR Y [dB] 5/3, K=32 Haar, K=32 Haar, K=16 Haar, K=8 Haar, K=2 Mother & Daughter, QCIF, 30 fps +

7. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 7 R-D Performance of M.C. Wavelet Coder 0500100015002000250030003500 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 R [kbit/s] PSNR Y [dB] 5/3, K=32 Haar, K=32 Haar, K=16 Haar, K=8 Haar, K=2 Mobile & Calendar, QCIF, 30 fps

8. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 8 Mathematical Model Can we explain the experimental findings by a mathematical model? Yes! Extend rate-distortion analysis of motion-compensated hybrid coding to motion-compensated subband coding B. Girod, "Efficiency Analysis of Multi-Hypothesis Motion-Compensated Prediction for Video Coding," IEEE Trans. Image Processing, vol. 9, no. 2, pp. 173-183, February 2000. B. Girod, "Motion-Compensating Prediction with Fractional-Pel Accuracy," IEEE Transactions on Communications, vol. 41, no. 4, pp. 604-612, April 1993. B. Girod, "The Efficiency of Motion-compensating Prediction for Hybrid Coding of Video Sequences," IEEE Journal on Selected Areas in Communications, vol. SAC-5, no. 7, pp. 1140-1154, August 1987.

9. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 9 Equivalent Motion-Compensated Haar Transform Assume invertible motion compensation operations Even frames Odd frames Low High

10. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 10 Equivalent Motion-Compensated Haar Transform

11. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 11 Mathematical Model of Motion-Compensated Transform vnoise-free picture kk k-th displacement error nknk k-th noise signal ckck k-th motion-compen- sated signal ykyk k-th transform signal Any input picture can be reference picture

12. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 12 Coding Gain of Motion-Compensated Transform Rate difference for each picture k Difference of rate-distortion functions at high rates Compares m.c. transform coding to independent coding of frames... for the same mean squared reconstruction error... for Gaussian signals Total rate difference

13. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 13 Rate Difference with Negligible Noise Calibration:  = 0.5 log 2 (12  2  ) Integer-pel  =0 Half-pel  =-1 Quarter-pel  =-2

14. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 14 Rate Difference with White Noise Calibration:  = 0.5 log 2 (12  2  ) Integer-pel  =0 Half-pel  =-1 Quarter-pel  =-2 White noise at -30 dB

15. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 15 Comparison to Predictive Coding Predictive coding scheme: Motion-compensated hybrid coder (like MPEG, H.263,... ) 16x16 block motion compensation with half-pel accuracy Previous reference frame only Intra-frame coding with 8x8 DCT and run-length coding Only one I-frame in the beginning of the sequence Same quantizer step-size for all P-frames Same components as motion-compensated wavelet coding scheme

16. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 16 Comparison to Predictive Coding Mother & Daughter, QCIF, 30 fps

17. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 17 Comparison to Predictive Coding Mobile & Calendar, QCIF, 30 fps

18. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 18 Comparison to Predictive Coding Calibration:  = 0.5 log 2 (12  2  ) Integer-pel  =0 Half-pel  =-1 Quarter-pel  =-2 0.5 bits

19. Flierl and Girod: Investigation of MC Lifted Wavelet Transforms April 23, 2003 19 Conclusions Investigated motion-compensated wavelet transform followed by intra-frame coder both experimentally and theoretically Biorthogonal 5/3 wavelet outperforms Haar wavelet Wavelet transform can outperform predictive coding with single reference frame Theory offers insights and some possible explanations Rate can decrease up to 1 bpp per displacement accuracy step Gain by accurate motion compensation is limited by residual noise Motion-compensated transform can outperform predictive coding by up to 0.5 bpp, due to better noise suppression Long GOPs needed for wavelet subband decomposition