Institute of Electronics, National Chiao Tung University Scalable Extension of H.264/AVC Student: Hung-Chih Lin Advisor: Prof. Hsueh-Ming Hang

Institute of Electronics, National Chiao Tung University 2660/03/30hclin2 References [1] Reichel, J., Hanke, K., Popescu, B.: Scalable Video Coding V1.0. ISO/IEC JTC1/SC29/WG11, N6372 (2004) [2] H. Schwarz, D. Marpe, and T. Wiegand, “Scalable Extension of H.264/AVC”, ISO/IEC JTC1/WG11 Doc. M10569/S03, Mar [3] I. Daubechies and W. Sweldens, “Factoring wavelet transforms into lifting steps”, J. Fourier Anal. Appl. 4(3), pp , [4] J. Reichel, H. Schwarz, and M.Wien, "Joint Scalable Video Model JSVM-2," 17th JVT meeting, JVT-Q202, Nice, France. [5] Tabatabai, A., Visharam, Z., Suzuki, T.: Compariosn of MCTF and closed-loop hierarchical B pictures. ISO/IEC JTC/SC29/WG11 and ITU-T SG16 Q.6, JVT-P059 (2005)

Institute of Electronics, National Chiao Tung University 2660/03/30hclin3 Outline Overview MCTF in JSVM Scalability Concepts JSVM Reference Software

Institute of Electronics, National Chiao Tung University 2660/03/30hclin4 Outline Overview –Motivation –Scalable Video Coding MCTF in JSVM Scalability Concepts JSVM Reference Software

Institute of Electronics, National Chiao Tung University 2660/03/30hclin5 Motivation To support clients with diverse capabilities in complexity, bandwidth, power, and display resolution.

Institute of Electronics, National Chiao Tung University 2660/03/30hclin6 Scalable Video Coding Approaches –wavelet-based 2D+t structure t+2D structure –AVC-based Layered coding concept

Institute of Electronics, National Chiao Tung University 2660/03/30hclin7 Approaches A wavelet-based approach with 2D+t structure

Institute of Electronics, National Chiao Tung University 2660/03/30hclin8 Approaches A wavelet-based approach with t+2D structure

Institute of Electronics, National Chiao Tung University 2660/03/30hclin9 Approaches An AVC/H.264-based structure

Institute of Electronics, National Chiao Tung University 2660/03/30hclin10 Scalabilities Temporal –fps Spatial –resolution SNR/Rate –quality schemeTemporalSpatialSNR/Rate wavelet-basedMCTFwavelet transform (multi-resolution) zero-tree coding AVC-basedMCTFLayered codingCABAC (CGS) Bit-plane coding (FGS)

Institute of Electronics, National Chiao Tung University 2660/03/30hclin11 Outline Overview MCTF in JSVM –Why MCTF ? –Base layer structure –Inter layer prediction –Adaptive Prediction/Update Steps –Progressive MCTF Scalability Concepts JSVM Reference Software

Institute of Electronics, National Chiao Tung University 2660/03/30hclin12 Why MCTF? MCTF = Motion-Compensated Temporal Filtering A temporal sub-band coding –2-channel filter bank in temporal direction Performs the wavelet decomposition / reconstruction along the motion trajectory Implementation technique –Lifting scheme (the main reason) : Any bi- orthogonal wavelet filters can be factorized by prediction and update steps

Institute of Electronics, National Chiao Tung University 2660/03/30hclin13 Lifting scheme Attraction –An in-place implementation like FFT. –Easy to build non-linear WT. –Insure PR. –All operations within one lifting step can be done entirely parallel. Computational complexity –~40% of original one (depend on the wavelet filter)

Institute of Electronics, National Chiao Tung University 2660/03/30hclin14 Lifting scheme

Institute of Electronics, National Chiao Tung University 2660/03/30hclin15 Factoring Wavelet Transforms into Lifting Steps 2-channel Filter Bank Bi-orthogonal 2H 0 (z) 2 y[n]x[n] F 0 (z) 2H 1 (z) 2F 1 (z) 2 2 y[n]x[n] 2 2

Institute of Electronics, National Chiao Tung University 2660/03/30hclin16 Factoring Wavelet Transforms into Lifting Steps PR condition Define

Institute of Electronics, National Chiao Tung University 2660/03/30hclin17 Factoring Wavelet Transforms into Lifting Steps Type 1 polyphase representation Define

Institute of Electronics, National Chiao Tung University 2660/03/30hclin18 Factoring Wavelet Transforms into Lifting Steps Noble identities H(z L ) LH(z) L MH(z M ) M

Institute of Electronics, National Chiao Tung University 2660/03/30hclin19 Factoring Wavelet Transforms into Lifting Steps We want and are FIR. By Euclidean algorithm, we can get

Institute of Electronics, National Chiao Tung University 2660/03/30hclin20 Factoring Wavelet Transforms into Lifting Steps

Institute of Electronics, National Chiao Tung University 2660/03/30hclin21 Wavelet filters 2-2 Filter Bank (Haar) 5-3 Filter Bank

Institute of Electronics, National Chiao Tung University 2660/03/30hclin22 Lifting scheme

Institute of Electronics, National Chiao Tung University 2660/03/30hclin23 MCTF HHHH H2H2 H2H2 H3H3 HHH H1H1 H2H2 H2H2 H3H3 L H4H4 15Hz Video Sequence 7.5Hz Video Sequence 30Hz Video Sequence 3.25Hz Video Sequence

Institute of Electronics, National Chiao Tung University 2660/03/30hclin24 MCTF (a) Without M.C. (b) With M.C.

Institute of Electronics, National Chiao Tung University 2660/03/30hclin25 Lifting scheme

Institute of Electronics, National Chiao Tung University 2660/03/30hclin26 Base layer Structure Compatible with AVC Main profile –Dyadic hierarchical B pictures –Only prediction step is performed. (UMCTF)

Institute of Electronics, National Chiao Tung University 2660/03/30hclin27 Base layer Structure Non-dyadic decomposition is available –Temporal scalability

Institute of Electronics, National Chiao Tung University 2660/03/30hclin28 Inter Layer Prediction Remove the redundancy among the different layers –Residues –Motion vectors

Institute of Electronics, National Chiao Tung University 2660/03/30hclin29 Inter Layer Prediction

Institute of Electronics, National Chiao Tung University 2660/03/30hclin30 Adaptive Prediction/Update Steps Goal –Control the encoding delay Method –GOP is partitioned into sub-groups Restrictions : no across the partition boundary –Backward prediction steps –Backward and forward update steps

Institute of Electronics, National Chiao Tung University 2660/03/30hclin31 Adaptive Prediction/Update Steps

Institute of Electronics, National Chiao Tung University 2660/03/30hclin32 Progressive MCTF Prediction steps and update steps are interlaced. Process the pictures in the reverse display order.

Institute of Electronics, National Chiao Tung University 2660/03/30hclin33 Progressive MCTF

Institute of Electronics, National Chiao Tung University 2660/03/30hclin34 Outline Overview OMCTF in JSVM Scalability Concepts –Three Scalabilities –Slice Types –Combined scalability JSVM Reference Software

Institute of Electronics, National Chiao Tung University 2660/03/30hclin35 Temporal Scalability

Institute of Electronics, National Chiao Tung University 2660/03/30hclin36 Spatial Scalability Interpolation filter: {1,-5,20,20,-5,1}

Institute of Electronics, National Chiao Tung University 2660/03/30hclin37 SNR Scalability

Institute of Electronics, National Chiao Tung University 2660/03/30hclin38 Slice Types Slice Type Supported macroblock modes INTRA_4x4INTRA_16x16INTRA_PCMINTRA_BASERESIDUALmotion-compensated modes MX (1) X IXXX PXXXX BXXXX IEXXXX PEXXXXX BEXXXXX EX HXXXX (2) HEXXXXX (2) (1) For M slices, the intra mode is called INTRA and it is not identical to the INTRA_4x4 mode. (2) The residual mode (RESIDUAL) is not indicated by the syntax element mb_type, instead the macroblocks that are coded in residual mode are specified by the corresponding prediction data array.

Institute of Electronics, National Chiao Tung University 2660/03/30hclin39 Slice Types Slice Type Usage MCoding of prediction data arrays I Coding of base-layer (SNR, spatial) representations of low-pass pictures P B IE Coding of enhancement-layer (SNR, spatial) representations of low-pass pictures PE BE E Coding of SNR enhancement-layer representations of high-pass pictures Coding of enhancement-layer (SNR, spatial) representations of low-pass pictures HCoding of base-layer (SNR, spatial) representations of high-pass pictures HECoding of spatial enhancement-layer representations of high-pass pictures

Institute of Electronics, National Chiao Tung University 2660/03/30hclin40 Combined Scalability

Institute of Electronics, National Chiao Tung University 2660/03/30hclin41 Outline Overview OMCTF in JSVM Scalability Concepts JSVM Reference Software –Tools –UMCTF at Decoder

Institute of Electronics, National Chiao Tung University 2660/03/30hclin42 Tools Converter –Spatial domain Upsample –Interpolation FIR filter Downsample –Apply an anti-aliasing FIR filter proir to 2D downsampling –Temporal domain PSNR

Institute of Electronics, National Chiao Tung University 2660/03/30hclin43 UMCTF at Decoder Update step –Improve coding efficiency –Increase significantly complexity of the decoder operation Additional M.C. operations Picture buffer management M.V. needs intensive branch instructions

Institute of Electronics, National Chiao Tung University 2660/03/30hclin44 UMCTF at Decoder UMCTF => update step at decoder side is omitted –The visual quality and PSNR of the decoded video is not degraded –UMCTF → purely predictive structure –Reduce the complexity of decoder by 50%

Institute of Electronics, National Chiao Tung University 2660/03/30hclin45 Normal Mode

Institute of Electronics, National Chiao Tung University 2660/03/30hclin46 High Quality (Qp = 0)

Institute of Electronics, National Chiao Tung University 2660/03/30hclin47 Qp = 24

Institute of Electronics, National Chiao Tung University 2660/03/30hclin48 Thank you !!