Communication & Multimedia C. -Y. Tsai 2005/12/15 1 Vidwav Wavelet Video Coding Specifications Student: Chia-Yang Tsai Advisor: Prof. Hsueh-Ming Hang Institute of Electronics, NCTU

Communication & Multimedia C. -Y. Tsai 2005/12/15 2 Outline Framework Framework Main modules Main modules Motion Motion Temporal transform Temporal transform Spatial transform Spatial transform Entropy coding Entropy coding Bitstream formation Bitstream formation Additional modules Additional modules Base layer Base layer In-band coding In-band coding Wavelet ringing reduction Wavelet ringing reduction Conclusions Conclusions

Communication & Multimedia C. -Y. Tsai 2005/12/15 3 References Microsoft Research Asia + ENST + INRIA, “ Vidwav Wavelet Video Coding Specifications ”, ISO/IEC JTC1/SC29/WG11, M12339, Poznan, July

Communication & Multimedia C. -Y. Tsai 2005/12/15 4 Framework

Communication & Multimedia C. -Y. Tsai 2005/12/15 5 T+2D (Pre-spatial decomposition is void)

Communication & Multimedia C. -Y. Tsai 2005/12/15 6 2D+t

Communication & Multimedia C. -Y. Tsai 2005/12/15 7 Motion ME/MC Modes/MV coding

Communication & Multimedia C. -Y. Tsai 2005/12/15 8 ME/MC Adopted from H.264/AVC. Adopted from H.264/AVC. Performed by minimizing the Lagrangian cost function Performed by minimizing the Lagrangian cost function J= D + λR J= D + λR

Communication & Multimedia C. -Y. Tsai 2005/12/15 9 MV Coding Median prediction Median prediction For squre MC block size For squre MC block size 16x16, 8x8, 4x4 16x16, 8x8, 4x4 (ABC) or (ABD) (ABC) or (ABD) Directional segmentation prediction Directional segmentation prediction If predictor is not available, use median prediction If predictor is not available, use median prediction 16x8 8x16 8x44x8

Communication & Multimedia C. -Y. Tsai 2005/12/15 10 Temporal Transform Motion aligned temporal filtering Prediction step Update step

Communication & Multimedia C. -Y. Tsai 2005/12/15 11 Motion Aligned Temporal Filtering Lifting structure Lifting structure MAP (motion aligned prediction) MAP (motion aligned prediction) MAU (motion aligned update) MAU (motion aligned update) SplitMAPMAUMAPMAUMerge

Communication & Multimedia C. -Y. Tsai 2005/12/15 12 Motion Aligned Temporal Filtering (Cont.) Temporal lifting steps of 5/3 biorthogonal wavelet Temporal lifting steps of 5/3 biorthogonal wavelet

Communication & Multimedia C. -Y. Tsai 2005/12/15 13 Prediction Bidirectional connection Bidirectional connection Uni-directional connection Uni-directional connection When left-unidirectional connected When right-unidirectional connected

Communication & Multimedia C. -Y. Tsai 2005/12/15 14 Prediction (Cont.) OBMC OBMC

Communication & Multimedia C. -Y. Tsai 2005/12/15 15 Update Generated from H-frames through MC Generated from H-frames through MC Clipping Clipping

Communication & Multimedia C. -Y. Tsai 2005/12/15 16 Spatial Transform

Communication & Multimedia C. -Y. Tsai 2005/12/15 17 Structure Decomposition structure Decomposition structure Description Description S<>S<>E<>E<>E<>E<>E<>E<>E<> S<> S<> S<> E<> E<> E<> E<>E<> E<> E<> S<> E<> E<> E<> E<>

Communication & Multimedia C. -Y. Tsai 2005/12/15 18 Entropy Coding 3D-EBCOT Bitstream

Communication & Multimedia C. -Y. Tsai 2005/12/15 19 EBCOT Embedded Block Coding Embedded Block Coding Code block Fractional bitplanes Bitplanes   coding pass

Communication & Multimedia C. -Y. Tsai 2005/12/15 20 Coding Pass Operation Previous bit-plane Current bit-plane Insignificant sample Significant sample In previous bit-plane Significant sample In current bit-plane Significant Propagation Pass Magnitude Refinement Pass Normalization Pass ZC ZC ZC SC MR ZC SC

Communication & Multimedia C. -Y. Tsai 2005/12/15 21 Bitstream N of total bitplanes in the block N of total bitplanes in the block The coded bitstream of the block, which consists of 3N-2 segments, each segment corresponding to the output bitstream of one coding pass. The coded bitstream of the block, which consists of 3N-2 segments, each segment corresponding to the output bitstream of one coding pass. The length of each bitstream segment and/or the position of end-point of each coding pass. The length of each bitstream segment and/or the position of end-point of each coding pass. The R-D slope information at the end of each coding pass. The R-D slope information at the end of each coding pass.

Communication & Multimedia C. -Y. Tsai 2005/12/15 22 Bitstream Formation Syntax Bitstream selection

Communication & Multimedia C. -Y. Tsai 2005/12/15 23 Syntax Global Global GOP GOP Group of picture for MCTF Group of picture for MCTF Layer Layer For quick bitstream truncation For quick bitstream truncation Packet Packet One component (Y, U, or V) of one temporal sunbbad One component (Y, U, or V) of one temporal sunbbad Subband Subband 3D subband 3D subband Block Block Block of 3D EBCOT Block of 3D EBCOT Pass Pass Coding pass Coding pass

Communication & Multimedia C. -Y. Tsai 2005/12/15 24 Bitstream Selection Optimized Truncation Optimized Truncation RD slope Block n Block n+1 iteration m iteration m-1 RnRn+1 ΣRi <= bitrate constrain Coding pass index

Communication & Multimedia C. -Y. Tsai 2005/12/15 25 Base Layer

Communication & Multimedia C. -Y. Tsai 2005/12/15 26 Embedded Base Layer Codec

Communication & Multimedia C. -Y. Tsai 2005/12/15 27 Motion Information Prediction BASESKIP mode BASESKIP mode Use base-layer motion vectors as motion predictors Use base-layer motion vectors as motion predictors Macroblock partitions mode map rule to generate the predictors for higher spatial resolution

Communication & Multimedia C. -Y. Tsai 2005/12/15 28 Hierarchical B-Pictures Provide extra- temporal resolutions Provide extra- temporal resolutions

Communication & Multimedia C. -Y. Tsai 2005/12/15 29 In-Band Coding Leaky motion compensation Mode-based temporal filtering

Communication & Multimedia C. -Y. Tsai 2005/12/15 30 In-Band MCTF The forming of different quality reference of LL The forming of different quality reference of LL Low quality reference as IP_DIR Low quality reference as IP_DIR High quality reference as IP_LBS High quality reference as IP_LBS

Communication & Multimedia C. -Y. Tsai 2005/12/15 31 In-Band MCTF Leaky motion compensation Leaky motion compensation leaky factor leaky factor Attenuate the prediction based on the unknown information at the decoder Attenuate the prediction based on the unknown information at the decoder make a good trade-off between drifting errors and coding efficiency make a good trade-off between drifting errors and coding efficiency

Communication & Multimedia C. -Y. Tsai 2005/12/15 32 In-Band MCTF Mode-based temporal filtering Mode-based temporal filtering Mode I: Low quality reference Mode I: Low quality reference Mode 2: High quality reference Mode 2: High quality reference Mode is selected by RD cost Mode is selected by RD cost

Communication & Multimedia C. -Y. Tsai 2005/12/15 33 Wavelet Ringing Reduction

Communication & Multimedia C. -Y. Tsai 2005/12/15 34 Description Purpose Purpose Reduce artifact after EDWT Reduce artifact after EDWT Similar like the de-blocking filter in DCT based coding Similar like the de-blocking filter in DCT based coding After experiments, After experiments,

Communication & Multimedia C. -Y. Tsai 2005/12/15 35 Conclusions

Communication & Multimedia C. -Y. Tsai 2005/12/15 36 Wavelet Based SVC Advantages Advantages Nature for multi-resolution scalability Nature for multi-resolution scalability Open-loop prediction structure Open-loop prediction structure Provides elegant SNR scalability without impairing full exploitation of spatial-temporal correlation Provides elegant SNR scalability without impairing full exploitation of spatial-temporal correlation Simplifies the R-D model of the bitstreams. Simplifies the R-D model of the bitstreams. Facilitates the bitstream truncation Facilitates the bitstream truncation each subband is independent with other subbands each subband is independent with other subbands

Communication & Multimedia C. -Y. Tsai 2005/12/15 37 Wavelet Based SVC Disadvantages Disadvantages Decomposition modes (coding modes) selection Decomposition modes (coding modes) selection Texture & side information trade off Texture & side information trade off Intra-prediction Intra-prediction Badly-matched blocks Badly-matched blocks Downsampling filter problems Downsampling filter problems

Communication & Multimedia C. -Y. Tsai 2005/12/15 38 Thanks for your attention! Any questions?