Shengkai Lin@CMLAB, NTUCSIE 2004 Ch. 6- H.264/AVC Part I (pp.160~199) Sheng-kai Lin Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Outline Introduction Baseline profile Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004

H.264/AVC Introduction (I) 4:2:0 Define VCL and NAL VCL: Video Coding Layer NAL: Network abstraction Variable block size Multiple reference frame ¼ sample motion compensation Shengkai Lin@CMLAB, NTUCSIE 2004

H.264/AVC Introduction (II) CAVLC CABAC Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Terms (I) Picture (frame) one or more slices Slice group of macroblock (1~ ) minimal inter-dependency between coded slices can limit the error propagation Macroblock Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Terms (II) I slice I macroblock only P slice P, I macroblock B slice B,I macroblock SI slice SP slice Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Terms (III) I macroblock intra prediction from decoded samples in the current slice for entire macroblock for each 4*4 block P macroblock inter prediction using past references 16*16, 16*8, 8*16, 8*8, 4*8, 8*4, 4*4 Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Terms (IV) B macroblock inter prediction using past/ future references list 0 past reference frame buffer list 1 future reference frame buffer RBSP (Raw Byte Sequence Payload) NAL payload Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 H.264/AVC profiles Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 H.264/AVC Encoder Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 H.264/AVC Decoder Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Outline Introduction Baseline profile Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004

Reference Picture Management (I) Reference pictures are stored in decoded picture buffer (DPB) short/long term reference picture A decoded frame may be marked as unused for reference short term picture long term picture Shengkai Lin@CMLAB, NTUCSIE 2004

Reference Picture Management (II) “Sliding Window” memory management Keep #(long_term_pic+ short_term_pic) Remove short term picture if lack of space Adaptive memory control issued by encoder change the type of the ref frame IDR (Instantaneous Decoder Refresh) clear ref buffer I frame Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Outline Introduction Baseline profile Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004

Redundant coded picture Send the duplicated part or all of a coded picture Error resilience Shengkai Lin@CMLAB, NTUCSIE 2004

Arbitrary Slice Order (ASO) The decoding order of the slices could be arbitrary Application example reduce end-end transmission delay in RT app Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Slice Group (I) former called “FMO” Flexible Macroblock Ordering a subset of the macroblocks and may contain one or more slices Application example Error resilience Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Slice Group (II) Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Outline Introduction Baseline profile Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Inter Prediction Variable block size ¼ pixel motion compensation Shengkai Lin@CMLAB, NTUCSIE 2004

Interpolation Samples (I) Generate ½ samples by six tap FIR b, h, m, s FIR para :(1/32, -5/32, 5/8, 5/8, -5/32, 1/32) Generate center ½ sample by FIR j Generate ¼ sample by linear interpolation Shengkai Lin@CMLAB, NTUCSIE 2004

Interpolation Samples (II) Shengkai Lin@CMLAB, NTUCSIE 2004

Motion vector prediction (I) Encoding MV can cost a lot of bits especially small partition sizes Neighboring MV is highly correlated predictable encode the diff MVD and transmit it Shengkai Lin@CMLAB, NTUCSIE 2004

Motion vector prediction (II) Partitions excluding 16 × 8 and 8 × 16, MVp = (MVA+ MVB + MVC) /3 For 16 × 8 partitions, MVp of the upper 16 × 8 = MVB ;MVp for the lower 16 × 8 = MVA Shengkai Lin@CMLAB, NTUCSIE 2004

Motion vector prediction (III) For 8 × 16 partitions, MVp for the left 8 × 16 = MVA ;MVp for the right 8 × 16 = MVC For skipped macroblocks, do as 16 × 16 Inter mode Shengkai Lin@CMLAB, NTUCSIE 2004

Motion vector prediction (IV) Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Outline Introduction Baseline profile Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Intra Prediction For Luma samples 4*4 block 9 prediction modes 16*16 block 4 modes I_PCM transmit the encoded samples w/o pred. & trans Shengkai Lin@CMLAB, NTUCSIE 2004

4*4 luma prediction modes Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 16*16 prediction modes Shengkai Lin@CMLAB, NTUCSIE 2004

8*8 Chroma Prediction Modes Similar to 16*16 luma intra-prediction except the different numbering. DC(mode 0) horizontal (mode 1) vertical(mode 2) plane (mode 3). Shengkai Lin@CMLAB, NTUCSIE 2004

Signaling Intra Prediction Modes (I) The choices of the mode should be signaled to the decoder. might cost lots of bits. Intra modes for neighboring 4 × 4 blocks are often correlated predict the parameters ! The prediction mode for luma coded in Intra-16×16 mode or chroma coded in Intra mode is signalled in the macroblock header and the following method is not used. Shengkai Lin@CMLAB, NTUCSIE 2004

Signaling Intra Prediction Modes (II) B A C If A and B are available, C = min (A,B) else if (neither A nor B are available) C = 2 (DC) else C = available (A,B) Shengkai Lin@CMLAB, NTUCSIE 2004

Signaling Intra Prediction Modes (III) Flag prev_intra4×4_pred_mode 1: most probable prediction mode is used. 0: Flag rem_intra4×4_pred_mode indicates the change of the mode Shengkai Lin@CMLAB, NTUCSIE 2004

Signaling Intra Prediction Modes (IV) If rem_intra4×4_pred_mode < most_probable_mode, the prediction mode = rem intra4×4 pred mode else the prediction mode = (rem_intra4×4_pred_mode+1) Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Outline Introduction Baseline profile Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Deblocking filter Filter 4 vertical/horizontal boundaries of luma Filter 2 vertical/horizontal boundaries of chroma Affect up to 3 samples on the either side. The filter is stronger at places where there is likely to be significant blocking distortion e.g.: such as the boundary of an intra coded macroblock or a boundary between blocks that contain coded coefficients. Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Outline Introduction Baseline profile Reference Picture Management Slice Inter-prediction Intra-prediction Deblocking filter Transform and Quantisation Shengkai Lin@CMLAB, NTUCSIE 2004

Transform and Quantisation 3 transforms DCT-base transform for all 4*4 residual block Hadamard transform for 4*4 luma DC coefficient (in 16*16 intra) Hadamard transform for 2*2 chroma DC coefficient Shengkai Lin@CMLAB, NTUCSIE 2004

DCT-base transform (I) Integer transform Might be zero mismatch The code part of the algo need add and shift only Scale multiplication could be integrated into quantiser. Inverse operation could be down by 16-bit arithmetic. Shengkai Lin@CMLAB, NTUCSIE 2004

DCT-base transform (II) a=1/2, b = (2/5)1/2, d = 1/2 Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Quantisation Combine with the scale of the DCT Skip the rest details Shengkai Lin@CMLAB, NTUCSIE 2004

4*4 DC luma Transform and Quantisatoion |ZD(i, j)| = (|YD(i, j)| MF(0,0) + 2f ) >> (qbits +1) sign (ZD(i, j)) = sign (YD(i, j)) Shengkai Lin@CMLAB, NTUCSIE 2004

2*2 DC chroma Transform and Quantisatoion |ZD(i, j)| = (|YD(i, j)| MF(0,0) + 2f ) >> (qbits +1) sign (ZD(i, j)) = sign (YD(i, j)) Shengkai Lin@CMLAB, NTUCSIE 2004

Shengkai Lin@CMLAB, NTUCSIE 2004 Thank you Next time, we’ll start at CAVLC Bye… Shengkai Lin@CMLAB, NTUCSIE 2004