Image/Video Coding Techniques for IPTV Applications Wen-Jyi Hwang ( 黃文吉 ) Department of Computer Science and Information Engineering, National Taiwan Normal University, Taipei, 117, Taiwan

Goal of This Talk  Overview of Image/Video Coding Techniques for IP TV Applications Image Coding: JPEG, JPEG2000 Video Coding: H.261, MPEG1, MPEG2, H.263, MPEG4, H.264

WHY Compression ? DSL ~200 kbps ~ 1,000 : 1 ~ 10,000 : 1 ITU-R Mbps SDTV broadcasting ~2 Mbps ~ 100 : 1 Dial-up modem, wireless link ~ 20 kbps

Image Coding Techniques  JPEG (Joint Photographic Experts Group) DCT-Based Technique Published in 1992  JPEG2000 Wavelet-Based Technique Published in 2000

Principles of Image Coding Technique Color Transform DCT/WaveletQuantization Bitstream Formation

Color Transform (JPEG) Process the data in blocks of 8×8 samples Convert RGB into Luminance (Y) and Chrominance (Cr and Cb). Use half resolution for Chrominance (because eye is more sensitive to Luminance)

DCT (JPEG) Transform each block of 8×8 samples into 64 DCT coefficients –energy tends to be concentrated into a few significant coefficients

Quantization (JPEG) Divide each DCT coefficient by an integer, discard remainder Typically, a few non-zero coefficients are left.

Quantization (JPEG)

Wavelet Transform (JPEG2000)

Quantization(JPEG2000)

Quantization for Resolution- Scalable Transmission

Quantization (JPEG2000)

Layers 1+2Layers 1+2+3Layers

Quantization for SNR-Scalable Transmission

JPEG2000-Based Layered Transmission

JPEG2000-Based Region of Interest (ROI) Encoding

Comparisons (JPEG and JPEG2000) JPEG (64:1) JPEG2000 (64:1)

Video Compression-History “It has been customary in the past to transmit successive complete images of the transmitted picture.” [...] “In accordance with this invention, this difficulty is avoided by transmitting only the difference between successive images of the object.”

Features of Moving Pictures  Moving images contain significant temporal redundancy successive frames are very similar

Motion Estimation and Compensation  The amount of data to be coded can be reduced significantly if the previous frame is subtracted from the current frame.

Block-Matching Results of block- matching The predicted current frame The reconstructed previous frameThe current frame

Predicted Frame and Prediction Error Predicted Current Frame Displaced Frame Difference (DFD, 估測誤差 )

I, B and P Frames  A video sequences may contain three different types of frames: (a)Intra frames, (I-frames) (b)Predicted frames (P-frames), (c)Bi-directional frames (B-frames)

Basic MPEG Standard Encoding order: I 0, P 3, B 1, B 2, P 6, B 4, B 5, I 9, B 7, B 8. Playback order: I 0, B 1, B 2, P 3, B 4, B 5, P 6, B 7, B 8, I 9.

Basic Video Encoder Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Video in DFD Motion Vector Encoded DFD Reconstructed DFD Reconstructed current frame Predicted current frame

Basic Video Decoder Encoded bitstream in Motion vector Encoded DFD Predicted current frame Reconstructed DFD Reconstructed current frame

Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in

Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in ¼-pixel accuracy

Fractional Motion Estimation

Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Adaptive block sizes...

Adaptive Block Size

Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Multiple Past Reference Frames

Advantages of Multiple Reference Frames

Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in Generalized B-Frames

Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Standards: H.261, MPEG-1, MPEG-2, H.263, MPEG-4, H.264/AVC Video in De-Blocking Filtering

without deblock filtering with deblock filtering

Comparisons (H.264 and JPEG2000) JPEG2000 H.264

Comparisons (MPEG-4 and H.264)

MPEG-4 H Mbps

Concluding Remarks  Image/Video Coding Techniques such as JPEG2000 and H.264 are able to provide high fidelity while significantly reduce bandwidth for image/video transmission.  These coding techniques therefore are effective for IPTV and/or DVB applications with limited bandwidth resources.