ENEE 631 Project Video Codec and Shot Segmentation Aravind Sundaresan Vikas Raykar 11/27/2018 ENEE 631 Project: Video Codec

Main features/ functionality Can encode monochrome video with frame dimensions (16*M, 16*N). Codes the sequence as a series of I/P frames. The I/P decision is made according to suitability of each method. (Example: when a scene change is detected, the subsequent frame is coded INTRA). The frames are periodically coded as INTRA according to the INTRA refresh rate parameter. Temporal prediction is closed loop. Performs full-pel Motion Estimation in a window of of dimensions 48 x 48. Macroblocks within an I-frame are coded as INTRA/ INTER according to the compression achieved. Has a resynchronization marker at frame level. 11/27/2018 ENEE 631 Project: Video Codec

ENEE 631 Project: Video Codec Video Codec Structure The Video Codec is split into two programs Encoder and Decoder. Both of them have three layers. Top Layer: Takes care of interface and I/O. Performs the necessary Initializations. Splits the input into frames and feeds them to the next layer sequentially. 11/27/2018 ENEE 631 Project: Video Codec

Video Encoder - Block diagram Top Layer Intermediate Layer Input Raw frame Recon. frame Bottom Layer Bitstream Residue frame Recon. Residue frame Encoder Blocks ME, MC 11/27/2018 ENEE 631 Project: Video Codec

ENEE 631 Project: Video Codec Video Codec Structure Intermediate Layer. This Layer performs the frame level manipulations and also takes care of the frame-level and macroblock-level decision making in the encoder. Performs Motion Estimation and Compensation or removes 128 from the frame to get Residue Frame. Feeds the residue frame to the frame encode layer. The reconstructed residue frame is used to reconstruct the current frame for future prediction. Bottom Layer: This layer performs the actual coding. A hybrid coding technique, that employs both predictive coding to remove temporal redundancy and transform coding to remove spatial redundancy is used. The frame is split into macroblocks and each macroblock is coded separately. The bits generated are put in the bitstream. 11/27/2018 ENEE 631 Project: Video Codec

Top Layer (Interface Layer) Performs the necessary initializations (such as Loading Huffman tables). Serves as interface between user and the actual encoder. Input sequence is read and passed as frames to the lower layer. Very first frame is forced to be INTRA. Subsequent frames are by default directed to be coded as INTER. The lower layer may dynamically decide to code such a frame AS INTRA according to various parameters (scene change / INTRA refresh). 11/27/2018 ENEE 631 Project: Video Codec

Intermediate Layer (Control Layer) This is an important layer in that most of the decision making is performed here. These decisions are aimed at selecting the best coding technique according to the input frame. The decisions made and the functions performed are listed below. Intra / Inter Decision. The top layer has the ability to force the Intra Option. The Intermediate layer has the option to change the INTER option to INTRA. If the number of consecutive frames coded as P-frames equals a certain parameter (INTRA REFRESH RATE), the next frame is coded as an I-frame. 11/27/2018 ENEE 631 Project: Video Codec

Intermediate Layer (Control Layer) Motion Estimation and Compensation. In case of Intra Macroblocks, 128 is subtracted from the Macroblock. Based on the output of the Motion Estimation a decision is made whether to code the frame as INTER. The frame to be coded is split into a MC frame and Residue frame (in case of INTRA frame, the MC frame comprises of pixels with value 128). The Residue frame is passed to the Encode frame layer. The layer returns the reconstructed residue frame which is added to the MC frame to be used for future prediction. (Closed Loop prediction to avoid error accumulation) 11/27/2018 ENEE 631 Project: Video Codec

Bottom Layer (Encoder Layer) The frame is split into Macroblocks each of which comprises of 4 blocks. The macroblocks are read in raster scan order and coded sequentially and the blocks in the Macroblock are also similarly coded. Each macroblock consists of a header followed by the coefficient data. The header contains Coded Information (Coded/ Not coded, INTRA/ INTER, etc) Motion Vector (for INTRA MBs) Coded Block Pattern Optionally include the Quantizer (or differential Quantizer). 11/27/2018 ENEE 631 Project: Video Codec

Bottom Layer (Encoder Layer) The coding Procedure is described below. Each block is transform coded using the DCT. The DCT coefficients are quantized. The Quantization tables for INTRA and INTER blocks are different. Resulting matrix is split into DC and AC coefficients which are scanned in a zigzag manner and coded using fixed Huffman tables and run-length coding techniques. For run-length values not found in the table the run-length and level values are coded using an ESCAPE code and fixed length codes. If none of the blocks contain any coefficients, the MB is 'not coded'. If only some of the blocks are coded, the corresponding bit is set in the CBP. 11/27/2018 ENEE 631 Project: Video Codec

Video Decoder - Block diagram Top Layer Intermediate Layer Output Ref. frame Cur. Recon. frame Bottom Layer Bitstream Recon. Residue frame Decoder Blocks MC 11/27/2018 ENEE 631 Project: Video Codec

ENEE 631 Project: Video Codec Decoded Frames 11/27/2018 ENEE 631 Project: Video Codec

ENEE 631 Project: Video Codec Results 30 frames encoded Size of compressed stream = 434096 bytes Size of 30 frames: 30.240.352 = 2534400 bytes Compression = 17% Greater Compression can be achieved by Increasing quantizer step Increase ratio of P frames to I frames (current ratio 10:1) 11/27/2018 ENEE 631 Project: Video Codec