1. CMPT365 Multimedia Systems 1 Media Compression - Video Spring 2015 CMPT 365 Multimedia Systems

2. CMPT365 Multimedia Systems 2 Introduction r What’s video ? m a time-ordered sequence of frames, i.e., images. r How to compress ? m Spatial redundancy – compression on each individual image (Motion JPEG) m Temporal redundancy – prediction based on previous images

3. CMPT365 Multimedia Systems 3 Temporal Redundancy r Characteristics of typical videos: m A lot of similarities between adjacent frames m Differences caused by object or camera motion Frame 1 Frame 2 Direct Difference

4. CMPT365 Multimedia Systems 4 Key Idea in Video Coding r Predict each frame from the previous frame and only encode the prediction error: m Pred. error has smaller energy and is easier to compress Intra-coded I-frame Predicted P-frame

5. CMPT365 Multimedia Systems 5 Motion ? Previous frame Current Frame

6. CMPT365 Multimedia Systems 6 Motion Estimation (ME) r For each block, find the best match in the previous frame (reference frame) m Upper-left corner of the block being encoded: (x0, y0) m Upper-left corner of the matched block in the reference frame: (x1, y1) m Motion vector (dx, dy): the offset of the two blocks: (dx, dy) = (x1 – x0, y1 – y0) (x0, y0) + (dx, dy) = (x1, y1) m Motion vector need to be sent to the decoder. (x1, y1) (x0, y0)

7. CMPT365 Multimedia Systems 7 Motion Estimation Example Reference Current Frame Plotted by quiver( ) in Matlab.

8. CMPT365 Multimedia Systems 8 Motion Compensation (MC) r Given reference frame and the motion vector, can obtain a prediction of the current frame r Prediction error: Difference between the current frame and the prediction. r The prediction error will be coded by DCT, quantization, and entropy coding.

9. CMPT365 Multimedia Systems 9 GOP, I, P, and B Frames I P P P … P P P …… r GOP: Group of pictures (frames). r I frames (Key frames): m Intra-coded frame, coded as a still image. Can be decoded directly. m Used for GOP head, or at scene changes. m I frames also improve the error resilience. r P frames: (Inter-coded frames) m Predication-based coding, based on previous frames. GOP I P P P … P P P …… GOP

10. CMPT365 Multimedia Systems 10 GOP, I, P, and B Frames 1 2 3 4 5 6 7 I B B P B B P r B frames: Bi-directional interpolated prediction frames m Predicted from both the previous frame and the next frame: more flexibilities  better prediction. r B frames are not used as reference for future frames: m B frames can be coded with lower quality or can be discarded without affecting future frames. r Encoding order: 1 4 2 3 7 5 6 r Decoding order: 1 4 2 3 7 5 6 r Display order: 1 2 3 4 5 6 7 r Need more buffers r Need buffer manipulations to display the correct order.

11. CMPT365 Multimedia Systems 11 Basic Encoder Block Diagram DCT Q Q I DCT MC Input frame Entropy Coding Intra Inter Prediction Recon Pred error Recon. Prediction Motion vectors Pred. error Memory ME Intra Inter Reconstructed Previous frame Use reconstructed error in the loop to prevent drifting. Original input is not available to the decoder. Need a buffer to keep the reference frame.

12. CMPT365 Multimedia Systems 12 Basic Decoder Block Diagram Q I DCTEntropy Decoding Intra Inter MC Motion vectors Recon Pred error Prediction Reconstructed frame Memory Reconstructed Previous frame r Decoder is simpler than the encoder: m No need to do motion estimation.

13. CMPT365 Multimedia Systems 13 Motion Estimation - Revisit r Formulation: r Find (i, j) in a search window (-p, p) that minimizes r Mean square error (MSE) m If z=2 r Mean absolute distance (MAD): m If z = 1. r # of search candidates: (2p+1) x (2p + 1)

14. CMPT365 Multimedia Systems 14 MAD-based Motion Estimation r Objective m Find vector (i, j) as the motion vector MV = (u;v), such that MAD(i,j) is minimum

15. CMPT365 Multimedia Systems 15 Naive Method r Sequential search (Full search): - sequentially search the whole (2p+1) (2p+1) window in the Reference frame m a macroblock centered at each of the positions within the window is compared to the macroblock in the Target frame, pixel by pixel m respective MAD is derived m vector (i, j) that offers the least MAD is designated as the MV (u, v) for the macroblock in the target frame

16. CMPT365 Multimedia Systems 16 Fast Motion Estimation r Full-search motion estimation is time consuming: m Each (i, j) candidate: N 2 summations m If search window size is W 2, need W 2 x N 2 comparisions / MB W=2p+1=31, N=16:  246016 comparisons / MB ! Each comparison three operations (subtraction, absolute value, addition) r Fast motion estimation is desired: m Lower the number of search candidates m Many methods

17. CMPT365 Multimedia Systems 17 2-D Log Search r Logarithmic search: m a cheaper version m suboptimal but still usually effective. r Procedure – similar to a binary search m Initially, only nine locations in the search window are used as seeds for a MAD-based search; marked as `1'. m After the one that yields the minimum MAD is located, the center of the new search region is moved to it and the step-size (“offset") is reduced to half. m In the next iteration, the nine new locations are marked as `2', and so on.

18. CMPT365 Multimedia Systems 18 Log Search

19. CMPT365 Multimedia Systems 19 Further Exploration r Textbook Chapter 10 r Other sources m A Java H.263 decoder by A.M. Tekalp m Digital Video and HDTV Algorithms and Interfaces by C.A. Poynton m Image and Video Compression Standards by V. Bhaskaran and K. Konstantinides m Video Coding: An introduction to standard codecs by M. Ghanbari m Video processing and communications by Y. Wang et al.