EE569 Digital Video Processing Roadmap Introduction Intra-frame coding Review of JPEG Inter-frame coding Conditional Replenishment (CR) Coding Motion Compensated Predictive (MCP) Coding Object-based and scalable video coding* Motion segmentation, scalability issues EE569 Digital Video Processing

Introduction to Video Coding Lossless vs. lossy data compression Source entropy H(X) Rate-Distortion function R(D) or D(R) Probabilistic modeling is at the heart of data compression What is P(X) for video source X? Is video coding more difficult than image coding? EE569 Digital Video Processing

EE569 Digital Video Processing Shannon’s Picture Distortion For Gaussian source N(0,2) Coder A Coder B Rate (bps) For video source, no one knows the limit (bound) EE569 Digital Video Processing

EE569 Digital Video Processing Distortion Measures Objective Mean Square Error (MSE) Peak Signal-to-Noise-Ratio (PSNR) Measure the fidelity to original video Subjective Human Vision System (HVS) based Emphasize visual quality rather than fidelity We only discuss objective measures in this course, but subjective video quality assessment is an open and important topic EE569 Digital Video Processing

Video Coding Applications EE569 Digital Video Processing

EE569 Digital Video Processing Roadmap Introduction Intra-frame coding Review of JPEG Inter-frame coding Conditional Replenishment (CR) Motion Compensated Prediction (MCP) Object-based and scalable video coding* Motion segmentation, scalability issues EE569 Digital Video Processing

A Tour of JPEG Coding Standard Key Components Transform -8×8 DCT -boundary padding Quantization -uniform quantization -DC/AC coefficients Coding -Zigzag scan -run length/Huffman coding EE569 Digital Video Processing

EE569 Digital Video Processing JPEG Baseline Coder Tour Example EE569 Digital Video Processing

EE569 Digital Video Processing Step 1: Transform • DC level shifting -128 • 2D DCT DCT EE569 Digital Video Processing

EE569 Digital Video Processing Step 2: Quantization Why increase from top-left to bottom-right? Q-table Q EE569 Digital Video Processing

EE569 Digital Video Processing Step 3: Entropy Coding Zigzag Scan (20,5,-3,-1,-2,-3,1,1,-1,-1, 0,0,1,2,3,-2,1,1,0,0,0,0,0, 0,1,1,0,1,EOB) End Of the Block: All following coefficients are zero Zigzag Scan EE569 Digital Video Processing

EE569 Digital Video Processing Roadmap Introduction Intra-frame coding Review of JPEG Inter-frame coding Conditional Replenishment (CR) Motion Compensated Prediction (MCP) Object-based and scalable video coding* Motion segmentation, scalability issues EE569 Digital Video Processing

Conditional Replenishment Based on motion detection rather than motion estimation Partition the current frame into “still areas” and “moving areas” Replenishment is applied to moving regions only Repetition is applied to still regions Need to transmit the location of moving areas as well as new (replenishment) information No motion vectors transmitted EE569 Digital Video Processing

Conditional Replenishment EE569 Digital Video Processing

EE569 Digital Video Processing Motion Detection EE569 Digital Video Processing

From Replenishment to Prediction Replenishment can be viewed as a degenerated case of prediction Only zero motion vector is considered Discard the history A more powerful approach of exploiting temporal dependency is prediction Locate the best match from the previous frame Use the history to predict the current EE569 Digital Video Processing

Differential Pulse Coded Modulation ^ xn yn yn ^ ^ yn xn _ Q + xn-1 ^ ^ ^ xn-1 xn D + D ^ xn-1 Decoder Encoder Xn,yn: unquantized samples and prediction residues ^ ^ Xn,yn: decoded samples and quantized prediction residues EE569 Digital Video Processing

Motion-Compensated Predictive Coding EE569 Digital Video Processing

EE569 Digital Video Processing A Closer Look EE569 Digital Video Processing

EE569 Digital Video Processing Key Components Motion Estimation/Compensation At the heart of MCP-based coding Coding of Motion Vectors (overhead) Lossless: errors in MV are catastrophic Coding of MCP residues Lossy: distortion is controlled by the quantization step-size Rate-Distortion optimization EE569 Digital Video Processing

Block-based Motion Model Block size Fixed vs. variable Motion accuracy Integer-pel vs. fractional-pel Number of hypothesis Overlapped Block Motion Compensation (OBMC) Multi-frame prediction EE569 Digital Video Processing

Quadtree Representation of Motion Field with Variable Blocksize Sullivan, G.J.; Baker, R.L., "Rate-distortion optimized motion compensation for video compression using fixed or variable size blocks," GLOBECOM '91. pp.85-90 vol.1, 2-5 Dec 1991 EE569 Digital Video Processing

EE569 Digital Video Processing Example counted bits using a VLC table EE569 Digital Video Processing

EE569 Digital Video Processing Fractional-pel BMA Recall the tradeoff between spending bits on motion and spending bits on MCP residues Intuitively speaking, going from integer-pel to fractional-pel is good for it dramatically reduces the variance of MCP residues for some video sequence. The gain quickly saturates as motion accuracy refines EE569 Digital Video Processing

EE569 Digital Video Processing Example 8-by-8 block, integer-pel, var(e)=220.8 8-by-8 block, half-pel, var(e)=123.8 MCP residue comparison for the first two frames of Mobile sequence EE569 Digital Video Processing

EE569 Digital Video Processing Fractional-pel MCP Girod, B., "Motion-compensating prediction with fractional-pel accuracy," IEEE Trans. on Communications, vol.41, no.4, pp.604-612, Apr 1993 EE569 Digital Video Processing

EE569 Digital Video Processing Multi-Hypothesis MCP Using one block from one reference frame represents a single-hypothesis MCP It is possible to formulate multiple hypothesis by considering Overlapped blocks More than one reference frame Why multi-hypothesis? The benefit of reducing variance of MCP residues outweighs the increased overhead on motion EE569 Digital Video Processing

EE569 Digital Video Processing Example: B-frame fn-1 fn fn+1 EE569 Digital Video Processing

EE569 Digital Video Processing Generalized B-frame fn-2 fn-1 fn+2 fn fn+1 EE569 Digital Video Processing

Overlapped Block Motion Compensation (OBMC)   EE569 Digital Video Processing

Overlapped Block Motion Compensation (OBMC) Conventional block motion compensation One best matching block is found from a reference frame The current block is predicted by the best matching block OBMC Each pixel in the current block is predicted by a weighted average of several corresponding pixels in the reference frame The corresponding pixels are determined by the MVs of the current as well as adjacent MBs The weights for each corresponding pixel depends on the expected accuracy of the associated MV EE569 Digital Video Processing

OBMC Using 4 Neighboring MBs Should be inversely proportional to the distance between x and the center of EE569 Digital Video Processing

Optimal Weighting Design* Convert to an optimization problem: Minimize Subject to Optimal weighting functions: EE569 Digital Video Processing

EE569 Digital Video Processing Multi-Hypothesis MCP EE569 Digital Video Processing

EE569 Digital Video Processing Key Components Motion Estimation At the heart of MCP-based coding Coding of Motion Vectors (overhead) Lossless: errors in MV are catastrophic Coding of MCP residues Lossy: distortion is controlled by the quantization step-size Rate-Distortion optimization EE569 Digital Video Processing

EE569 Digital Video Processing Motion Vector Coding 2D lossless DPCM Spatially (temporally) adjacent motion vectors are correlated Use causal neighbors to predict the current one Code Motion Vector Difference (MVD) instead of MVs Entropy coding techniques Variable length codes (VLC) Arithmetic coding EE569 Digital Video Processing

EE569 Digital Video Processing MVD Example MV1 MV2 MV3 MV Due to smoothness of MV field, MVD usually has a smaller variance than MV EE569 Digital Video Processing

EE569 Digital Video Processing VLC Example MVx/MVy symbol codeword 1 1 1 2 010 -1 3 011 2 4 00100 -2 5 00101 3 6 00110 Exponential Golomb Codes: 0…01x…x m m-1 EE569 Digital Video Processing

EE569 Digital Video Processing Key Components Motion Estimation At the heart of MCP-based coding Coding of Motion Vectors (overhead) Lossless: errors in MV are catastrophic Coding of MCP residues Lossy: distortion is controlled by the quantization step-size Rate-Distortion optimization EE569 Digital Video Processing

EE569 Digital Video Processing MCP Residue Coding Transform Quantization Coding Conceptually similar to JPEG Transform: unitary transform Quantization: Deadzone quantization Coding: Run-length coding EE569 Digital Video Processing

EE569 Digital Video Processing Transform Unitary matrix: A is real, A-1=AT Unitary transform: A is unitary, Y=AXAT Examples 8-by-8 DCT 4-by-4 integer transform EE569 Digital Video Processing

Deadzone Quantization    2    codewords EE569 Digital Video Processing

EE569 Digital Video Processing Key Components Motion Estimation At the heart of MCP-based coding Coding of Motion Vectors (overhead) Lossless: errors in MV are catastrophic Coding of MCP residues Lossy: distortion is controlled by the quantization step-size Rate-Distortion optimization EE569 Digital Video Processing

Constrained Optimization Min f(x,y) subject to g(x,y)=c

Lagrangian Multiplier Method Motion estimation Mode selection QUANT: a user-specified parameter controlling quantization stepsize EE569 Digital Video Processing

Example: Rate-Distortion Optimized BMA Distortion alone Rate and Distortion counted bits using a VLC table EE569 Digital Video Processing

EE569 Digital Video Processing Experimental Results Cited from G. Sullivan and L. Baker, “Rate-Distortion optimized motion compensation for video compression using fixed or variable size blocks”, Globecom’1991 EE569 Digital Video Processing

EE569 Digital Video Processing Summary How does MCP coding work? The predictive model captures the slow-varying trend of the samples {fn} The modeling of prediction residues {en} is easier than that of original samples {fn} Fundamental weakness Quantization error will propagate unless the memory of predictor is refreshed Not suitable for scalable coding applications EE569 Digital Video Processing