1. Multi-Frame Motion Estimation and Mode Decision in H.264 Codec Shauli Rozen Amit Yedidia Supervised by Dr. Shlomo Greenberg Communication Systems Engineering Department Ben-Gurion University Beer - Sheva,Israel Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

2. Presentation Content H.264 brief overview. Introduction to Multi-Frame Reference Motion Estimation & Mode Decision. Complexity Analysis. The proposed Algorithm. Status & Current results. Future work. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

3. Video Standard H.264 / MPEG4 AVC H.261 H.262 / MPEG2 H.263 H.263+ H.263 ++ ITU-T Standards Joint ITU-T & ISO/MPEG Standards ISO/MPEG Standards 19841986198819901992199419961998200020022004 MPEG1MPEG4 Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

4. New Features in H.264 Multi-Frame reference Motion Estimation. 7 partitioning modes in Inter frames. Multi-mode intra-prediction. Motion vector can point out of image border. 1/4-, 1/8-pixel motion vector precision. B-frame prediction weighting. 4  4 integer transform. UVLC (Uniform Variable Length Coding). NAL (Network Abstraction Layer). SP-slices. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

5. H.264 Encoder Entropy Coding Scaling & Inv. Transform Motion- Compensation Control Data Quant. Transf. coeffs Motion Data Intra/Inter Coder Contro l Decoder Motion Estimation Transform/ Scal./Quant. - Input Video Signal Split into Macroblocks 16x16 pixels Intra-frame Prediction De-blocking Filter Output Video Signal Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

6. Motion Estimation in H.264 Various block sizes and shapes 8x8 0 4x8 01 01 23 4x4 8x4 1 0 8x8 Types 0 16x16 01 8x16 MB Types 8x8 01 23 16x8 1 0 Multiple Reference Frames for Motion Compensation [t-1][t-4][t-3][t-2][t-5] Each 16x16 MB can be partitioned in 259 different modes. Each block can be searched within the 16 preceding frames Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

7. Why Multi-Frame Reference Motion Estimation? Hiding Objects – After revealing. The best match might be found in frame before the hiding. Periodic Movements – When the object is moving but repeat its original position every few frames. The best match might be found in the last frame where the object was with the same position. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

8. Why Variable Block Sizes? Increased spatial & temporal correlation. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

9. Complexity Analysis Each 16x16 MB can be decoded as Inter, Intra or skip. Intra: 2 modes - 4x4 or 16x16  Intra4x4 – 9 prediction modes (16x9 calculation of 16 pixels)  Intra16x16 – 4 prediction mode (4 calculations of 256 pixels) Inter: 7 modes - 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, 4x4 (259 partition options, 41 searches, 5 frames). search of [W x L] mode in window size (2W+1)(2L+1) requires (W+1)(L+1) calculation of W x L pixels. with Fix window size 33x33 - 5,190,400 MAC’s with relative window size (2W+1)(2L+1) -1,012,480 MAC’s Motivation for complexity reduction is obvious!!!

10. Rate Distortion A new decision criteria introduced by the standard. The Rate Distortion is taking in account the prediction error (Diff) and length of the needed bit- stream ( ). Now it is hard to tell in advanced which mode or reference frame will be selected. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

11. Rate Distortion in H.264 Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

12. Multi-Frame Reference Full Search Brute Force. MB Mode: 16x16 Search window size: (2*16+1)(2*16+1)=33x33 Number of reference frames: 5 Error criterion: MSE. Complexity (for one MB) : (16+1)(16+1)*16*16*5=369,920 MAC’s Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

13. Multi-Frame Reference Usage Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

14. PSNR Gain of MFS GainFSMFS 0.9825.1926.17Bus 0.9443.0043.95Akiyo 0.4929.9830.48Coastguard 1.1333.7434.87Foreman 0.6640.1740.83Mother 0.4537.1837.63News 0.7734.8835.65Average Notice: This Gain achieved by using the Multi-Frame Reference feature only (without the different partition modes). Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

15. Why Adaptive? In non adaptive Block Matching Algorithms: -The search area is constant in place and size. -The number of searches made for each Macro-Block is constant. -If there is fast motion in the scene and the search area is too small, the object will go out of the search window and will get poor results. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

16. Why Adaptive? (cont ’ d) In adaptive Block Matching Algorithms: -The search area is not constant in size. -The method of search can be changed and the location of the search window can be changed. -This concludes to fewer searches and better PSNR results even in scenes with fast motion. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

17. Motivation Frame [t-1] Frame [t] Obvious temporal and spatial correlation of MV’s and partition modes. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

18. Adaptive Multi-Frame Block Matching Algorithm Step 1 - Predictors selection Step 2 - Thresholds setting. Step 3 - Applying Predictors. Step 4 - Decision & Refinements. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

19. Temporal Neighbors Predictors [t-2][t-4][t-3][t-5] [t-1] Current Frame [t] Spatial Neighbors Predictors Step 1 – Predictors selection. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

20. Step 2 – Threshold setting Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

21. Step 3 – Applying Predictors [t-2][t-4][t-3][t-5] [t-1] Current Frame [t] MSE min MV o Calculate all MSE’s and set MVo as the MV which achieved the lowest prediction error. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

22. Step 4 - Decision & Refinement [t-3] Current Frame [t] MSEmin<MSEpmin – Search is stopped (early termination) MV o Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

23. Step 4 (cont ’ d) [t-3] Current Frame [t] MV o MSEpmin < MSEmin < MSEpavg – Refinement search is applied in [3x3] search window around MV o Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

24. Step 4 (cont ’ d) [t-3] Current Frame [t] MV o MSEpavg < MSEmin Refinement with distinctive window size (3x3,4x4,5x5) is done around the three predictors from the initial set of predictors and which provided the minimal MSE’s. [t-1][t-4] MV 2 MV 1 Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

25. PSNR Gain AMFBMAFSMFS 25.8125.1926.17Bus 43.9243.0043.95Akiyo 30.4329.9830.48Coastguard 34.7633.7434.87Foreman 40.5440.1740.83Mother 37.4937.1837.63News 35.4934.8835.65Avearge Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

26. Computational Complexity Reduction (compared to MFS) 9.31%Bus 5.42%Akiyo 9.05%Coastguard 9.42%Foreman 11.13%Mother 7.65%News 8.67%Average Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

27. Our Goal Integration in the Open source software (JM)  Matlab cant provide Rate-Distortion statistics. Improve the proposed methods.  Predictors early elimination.  Predictors priority mechanism.  Improved refinement search pattern – less searches in the refinement step. Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel

28. Questions ? Communication Systems Engineering Department, Ben-Gurion University of the Negev, Beer-Sheva, Israel