1. Digital Video Compression Fundamentals and Standards Web Technology

2. 2008/12/26 Digital Video Compression Fundamentals and Standards 2 Outline Introduction Video Compression Standards Simulation Reference Software Future Work and Conclusions

3. 2008/12/26 Digital Video Compression Fundamentals and Standards 3 Outline Introduction Video Compression Standards Simulation Reference Software Future Work and Conclusions

4. 2008/12/26 Digital Video Compression Fundamentals and Standards 4 Introduction (1/2) Why video compression technique is important ? One movie video without compression 720 x 480 pixels per frame 30 frames per second Total 90 minutes Full color The total quantity of data = 167.96 G Bytes !!

5. 2008/12/26 Digital Video Compression Fundamentals and Standards 5 Introduction (2/2) What is the difference between video compression and image compression? Temporal Redundancy Coding method to remove redundancy Intraframe Coding Remove spatial redundancy Interframe Coding Remove temporal redundancy

6. 2008/12/26 Digital Video Compression Fundamentals and Standards 6 The most intuitive method to remove Spatiotemporal redundancy 3-Dimensional DCT Remove spatiotemporal correlation Good for low motion video Bad for high motion video

7. 2008/12/26 Digital Video Compression Fundamentals and Standards 7 The most popular method to remove temporal redundancy The Block-Matching Algorithm

8. 2008/12/26 Digital Video Compression Fundamentals and Standards 8 Matching Function The dissimilarity between two blocks and The matching criteria Mean square error (MSE) High precision is needed Mean absolute difference (MAD) Low precision is enough

9. 2008/12/26 Digital Video Compression Fundamentals and Standards 9 The Exhaustive Block-Matching Algorithm Reference FrameCurrent Frame Search Range Motion Vector 11 22 33 44 55 66 77 88 12 22 33 44 55 66 77 88 11 23 34 44 55 66 77 88 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 |A|=12

10. 2008/12/26 Digital Video Compression Fundamentals and Standards 10 Fast Block-Matching Algorithms EBMA needs Intensive computation Fast Algorithm is needed Find the possible local optimal Reference FrameCurrent Frame

11. 2008/12/26 Digital Video Compression Fundamentals and Standards 11 Fast Block-Matching Algorithms The characteristics of fast algorithm Not accurate as EBMA Save large computation Two famous fast algorithm 2-D logarithm Search Method Three Steps Search Method

12. 2008/12/26 Digital Video Compression Fundamentals and Standards 12 2-D logarithm Search Method 11 1 1 12 2 2 3 3 4 44 44 1 23 4

13. 2008/12/26 Digital Video Compression Fundamentals and Standards 13 Three Step Search Method 111 1 1 22 2 2 11 11 2 2 2 2 333 33 333 12 3

14. 2008/12/26 Digital Video Compression Fundamentals and Standards 14 Multiresolution Motion Estimation (1/3) The number of levels is L l-th level images of the target frames where is set of pixels at level L At the l-th level, the MV is At the l-th level, the estimated MV is Determine update such that error is minimized The new motion vector is

15. 2008/12/26 Digital Video Compression Fundamentals and Standards 15 Multiresolution Motion Estimation (2/3) Variable block size method

16. 2008/12/26 Digital Video Compression Fundamentals and Standards 16 Multiresolution Motion Estimation (3/3)

17. 2008/12/26 Digital Video Compression Fundamentals and Standards 17 Outline Introduction Video Compression Standards Simulation Reference Software Future Work and Conclusions

18. 2008/12/26 Digital Video Compression Fundamentals and Standards 18 The Development of Video Compression Standards

19. 2008/12/26 Digital Video Compression Fundamentals and Standards 19 The MPEG-1 Standard Group of Pictures Motion Estimation Motion Compensation Differential Coding DCTQuantization Entropy Coding

20. 2008/12/26 Digital Video Compression Fundamentals and Standards 20 Group of Pictures (1/2) I-frame (Intracoded Frame) Coded in one frame such as DCT. This type of frame do not need previous frame P-frame (Predictive Frame) One directional motion prediction from a previous frame The reference can be either I-frame or P-frame Generally referred to as inter-frame B-frame (Bi-directional predictive frame) Bi-directional motion prediction from a previous or future frame The reference can be either I-frame or P-frame Generally referred to as inter-frame

21. 2008/12/26 Digital Video Compression Fundamentals and Standards 21 Group of Pictures (2/2) The distance between two nearest P-frame or P-frame and I-frame denoted by M The distance between two nearest I-frames denoted by N IBBPBBPBBI GOP Bidirectional Motion Compensation Forward Motion Compensation N=9 M=3

22. 2008/12/26 Digital Video Compression Fundamentals and Standards 22 The MPEG-1 Encoder (1/4) DCTQ Entropy Coding Q -1 IDCT Motion Compensation Frame Memory Motion Estimation DCTQ Entropy Coding DCTQ Q -1 IDCT Motion Compensation Frame Memory Motion Estimation Intra-frame Entropy Coding Residue Motion Vector Inter-frame

23. 2008/12/26 Digital Video Compression Fundamentals and Standards 23 The MPEG-1 Encoder (2/4) Differential Coding is the input image is the input image is the predictive image is the predictive imageDCT

24. 2008/12/26 Digital Video Compression Fundamentals and Standards 24 The MPEG-1 Encoder (3/4) Quantization Intra quantization matrix Inter quantization matrix

25. 2008/12/26 Digital Video Compression Fundamentals and Standards 25 The MPEG-1 Encoder (4/4) Motion Compensation Exploit motion vector and the previous reconstructed frame to generate the predictive frame is the compensated image is the compensated image is the previous image is the previous image is the motion vector is the motion vector Reference frame Target frame

26. 2008/12/26 Digital Video Compression Fundamentals and Standards 26 The MPEG-2 Standard Field/Frame DCT Coding Field/Frame Prediction Mode Selection Alternative Scan Order Various Picture Sampling Formats User Defined Quantization Matrix

27. 2008/12/26 Digital Video Compression Fundamentals and Standards 27 Progressive Scan and Interlaced Scan Progressive Scan Interlaced Scan

28. 2008/12/26 Digital Video Compression Fundamentals and Standards 28 Field/Frame DCT Coding The field type DCT Fast motion video The frame type DCT Slow motion video Field DCT CodingFrame DCT CodingLuminance MB

29. 2008/12/26 Digital Video Compression Fundamentals and Standards 29 Alternative Scan Order Zigzag scan order Frame DCT Alternative scan order Field DCT Zig-zag scanAlternate scan 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 4 0 0 0 0 0 0 0 0.72 0 0 0 0 0 0 0 0 0 0 0 0.85 0 0 0 0 0 0 0 0 0 0 0 1.27 0 0 0 0 0 0 0 0 0 0 0 3.62 0 0 0 0 0 0 0 2D DCT

30. 2008/12/26 Digital Video Compression Fundamentals and Standards 30 The MPEG-2 Encoder (2/2) Quantization User can change the quantization if necessary Intra quantization matrix Inter quantization matrix Various picture sampling formats 4:4:44:2:24:2:0

31. 2008/12/26 Digital Video Compression Fundamentals and Standards 31 The MPEG-2 Encoder (1/2) Base Layer Basic quality requirement For SDTV Enhanced Layer High quality service For HDTV DCTQ Entropy Coding Q -1 IDCT Motion Compensation Frame Memory Motion Estimation Q Entropy Coding Q -1 + + + + + Bits Enhance SNR Enhanced Layer + Base Layer Bits Base

32. 2008/12/26 Digital Video Compression Fundamentals and Standards 32 H.264/AVC Variable Block Size Multiple Reference Frames Integer Transform Intra Prediction In-loop Deblocking Filtering 1/4-pel Resolution Motion Estimation CAVLC

33. 2008/12/26 Digital Video Compression Fundamentals and Standards 33 Variable Block Size The fixed block size may not be suitable for all motion objects Improve the flexibility of comparison Reduce the error of comparison 7 types of blocks for selection 0 0 1 023 01 1 16 x 1616 x 88 x 168 x 8 0 0 1 023 01 1 8 x 44 x 84 x 4

34. 2008/12/26 Digital Video Compression Fundamentals and Standards 34 Multiple Reference Frames The neighboring frames are not the most similar in some cases The B-frame can be reference frame B-frame is close to the target frame in many situations

35. 2008/12/26 Digital Video Compression Fundamentals and Standards 35 Integer Transform for Reducing The Spatial Redundancy (1/2) The transform matrix C 4×4 Block Size Separable Integer Transform The transform coefficients are CXC T

36. 2008/12/26 Digital Video Compression Fundamentals and Standards 36 Integer Transform for Reducing The Spatial Redundancy (2/2) CX CTCTCTCTE

37. 2008/12/26 Digital Video Compression Fundamentals and Standards 37 Intra Prediction Predict the similarity between the neighboring pixels in one frame in advance, and exploit differential coding transform coding to remove the redundancy. Vertical Horizontal ++++ + + + + Mean DCDiagonal down-left Horizontal up Diagonal right Vertical rightVertical leftHorizontal down Intra Prediction Transform/ Quantization Entropy Coding

38. 2008/12/26 Digital Video Compression Fundamentals and Standards 38 Remove Perceptual Redundancy In-loop deblocking filtering Remove blocking artifact Result from block based motion compensation Result from block based transform coding p3p3 p2p2 p1p1 p0p0 q0q0 q1q1 q2q2 q3q3 QP

39. 2008/12/26 Digital Video Compression Fundamentals and Standards 39 1/4-pel Resolution Motion Estimation Integer search positions Best integer match Half- pel search positions Best half- pel match Quarter- pel search positions Best quarter-pel match

40. 2008/12/26 Digital Video Compression Fundamentals and Standards 40 The H.264/AVC Encoder Transform/ Quantization Entropy Coding Inverse Transform/ De-Quantization Motion Compensation De-blocking Filter Motion Estimation Residue Motion Vector Intra-frame Prediction Coder Controller Control Data

41. 2008/12/26 Digital Video Compression Fundamentals and Standards 41 Outline Introduction Video Compression Standards Simulation Reference Software Future Work and Conclusions

42. 2008/12/26 Digital Video Compression Fundamentals and Standards 42 H.264 Reference Software JM Current software version: JM 15 Benchmark http://iphome.hhi.de/suehring/tml/

43. 2008/12/26 Digital Video Compression Fundamentals and Standards 43 Reference Software Demo

44. 2008/12/26 Digital Video Compression Fundamentals and Standards 44 Outline Introduction Video Compression Standards Simulation Reference Software Future Work and Conclusions

45. 2008/12/26 Digital Video Compression Fundamentals and Standards 45 Future Work Fast Mode Decision Algorithm Interpolation Filter Design Deblocking Filter Design DCT-Based Motion Estimation Implementation Based on TI DSP

46. 2008/12/26 Digital Video Compression Fundamentals and Standards 46 Fast Mode Decision Algorithm The computational cost of H.264 is large Variable block-size ME Multiple reference frames Fast mode decision is needed for reducing the computation time

47. 2008/12/26 Digital Video Compression Fundamentals and Standards 47 Interpolation Filter Design In order to estimate and compensate the fractional-pel displacements Adaptive Interpolation filter for replacing the fixed coefficient filter

48. 2008/12/26 Digital Video Compression Fundamentals and Standards 48 Deblocking Filter Design Block based ME and Transform result in the annoying blocking artifact Reduce the blocking artifact can increase the quality of compressed video

49. 2008/12/26 Digital Video Compression Fundamentals and Standards 49 DCT Based Motion Estimation Robust even in noisy environment Complexity comparison DCT Based ME O(M 2 ) Block Based ME O(N 2 ×M 2 ) Concept Pseudo Phase Similar to

50. 2008/12/26 Digital Video Compression Fundamentals and Standards 50 Implementation Based on TI DSP TMS320C6416 TI DM642

51. 2008/12/26 Digital Video Compression Fundamentals and Standards 51 Conclusions Thank You Q & A

52. 2008/12/26 Digital Video Compression Fundamentals and Standards 52 References (1/2) [1] Yun Q.Shi and Huifang Sun, “Image and Video Compression for Multimedia Engineering: Fundamentals, Algorithms, and Standards”, CRC press, 2000. [2] Yao Wand, Jorn Ostermann and Ya-Qin Zhang, “Video Processing and Communications”, Prentice Hall, 2007. [3] Richardson, Lain E. G., “Video Codec Design: Developing Image and Video Compression Systems”, John Wiley & Sons Inc, 2002. [4] Barry G, Haskell, Atul Puri and Arun N. Netravali, “Digital Video : An Introduction to MPEG-2”, Boston : Kluwer Academic, 1999. [5] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the H.264/AVC video coding standard”, IEEE Trans. on Circuits and systems for video Technology, vol. 13, no. 7, pp. 560-576, July 2003. [6] G. Sullivan and T. Wiegand, “Video Compression - From Concepts to the H.264/AVC Standard”, Proceedings of the IEEE, Special Issue on Advances in Video Coding and Delivery, December 2004. [7] 酒井善則、吉田俊之 共著，白執善 編譯， “ 影像壓縮技術 ” ，全華， 2004.

53. 2008/12/26 Digital Video Compression Fundamentals and Standards 53 References (2/2) [8] Thomas Wedi, “Adaptive Interpolation Filters and High-Resolution Displacements For Video Coding”, IEEE Trans. on Circuits and Systems For Video Technology, vol. [9] Dong-Hwan Kim, Hwa-Yong Oh, O˘guzhan Urhan, Sarp Ertürk and Tae- Gyu Chang, “Optimal Post-Process/In-Loop Filtering for Improved Video Compression Performance”, IEEE Trans. on Consumer Electronics, vol. 53, no. 4, Nov. 2007. Circuits and Systems, 2005. [10] Shu-Fa Lin, Meng-Ting Lu, and Homer Chen, ” Fast Multi-Frame Motion Estimation for H.264 and Its Applications to Complexity-Aware Streaming”, IEEE International Symposium on Circuits and Systems, 2005. [11] Kai-Ting Cheng and Hsueh-Ming Hang, “Acceleration and Implementation of H.264 Encoder and Scalable Extension of H.264 Decoder on TI DSP Platform”, master thesis, June 2007