1. A performance Comparison of Fractional-Pel Interpolation filters in HEVC and H.264/AVC Guided by: DR.K.R.RAO Submitted by: Lohith Subramanya 1000928742 EE-5359 FINAL PROJECT PRESENTATION

2. OBJECTIVE The objective of this project is to compare and analyze the fractional-pel interpolation filters in HEVC [1] and H.264/AVC [17] based on their frequency responses, coding performance and performance gain. BD-PSNR [33] and BD-Bit Rate [33] are the metrics used for comparison.

3. INTRODUCTION The fractional-pel interpolation filters (6-tap FIR [24] and Average) adopted in H.264/AVC [17] improve motion compensation greatly. Similarly, the DCT - based fractional-pel interpolation filters (7-tap and 8-tap) are adopted in the HEVC [1] standard. This project involves the differences in performance between these two types of filters.

4. H.264 [7] It is an industry standard for video compression, the process of converting digital video into a format that takes up less capacity when it is stored or transmitted. The encoder converts video into a compressed format and the decoder converts compressed video back into an uncompressed format.

5. H.264 Block Diagram [23]

6. HEVC [1] High Efficiency Video Coding (HEVC) [1] is the current joint video coding standardization project of the ITU-T Video Coding Experts Group (VCEG) (ITU-T Q.6/SG 16) and ISO/IEC Moving Picture Experts Group (MPEG) (ISO/IEC JTC 1/SC 29/WG 11).

7. HEVC Block Diagram [6]

8. Why use Interpolation? Motion-compensated prediction (MCP) [8] is the key to the success of the modern video coding standards, as it removes the temporal redundancy in video signals and reduces the size of bitstreams significantly. With MCP, the pixels to be coded are predicted from the temporally neighboring ones, and only the prediction errors and the motion vectors (MV) [8] are transmitted.

9. Interpolation(Continued..) However, due to the finite sampling rate, the actual position of the prediction in the neighboring frames may be out of the sampling grid, where the intensity is unknown. So, the intensities of the positions in between the integer pixels, called sub-positions, must be interpolated and the resolution of MV [8] is increased accordingly.

10. N-Tap FIR Filter [27]

11. Magnitude Response of Half-Pel Interpolation Filters [10] Solid graph: DCTIF 8-Tap Filter Dashed graph: H.264/AVC Filter Dotted graph: DCTIF 6-Tap Filter

12. Filter Co-efficients for half-pel and quarter-pel pixels

13. Filter Weights of Pixels in HEVC

14. Representation of integer and fractional-pels [20]

15. The interpolation filters used in H.264 [17] are 6 tap FIR filter for half-pel interpolation and the average filter for quarter-pel interpolation. Similarly, in HEVC [3], an 8-tap DCTIF is used for half-pel interpolation and a 7-tap DCTIF is used for quarter-pel interpolation. The comparison of the modified filter coefficients based on frequency response that are obtained, are further assessed for the required parametric results mentioned in “A comparison of Fractional-Pel Interpolation Filters in HEVC and H.264/AVC” [10]

16. Results

20. Waterfall_cif.yuv Frame Height: 352 Frame Width: 288 Frame Rate: 25fps No. of frames encoded: 25 HEVCH.264 Encoding Time(seconds)30.73126.268 Bitrate(kbits/sec)232.5279233.7748 Y-PSNR(dB)35.536034.589 U-PSNR(dB)36.871435.170 V-PSNR(dB)37.940837.442 Average PSNR(dB)36.253535.2182

21. Results

22. Bus_qcif.yuv Frame Height: 176 Frame Width: 144 Frame Rate: 25fps No. of frames encoded: 25 HEVCH.264 Encoding Time(seconds)8.7408.200 Bitrate(kbits/sec)320.3269344.89 Y-PSNR(dB)33.899331.8140 U-PSNR(dB)38.268437.5162 V-PSNR(dB)38.812940.0570 Average PSNR(dB)36.259634.8071

23. Results

24. Coastguard.yuv Frame Height: 352 Frame Width: 288 Frame Rate: 15fps No. of frames encoded: 15 HEVCH.264 Encoding Time29.002s27.419s Bitrate(kbits/sec)543.0624577.6629 Y-PSNR(dB)33.971733.5862 U-PSNR(dB)43.802544.6524 V-PSNR(dB)44.628443.2639 Average PSNR(dB)36.257635.4288

25. Results

26. Stefan_cif.yuv Frame Height: 352 Frame Width: 288 Frame Rate: 30fps No. of frames encoded: 30 HEVCH.264 Encoding Time30.746s29.143s Bitrate(kbits/sec)748.1192769.14 Y-PSNR(dB)35.456534.7083 U-PSNR(dB)38.819337.9758 V-PSNR(dB)38.797738.0275 Average PSNR(dB)36.294535.5312

27. Results

28. Conclusion The results obtained from the simulation of the test video sequences show that there is not much difference in the BD-Rates [33] when the half-pel interpolation filter co-efficients are interchanged between HEVC [1] [11] [18] and H.264 [17]. But the interchange between the quarter-pel interpolation filter co-efficients yield a gain of approximately 11 % in the PSNR in HEVC [1] [11] [18] whereas the performance is relatively poor in H.264 [17] which is observed in the results presented. Another important factor that affects the performance is the number of filter taps i.e. more the number of filter taps, better the performance.

29. Future Work  This project is implemented using HM 13 [16] and JM 18.6 [32] reference soft-wares for HEVC [1] [11] [18] and H.264 [17] respectively. If the same could be implemented in HM 14 [36], there may be a possibility of yielding better results.  Complexity analysis of the fractional-pel interpolation pixels both in HEVC [1] [11] [18] and H.264 [17] with respect to memory accesses and arithmetic operations could be carried out. The constraint for carrying out complexity analysis is that it requires a SIMD [38] hardware device. Example: ARM Neon Device [37]

30. List of ACRONYMS AIF: Adaptive Interpolation Filter ALF: Adaptive Loop Filter APEC: Adaptive Prediction Error Coding AVC: Advanced Video Coding AQMS: Adaptive Quantization Matrix Selection CABAC: Context Adaptive Binary Arithmetic Coding CAVLC: Context Adaptive Variable Length Coding CSVT: Circuits and Systems for Video Technology DCT: Discrete Cosine Transform DCTIF: Discrete Cosine Transform Interpolation Filter DMVD: Decoder-side Motion Vector Deviation DSP: Digital Signal Processing EMS: Extended Macro-block Size FIR: Finite Impulse Response HEVC: High Efficiency Video Coding HP: High Profile IBDI: Internal Bit Depth Increasing ITU-T: International Telecommunication Union – Telecommunication Standardization Sector

31. List of ACRONYMS(Continued..) JCT-VC: Joint Collaborative Team on Video Coding JPEG: Joint Photographic Experts Group KLT: Karhunen - Loeve Transform LTS: Larger Transform Size MCP: Motion Compensated Prediction MP: Main Profile MPEG: Moving Picture Experts Group MV: Motion Vectors RDO: Rate Distortion Optimization SIMD: Single Instruction Multiple Data SOC: System On Chip SVN: Sub-Version UVLC: Universal Variable Length Coding VCEG: Video Coding Experts Group VCIP: Visual Communications and Image Processing

32. References 1.Fraunhofer Heinrich Hertz Institute - http://hevc.hhi.fraunhofer.de/http://hevc.hhi.fraunhofer.de/ 2.Open Patents and Standards Platform - http://www.iplytics.com/en/tag/hevc/http://www.iplytics.com/en/tag/hevc/ 3.HEVC Review Site- http://telcogroup.ru/files/materials- pdf/High_Efficiency_Video_Coding_H265.pdfhttp://telcogroup.ru/files/materials- pdf/High_Efficiency_Video_Coding_H265.pdf 4.Overview of HEVC - http://iphome.hhi.de/wiegand/assets/pdfs/2012_12_IEEE-HEVC- Overview.pdfhttp://iphome.hhi.de/wiegand/assets/pdfs/2012_12_IEEE-HEVC- Overview.pdf 5.HEVC Blog: http://www.extremetech.com/computing/162027-h-265-benchmarked- does-the-next-generation-video-codec-live-up-to-expectationshttp://www.extremetech.com/computing/162027-h-265-benchmarked- does-the-next-generation-video-codec-live-up-to-expectations 6.Altera Technologies: http://www.altera.com/technology/system- design/articles/2013/tv-studio-system.htmlhttp://www.altera.com/technology/system- design/articles/2013/tv-studio-system.html 7.I.Richardson, “ Real time implementation of H.264 Video Coding”, 2008 IEEE International SOC Conference, PP: 390, Sept. 2008 8.H.265 Blog http://www.h265.net/2010/07/adaptive-interpolation-filter-for-video- coding.htmlhttp://www.h265.net/2010/07/adaptive-interpolation-filter-for-video- coding.html

33. 9.CNET Blog on HEVC http://news.cnet.com/8301-11386_3-57566116-76/hevc-video-standard- finished-high-end-improvements-coming/http://news.cnet.com/8301-11386_3-57566116-76/hevc-video-standard- finished-high-end-improvements-coming/ 10.H.Lv, et al, “ A comparison of fractional-pel interpolation in HEVC and H.264/AVC”, 2012 IEEE Conference on Visual Communications and Image Processing (VCIP), PP: 1-6, Nov 2012 11.G.J.Sullivan, et al, “ Overview of the HEVC Standard”, 2012 IEEE Transactions on Circuits and Systems for Video Technology(CSVT), Vol: 22, No: 12, PP: 1649-1668, Sept 2012 12.B.Lee, et al, “Performance Comparison of various interpolation methods for color filter arrays”, 2001 IEEE Symposium on Industrial Electronics, Vol: 1, PP: 232-236, June 2001 13.V.Yu and J.Ostermann, “Locally Adaptive Non-Separable Interpolation Filter for H.264/AVC”, 2006 IEEE International Conference on Image Processing, PP: 33-36, Oct 2006 14.Video Test Sequences: http://trace.eas.asu.edu/yuv/http://trace.eas.asu.edu/yuv/ 15.Tortoise SVN Downloadable Software Link: http://tortoisesvn.net/downloads.htmlhttp://tortoisesvn.net/downloads.html

34. 16.HM 13 Software Link: https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM- 13.0+RExt-6.0rc1/https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM- 13.0+RExt-6.0rc1/ 17. H.264 Advanced Video Coding Blog https://www.vcodex.com/h264.htmlhttps://www.vcodex.com/h264.html 18. G.J.Sullivan, et al, “ Standardized Extensions of HEVC”, 2013 IEEE Journal of Selected Topics in Signal Processing, Vol : 7, No: 6, PP: 1001-1016, Dec. 2013 19. K.R.Rao, D.N.Kim and J.J.Hwang, “Video coding standards”, Springer Publications, Jan. 2014: http://www.springer.com/physics/book/978-94-007-6741-6 http://www.springer.com/physics/book/978-94-007-6741-6 20. SPIE Digital Library Article on HEVC: http://electronicimaging.spiedigitallibrary.org/article.aspx?articleid=1730243 http://electronicimaging.spiedigitallibrary.org/article.aspx?articleid=1730243 21. Karhunen-Loeve Transform: http://en.wikipedia.org/wiki/Karhunen%E2%80%93Lo%C3%A8ve_theorem http://en.wikipedia.org/wiki/Karhunen%E2%80%93Lo%C3%A8ve_theorem 22. Sharp 8Kx4K TV: http://www.sound-news.net/index.php/the-novosti/hifi-av- novosti/item/552-sharp-8kx4k-tvhttp://www.sound-news.net/index.php/the-novosti/hifi-av- novosti/item/552-sharp-8kx4k-tv

35. 23. Institute of Computer and Communication Engineering – Article on HEVC: http://research.ncku.edu.tw/re/articles/e/20071102/2.html http://research.ncku.edu.tw/re/articles/e/20071102/2.html 24.FIR Filter: http://en.wikipedia.org/wiki/Finite_impulse_responsehttp://en.wikipedia.org/wiki/Finite_impulse_response 25.JCT-VC Document Management System: http://phenix.int-evry.fr/jct/http://phenix.int-evry.fr/jct/ 26.T.Wiegand, et al, “Overview of the H.264/AVC Video Coding Standard”, 2003 IEEE Transactions on Circuits and Systems for Video Technology, Vol: 13, No: 7, PP: 560-576, July 2003 27.Iowegian International DSP Site: http://www.dspguru.com/dsp/faqs/fir/basicshttp://www.dspguru.com/dsp/faqs/fir/basics 28.N-Tap FIR Filter: http://www.analog.com/static/imported- files/seminars_webcasts/MixedSignal_Sect6.pdfhttp://www.analog.com/static/imported- files/seminars_webcasts/MixedSignal_Sect6.pdf 29.I.Richardson, “ The H.264 Advanced Video Compression Standard”, Wiley Publications, Aug. 2010: http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470516925.html http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470516925.html 30.HM 13 Software Reference Manual: http://mpeg.chiariglione.org/standards/mpeg-h/high-efficiency- video-coding/high-efficiency-video-coding-hevc-encoder-descriptionhttp://mpeg.chiariglione.org/standards/mpeg-h/high-efficiency- video-coding/high-efficiency-video-coding-hevc-encoder-description 31.JPEG: http://www.jpeg.org/http://www.jpeg.org/ 32.JM 18.6 Software Repository: http://iphome.hhi.de/suehring/tml/download/http://iphome.hhi.de/suehring/tml/download/ 33.BD-Metrics: http://www.mathworks.com/matlabcentral/fileexchange/27798-bjontegaard- metric/content/bjontegaard.mhttp://www.mathworks.com/matlabcentral/fileexchange/27798-bjontegaard- metric/content/bjontegaard.m

36. 34.Special issue on emerging research and standards in next generation video coding, IEEE Transactions on Circuits and Systems for Video Technology (CSVT), Vol: 22, PP: 1646-1909, Dec. 2012 35.Special issue on emerging research and standards in next generation video coding, IEEE Transactions on Circuits and Systems for Video Technology (CSVT), Vol: 23, PP: 2009-2142, Dec. 2013 36.HM 14 Software Repository : https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-14.0/ https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-14.0/ 37.ARM NEON: http://www.arm.com/products/processors/technologies/neon.phphttp://www.arm.com/products/processors/technologies/neon.php 38.SIMD: http://en.wikipedia.org/wiki/SIMDhttp://en.wikipedia.org/wiki/SIMD 39.K. Iguchi et al, “HEVC encoder for Super Hi-Vision”, IEEE International Conference on Consumer Electronics, PP: 61-62, Las Vegas, NV, Jan. 2014