1. Parallelization of HEVC Deblocking filters using CUDA GPU A PROJECT PROPOSAL UNDER THE GUIDANCE OF DR. K. R. RAO COURSE: EE5359 - MULTIMEDIA PROCESSING, SPRING 2016 SUBMISSION DATE: 14th April 2016 SUBMITTED BY ARPITA YAGNIK UT ARLINGTON ID: 1000810583 EMAIL ID: arpita.yagnik@mavs.uta.eduarpita.yagnik@mavs.uta.edu

2. TABLE OF CONTENTS 1.Acronyms 2.Objective and Action Plan 3.Timing Analysis for encoding 4.Introduction to CUDA GPU 5.Overview of CUDA memory 6.Image mapping to memory grid 7.GPU grid computing 8.Steps of programming GPU 9.Proposed algorithm for parallel deblocking 10.Challenges 11.References

3. Acronyms AVC: Advanced Video Coding BS: Boundary Strength CODEC: COder/DECoder Croma: Chrominance CPU: Central Processing Unit CTU: Coding Tree Unit CU: Coding Unit CUDA: Compute Unified Device Architecture DBF: Deblocking Filter DCT: Discrete Cosine Transform DFT: Discrete Fourier Transform GPU: Graphics Processing Unit HEVC: High Efficiency Video Coding IEC: International Electrotechnical Commission ISO: International Standards Organization ITU-T: International Telecommunication Union (Telecommunication Standardization Sector) JBIG: Joint Bi-level Image Experts Group JPEG: Joint photographic experts group JCT-VC: Joint collaborative team on video coding LOT: Lapped Orthogonal Transform Luma: Luminance MB: Macro Block MPEG: Moving picture experts group OBMC: Overlapped Block Motion Compensation PU: Prediction Unit QP: Quantization Parameter SAO: Sample Adaptive Offset TU: Transform Unit

4. Objective and Action Plan Objective: To Implement the parallelization of Deblocking filter for HEVC CODEC using CUDA GPU. 1.Determine a way to program CUDA GPU. 2.Implement an algorithm for parallel processing of Deblocking filter operation. 3.Find the place to implement that algorithm at the appropriate location in HM code. 4.Execute the algorithm with both modes CPU only and CPU+GPU mode to compare the processing time and quality parameters. 5.Device a novel algorithm for the same.

5. Background study Analysis of Timing taken by Deblocking filter

6. Timing Analysis for encoding Comparison of encoding timings for different sequences[52]

7. Timing without Deblock filtering Sequence:waterfall_cif.yuv 352x288 FPS:25 No of frames:50 QP:32 Random access main configuration

8. Timing with Deblocking filter ON

9. Step 1 Way to program CUDA GPU

10. Introduction to CUDA GPU CUDA® (Compute Unified Device Architecture) is a parallel computing platform and programming model invented by NVIDIA. It enables dramatic increases in computing performance by harnessing the power of the graphics processing unit (GPU).[51] CUDA is widely used in general purpose computation, such as astronomical calculation, computational fluid dynamics simulation, image processing and video codec.

11. Contd… CUDA Programming model[52]

12. Contd… Thread view[54]

13. Overview of CUDA memory Memory interface[54]

14. Image to grid mapping 1x128 8 [53]

15. Steps of programming GPU [54]

16. GPU Grid Computing [53]

17. Contd… [53]

18. Step 2 Algorithm proposed for the parallel processing of Deblocking Filter

19. The proposed Algorithm for parallel deblocking [55]

20. Contd… [55]

21. Step 4 Incorporating GPU code with HM code

22. Challenges The challenge is to incorporate CUDA C code along with HM C++ code. GPU library follows C/C++ codes along with FORTRAN library but basically it is called CUDA C.

23. References 1.Andrey Norkin et al.,”HEVC Deblocking Filter”, IEEE Transactions on CSVT, vol. 22, no. 12, pp. 1746-1754, Dec. 2012 2.Wei-Yi Wei, “Deblocking Algorithms in Video and Image Compression Coding”, National Taiwan University, Taipei, Taiwan, ROC 3.B. Bross, et al., High Efficiency Video Coding (HEVC) Text Specification Draft 8, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVTC-J1003, Joint Collaborative Team on Video Coding (JCTVC), Stockholm, Sweden, Jul. 2012. 4.ITU-T and ISO/IEC JCT 1, Advanced Video Coding for Generic Audiovisual Services, ITU-T Rec. H.264 and ISO/IEC 14496-10 (AVC), May 2003 (and subsequent editions). 5.T. Wedi and H. G. Musmann, “Motion and aliasing compensated prediction for hybrid video coding,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 577–586, Jul. 2003. 6.P. List, et al., “Adaptive deblocking filter,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 614– 619, Jul. 2003. 7.K. Ugur, K. R. Andersson, and A. Fuldseth, Video Coding Technology Proposal by Tandberg, Nokia, and Ericsson, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-A119, Joint Collaborative Team on Video Coding (JCTVC), Dresden, Germany, Apr. 2010. 8.A. Norkin, et al., CE12: Ericsson’s and MediaTek’s Deblocking Filter, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-F118, Joint Collaborative Team on Video Coding (JCTVC), Turin, Italy, Jul. 2011. 9.M. Ikeda and T. Suzuki, Non-CE10: Introduction of Strong Filter Clipping in Deblocking Filter, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-H0275, Joint Collaborative Team on Video Coding (JCTVC), San Jose, CA, Feb. 2012. 10.M. Ikeda, J. Tanaka, and T. Suzuki, CE12 Subset2: Parallel Deblocking Filter, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-E181, Joint Collaborative Team on Video Coding (JCTVC),Geneva, Switzerland, Mar. 2011.

24. Contd… 11. M. Narroschke, S. Esenlik, and T. Wedi, CE12 Subtest 1: Results for Modified Decisions for Deblocking, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-G590, Joint Collaborative Team on Video Coding (JCTVC), Geneva, Switzerland, Nov. 2011. 12. A. Norkin, CE10.3: Deblocking Filter Simplifications: Bs Computation and Strong Filtering Decision, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-H0473, Joint Collaborative Team on Video Coding (JCTVC), San Jose, CA, Feb. 2012. 13. A. Fuldseth, et al., Tiles, ITU-TSG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-F335,Joint Collaborative Team on Video Coding (JCTVC), Turin, Italy, Jul. 2011. 14. T. Yamakage, et al.,CE12: Deblocking Filter Parameter Adjustment in Slice Level, ITUT SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVCG174,Joint Collaborative Team on Video Coding (JCTVC), Geneva, Switzerland, Nov. 2011 15. G. Van der Auwera,et al. (Panasonic), Support of Varying QP in Deblocking, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVCG1031,Joint Collaborative Team on Video Coding (JCTVC), Geneva, Switzerland, Nov. 2011. 16. M. Zhou, O. Sezer, and V. Sze, CE12 Subset 2: Test Results and Architectural Study on De-Blocking Filter Without Parallel on/off Filter Decision, ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 document JCTVC-G088, Joint Collaborative Team on Video Coding (JCTVC), Geneva, Switzerland, Nov. 2011. 17. G. Bjontegaard, Calculation of Average PSNR Differences Between RDCurves,ITU-T-T SG16 document VCEG-M33, Joint Collaborative Team on Video Coding (JCTVC), 2001. 18.F. Bossen, Common Test Conditions, JCTVC-H1100, Joint Collaborative Team on Video Coding (JCTVC), San Jose, CA, 2012. 19.Po-Kai Hsu and Chung-An Shen, The VLSI Architecture of a Highly Efficient Deblocking Filter for HEVC Systems, DOI 10.1109/TCSVT.2016.2515306, IEEE Transactions on Circuits and Systems for Video Technology 20.HEVC presentation: http://www.hardware.fr/news/12901/hevc-passe-ratifie.htmlhttp://www.hardware.fr/news/12901/hevc-passe-ratifie.html 21.Overview of H.264/AVC: http://www.csee.wvu.edu/~xinl/courses/ee569/H264_tutorial.pdfhttp://www.csee.wvu.edu/~xinl/courses/ee569/H264_tutorial.pdf 22.Detailed overview of HEVC/H.265: https://app.box.com/s/rxxxzr1a1lnh7709yvihhttps://app.box.com/s/rxxxzr1a1lnh7709yvih

25. Contd… 23. I.E.G. Richardson, “Video Codec Design: Developing Image and Video Compression Systems”, Wiley, 2002. 24. I.E.G. Richardson, “The H.264 advanced video compression standard”, 2nd Edition, Hoboken, NJ, Wiley, 2010. 25. K. Sayood, “Introduction to Data compression”, Third Edition, Morgan Kaufmann Series in Multimedia Information and Systems, San Francisco, CA, 2005. 26. V. Sze and M. Budagavi, “Design and Implementation of Next Generation Video Coding Systems (H.265/HEVC Tutorial)”, IEEE International Symposium on Circuits and Systems (ISCAS), Melbourne, Australia, June 2014. 27.V. Sze, M. Budagavi and G.J. Sullivan (Editors), “High Efficiency Video Coding (HEVC): Algorithms and Architectures”, Springer, 2014. 28.G. J. Sullivan et al, “Overview of the High Efficiency Video Coding (HEVC) Standard”, IEEE Trans. on Circuits and Systems for Video Technology, Vol. 22, No. 12, pp. 1649-1668, Dec. 2012. 29.G. J. Sullivan et al,“Standardized Extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of selected topics in Signal Processing, vol. 7, pp.1001-1016, Dec. 2013. 30.K.R. Rao, D.N. Kim and J.J. Hwang, “Video Coding Standards: AVS China, H.264/MPEG-4 Part 10, HEVC, VP6, DIRAC and VC-1”, Springer, 2014. 31.D. Grois, B. Bross and D. Marpe, “HEVC/H.265 Video Coding Standard (Version 2) including the Range Extensions, Scalable Extensions, and Multiview Extensions,” (Tutorial) Sunday 27 Sept 2015, 9:00 am to 12:30 pm), IEEE ICIP, Quebec City, Canada, 27 – 30 Sept. 2015. 32.Generic quadtree based approach for block partitioning http://www.hhi.fraunhofer.de/fields-of-competence/image-processing/research- groups/image-video-coding/hevchigh-efficiency-video-coding/generic-quadtree-based-approach-for-block-partitioning.htmlwww.hhi.fraunhofer.de/fields-of-competence/image-processing/research- groups/image-video-coding/hevchigh-efficiency-video-coding/generic-quadtree-based-approach-for-block-partitioning.html 33.The tutorial below is for personal use only [Password: a2FazmgNK ] https://datacloud.hhi.fraunhofer.de/owncloud/public.php?service=files&t=8edc97d26d46d4458a9c1a17964bf881 https://datacloud.hhi.fraunhofer.de/owncloud/public.php?service=files&t=8edc97d26d46d4458a9c1a17964bf881 34. Please find the links to YouTube videos on the tutorial - HEVC/H.265 Video Coding Standard including the Range Extensions Scalable Extensions and Multiview Extensions below: https://www.youtube.com/watch?v=TLNkK5C1KN8 34.HEVC tutorial by I.E.G. Richardson: http://www.vcodex.com/h265.htmlhttp://www.vcodex.com/h265.html 35.“Special issue on HEVC extensions and efficient HEVC implementations”, IEEE Trans. on Circuits and Systems for Video Technology, Vol. 26, pp. 1-249, Jan. 2016. 36.K.R. Rao and J.J. Hwang, “Techniques and standards for image/video/audio coding”, Prentice Hall, 1996.

26. Contd… 37.Video lectures from IITs and IISC: http://nptel.iitm.ac.in/http://nptel.iitm.ac.in/ 38.Image and video processing courses at UT Arlington (EE 5351, EE 5355, EE 5356 and EE 5359) : http://www.uta.edu/faculty/krrao/dip/http://www.uta.edu/faculty/krrao/dip/ 39.HEVC chapter 1: http://www.uta.edu/faculty/krrao/dip/Courses/EE5359/HEVCCH1a_updated.dochttp://www.uta.edu/faculty/krrao/dip/Courses/EE5359/HEVCCH1a_updated.doc 40.Online course on fundamentals of digital image and video processing from Coursera: https://www.coursera.org/course/digitalhttps://www.coursera.org/course/digital 41.Access to HM 16.0 Software Manual: http://iphome.hhi.de/marpe/download/Performance_HEVC_VP9_X264_PCS_2013_preprint.pdfhttp://iphome.hhi.de/marpe/download/Performance_HEVC_VP9_X264_PCS_2013_preprint.pdf 42.Test Sequences: ftp://ftp.kw.bbc.co.uk/hevc/hm-11.0-anchors/bitstreams/ftp://ftp.kw.bbc.co.uk/hevc/hm-11.0-anchors/bitstreams/ 43.HEVC white paper-Ittiam Systems: http://www.ittiam.com/Downloads/en/documentation.aspxhttp://www.ittiam.com/Downloads/en/documentation.aspx 44.HEVC white paper-Elemental Technologies: http://www.elementaltechnologies.com/lp/hevc-h265-demystified-white-paperhttp://www.elementaltechnologies.com/lp/hevc-h265-demystified-white-paper 45.Access to HM 16.0 Reference Software: http://hevc.hhi.fraunhofer.de/http://hevc.hhi.fraunhofer.de/ 46.Han W-J, et al. (2010), “Improved video compression efficiency through flexible unit representation and corresponding extension of coding tools”, IEEE Trans. on Circuits and Systems for Video Technology, Vol. 20, no.12, pp. 1709-1720, Dec. 2010. 47.Norkin A (2012) Non-CE1: non-normative improvement to deblocking filtering, Joint Collaborative Team on Video Coding (JCT-VC), Document JCTVC-K0289, Shanghai, Oct. 2012 48.Norkin A, Andersson K, Fuldseth A, Bjøntegaard G (2012) HEVC deblocking filtering and decisions. In: Proc. SPIE. 8499, Applications of Digital Image Processing XXXV, no. 849912, Oct. 2012 49. Norkin A, Andersson K, Kulyk V (2013) “Two HEVC encoder methods for block artifact reduction”. In: Proceedings of the IEEE international conference on visual communications and image processing (VCIP) 2013, Kuching, Sarawak, pp. 1–6, Nov. 2013 50.Norkin A, Andersson K, Sjöberg R (2013) AHG6: on deblocking filter and parameters signaling, Joint Collaborative Team on Video Coding (JCT-VC), Document JCTVC-L0232, Geneva, Jan. 2013 51.Information on GPU accelearted computing : http://www.nvidia.com/object/what-is-gpu-computing.htmlhttp://www.nvidia.com/object/what-is-gpu-computing.html 52.Xiaoou Sun et al, “Aceelerating IEEE 1857 Deblocking Filter on GPU using CUDA’, IEEE International Conference on Multimedia Big Data, pp. 415-419, Apr. 2015.

27. Contd… 53. Course on parallel computing: https://classroom.udacity.com/courses/cs344/lessons/55120467/concepts/967066740923https://classroom.udacity.com/courses/cs344/lessons/55120467/concepts/967066740923 54. Course on heterogeneous parallel programming https://class.coursera.org/hetero-004/wiki/Week_%231https://class.coursera.org/hetero-004/wiki/Week_%231 55. Anand Meher Kotra et al, “Comparison of different parallel implementations for deblocking filter of HEVC”, IEEE International conference on Acoustics, speech and signal processing, pp. 2721-2725, Mar. 2013. 56. Test sequence: http://trace.eas.asu.edu/yuv/http://trace.eas.asu.edu/yuv/

28. Thank you