PROJECT INTERIM REPORT HEVC DEBLOCKING FILTER AND ITS IMPLEMENTATION RAKESH SAI SRIRAMBHATLA rakeshsai.srirambhatla@mavs.uta.edu UTA ID: 1000869391 EE 5359 Under the guidance of DR. K. R. RAO

Contents Objective HEVC overview Video compression standards Working of HEVC Deblocking filter SAO filter Scope of the project Acronyms References

OBJECTIVE Scope of this project is to primarily focus on the study of in-loop deblocking filter in HEVC. Besides this study, modification is done on the deblocking filter which results in the performance improvement of the codec. To compare the HEVC performance after implementing in HM codec with H.264/AVC codec [22] using processor clock cycle, mean square error (MSE) and PSNR [25]and BD bitrate [26] as metrics.

HEVC OVERVIEW High Efficiency Video Coding (HEVC) is finalized as the newest video coding standard of the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group. [2] Joint collaborative team on video coding (JCT-VC) developed HEVC video coding standard. JCT-VC is a joint partnership team between video coding experts group (VCEG) and moving picture experts group (MPEG). [2] Main , main 10 and main intra profiles have been finalized in 2013. Extensions such as 3D video, scalable HEVC, 4:4:4, 4:2:2 formats are to be finalized in 2014. [3]

HEVC has many advantages over previous coding standards H HEVC has many advantages over previous coding standards H.264 [22] and latter versions, like significant rate distortion efficiency achieving higher bit rate reduction, error resilience and is most network friendly compared to other standards. [2] The main goal of the HEVC standardization effort is to enable significantly improved compression performance relative to existing standards—in the range of 50% bit-rate reduction for equal perceptual video quality. [2]

While H.264/AVC[22] divides a picture into fixed size macroblocks of 16×16 samples, HEVC divides a picture into coding tree units (CTU) of 16 × 16, 32 × 32 or 64 × 64 samples. The coding tree units can be further divided into smaller blocks using a quadtree structure; such a block, called a coding unit (CU), can further be split into prediction units (PUs) HEVC has been designed to address essentially all existing applications of H.264/MPEG-4 AVC [22] and to particularly focus on two key issues: increased video resolution and increased use of parallel processing architectures.

Subdivision of a CTB into CBs [and transform block (TBs)]. Solid lines indicate CB boundaries and dotted lines indicate TB boundaries. CTB with its partitioning. (b) Corresponding quadtree. [2] (a) [b]

HIGH EFFICIENCY VIDEO CODING VIDEO COMPRESSION STANDARDS STANDARD SOFTWARE MAIN APPLICATION YEAR JPEG2000 JasPer [21] IMAGE 2000 H.264/MPEG-4 part 10 JM software[22] ADVANCED VIDEO CODING 2003 H.265 HM Software [23] HIGH EFFICIENCY VIDEO CODING 2013

GENERATIONS OF VIDEO CODING STANDARDS [6]

H.264 functional block diagram [28]

BLOCK DIAGRAM OF HEVC ENCODER [2]

HEVC DECODER BLOCK DIAGRAM [27]

GOP - Group Of Pictures ( Video Frames ). [7] I-Intra Frame : P – Predicted frame : B – Bidirectional predicted frame Order in which different types of frames are placed makes a GOP structure A classic GOP structure is shown below

STRUCTURE OF HEVC CODEC [5]

Encoder Partitioning each unit into multiple units. Predicting each unit using intra or inter prediction and subtracting the prediction from the unit Transforming and quantizing the residual. Entopy encoding the transform output, prediction information and headers. Decoder Entropy decoding and extracting the elements from the coded sequence. Rescaling and inverting the transform stage. Predicting each unit and adding the prediction to the output of the inverse transform. Reconstructing a decoded video image.

DEBLOCKING FILTER AND ITS WORKING The deblocking filter performs detection of the artifacts at the coded block boundaries and attenuates them by applying a selected filter. Compared to the H.264/AVC deblocking filter, the HEVC deblocking filter has lower computational complexity and better parallel processing capabilities while still achieving significant reduction of the visual artifacts. [13] In HEVC, two processing steps, namely a deblocking filter (DBF) followed by an SAO filter, are applied to the reconstructed samples before writing them into the decoded picture buffer in the decoder loop. The deblocking filter is intended to reduce the blocking artifacts due to block-based coding.

Sample adaptive offset filter A nonlinear amplitude mapping is introduced within the interpicture prediction loop after the deblocking filter. Its goal is to better reconstruct the original signal amplitudes by using a look-up table that is described by a few additional parameters that can be determined by histogram analysis at the encoder side. The key idea of SAO is to reduce sample distortion by first classifying reconstructed samples into different categories, obtaining an offset for each category, and then adding the offset to each sample of the category. The latter consideration is important since the deblocking filter of the H.264/AVC [22] standard, constitutes a significant part of the decoder complexity. [2]

The deblocking filter is applied to all samples adjacent to a PU or TU boundary except the case when the boundary is also a picture boundary Unlike H.264/MPEG-4 AVC [22], where the deblocking filter is applied on a 4×4 sample grid basis, HEVC applies the deblocking filter only to the edges that are aligned on an 8×8 sample grid, for both the luma and chroma samples.

Schematic showing the edges of Prediction unit, Transform unit and picture boundary [11] V- vertical edge H- horizontal edge

Test sequences [18]

Basketball.yuv [18] Picture of test sequence Basketball.yuv with resolution 832×480 with the frame rate = 50 frames/sec.

BQMall.yuv [18] Picture of test sequence BQmall.yuv with resolution 832×480 with the frame rate = 60 frames/sec.

Racehorses.yuv [18] Picture of test sequence Racehorses.yuv with resolution 416×240 with the frame rate = 30 frames/sec.

Kirsten&Sara.yuv [18] Picture of test sequence Kirsten&Sara.yuv with resolution 1280×720 with the frame rate = 60 frames/sec.

Scope of this project To study deblocking filter operation in HEVC and implement for reduction of blocking visual artifacts using the filter. To compare the performance of HEVC after enabling the deblocking filter and SAO (sample adaptive offset) with a default HEVC setting using the metrics : complutaional time, BD bitrate and PSNR [25][26]. To compare the HEVC performance in HM codec with H.264/AVC [22] codec performance using metrics: BD bitrate [26], BD PSNR [25] and MSE.

Implementation Test sequence and test environment details In the initial implementation, Kirsten&Sara.yuv [18] is considered as the test sequence. Picture of test sequence Kirsten&Sara.yuv [18] with resolution 1280×720 with the frame rate = 60 frames/sec. Total of 10 frames are considered for the test. HM.13 HEVC codec software has been used for execution of the test. HEVC main profile is considered for the initial implementation of encoder.

Deblocking filter implementation Enable/Disable Deblocking Filter [23] LoopFilterDisable parameter – under Deblock Filter – alone would not do the job of disabling the deblocking filter [23] Have to use combination of following deblocking filter parameters: [23] DeblockingFilterControlPresent LoopFilterDisable LoopFilterBetaOffset_div2 LoopFilterTcOffset_div2

Metrics for the test BD PSNR is plotted for all the test sequences for different quantization parameters. Total encoding time is plotted for all the test sequences for different quantization parameters. BD bitrate is also plotted for all the test sequence for different quantization parameters. [1]

Results and observations Tables Table 1 represents the metrics when the deblocking filter disabled. Kristen & Sara Bitrate Computational Time PSNR QP = 32 7899.744 Kbps 266.74 secs 38.879 Table 1 Table 2 represents the metrics when the deblocking filter is enabled. Kristen & Sara Bitrate Computational Time PSNR QP = 32 7899.744 Kbps 266.74 secs 38.879 Table 2

Computational time (in seconds) Kirsten&Sara.yuv [18] with resolution 1280×720 with the frame rate = 60 frames/sec. 10 frames are considered.

BD Bitrate ( in Kbps) [26] Kirsten&Sara.yuv [18] with resolution 1280×720 with the frame rate = 60 frames/sec. 10 frames are considered.

BD PSNR ( in dB) [25] Kirsten&Sara.yuv [18] with resolution 1280×720 with the frame rate = 60 frames/sec. 10 frames are considered.

Conclusion The computational time for the test sequence has increased from 266.754secs to 294.684secs when the deblocking filter was applied to the encoder since the complexity has increased because of the addition of deblocking filter to the elements of the HEVC block. PSNR and bitrate are the performance metrics. Both PSNR and bitrate have increased with the addition of deblocking filter which shows improvement in performance of the codec. In the final implementation, these metrics are calculate for main and low delay profiles and HEVC deblocking filter performance would be compared with the deblocking filter in H.264. All the tests are repeated for four test sequences and an aggregated result will be given in the final report

Acronyms AVC Advanced Video Coding ALF Adaptive loop filter B Bi-predictive CB Coding Block DBF Deblocking filter CSVT Circuits and Systems for Video Technology CTB Coding Tree Block CTU Coding Tree Unit CU Coding Unit GOP Group of pictures HEVC High Efficiency Video Coding I Intra frame ISO International Standardization Organization ITU-T International Telecommunication Union - Telecommunication Standardization Sector JCT-VC Joint Collaborative Team on Video Coding LCU – Largest coding unit MPEG Moving Picture Experts Group MSE – Mean square error PB Prediction Block PSNR – Peak signal to noise ratio QP – Quantization parameter SAO – Sample adaptive offset  SNR – Signal to noise ratio  SPS – Sequence parameter set TB Transform Block TU Transform Unit VCEG Video Coding Experts Group

REFERENCES [1] G.J Sullivan et al, “Overview of the H.264/AVC Video Coding Standard”, IEEE Trans. CSVT, vol.13, pp. 560-576, July 2003 [2] G.J. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Trans. CSVT, vol.22, pp. 1649-1668, Dec. 2012 [3] G.J. Sullivan et al, “Standardized extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of Selected topics in signal processing, vol. 7, pp. 1001-1016, Dec. 2013 [4] G.J. Sullivan et al, “Comparison of the Coding Efficiency of Video Coding Standards-Including High Efficiency Video Coding (HEVC)”, IEEE Trans. CSVT, Vol. 22, pp. 1669-1684, Dec.2012

[5] http://www. vcodex. com/images/uploaded/342512928230717 [5] http://www.vcodex.com/images/uploaded/342512928230717.pdf , URL for document on introduction to HEVC [6 ] http://en.wikipedia.org/wiki/Scalable_Video_Coding Wikipedia URL for scalable video coding which has the definitions of video coding standards. [7 ] http://www.youtube.com/watch?v=f9n-7mgNsNQ URL for NPTEL video on basics of video coding standards. [8] http://en.wikipedia.org/wiki/Inter_frame Wikipedia search URL for types of frames in video compression. [10] N. Ling, “High efficiency video coding and its 3D extension: A research perspective," 7th IEEE, Trans. on ICIEA, pp. 2150-2155, July 2012. [11] M. Li et al, “De-blocking Filter Design for HEVC and H.264/AVC”, PCM 2012, Lecture Notes in Computer Science 7674, pp. 273–284, 2012.   [12] M. Naccari et al, “Low Complexity Deblocking Filter Perceptual Optimization For The HEVC Codec”, 18th IEEE International Conference on Image Processing, pp. 737-740, 2011.

  [13] A. Norkin et al, “HEVC Deblocking Filter”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 22, No. 12, pp. 1746-1754, Dec. 2012. [14] JM software download for H.264/AVC: http://iphome.hhi.de/suehring/tml/   [15] HM codec download for H.265: https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/branches [16] HEVC standard test video sequences: ftp://ftp.tnt.uni-hannover.de/testsequences [18] Test sequence source : http://media.xiph.org/video/derf/ [19] HEVC Reference Software HM13.0. https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-13.0rc1/ [20] 10.2 JM H.264 software http://iphome.hhi.de/suehring/tml/ [21] JPEG2000 latest reference software (jasper version 1.900.0) Website: http://www.ece.uvic.ca/~mdadams/jasper/

[22] JM reference software, Fraunhofer institute for telecommunications Heinrich Hertz institute. http://iphome.hhi.de/suehring/tml/. [23] HM reference software, Fraunhofer institute for telecommunications Heinrich hertz institute.http://hevc.hhi.fraunhofer.de/. [24] C. M Fu et al, “Sample Adaptive Offset in the HEVC Standard”, IEEE Trans. CSVT, Vol.22, pp. 1755 – 1764, Dec 2012. [25] PSNR reference http://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio [26] BD bitrate reference Open source article, “Bit rate”. [Online]. Available: http://en.wikipedia.org/wiki/Bit_rate [27] B. Bross et al, “High efficiency video coding (HEVC) text specification draft 10”, 12th Meeting Geneva January 2013. [Online]. Available: http://phenix.it-sudparis.eu/jct/doc_end_user/current_document.php?id=7243 [28] Yu-Wen Huang et al, “Architecture design for deblocking filter in H.264/JVT/AVC”, 2003 International Conference on Multimedia and Expo, 2003. ICME '03. Proceedings.  Pages I - 693-6, vol.1, July 2003

[29] K.R. Rao, D.N. Kim and J.J. Huang, ” Video coding standards”, Springer, 2014. [30] Special issue on emerging research and standards in next generation video coding, IEEE transactions on circuits and systems for video technology (CSVT), vol. 22, pp. 1646-1909, Dec. 2012. [31] Special issue on emerging research and standards in next generation video coding, IEEE transactions on circuits and systems for video technology (CSVT), vol. 22, pp. 1646-1909, Dec. 2013. [32] IEEE Journal of Selected Topics in Signal Processing, vol. 7, pp.931-1151, Dec. 2013.