Topics in Signal Processing Interim Report Sujatha Gopalakrishnan 1001024145

Basic Acronyms HEVC- High Efficiency Video Coding TU- Transform Unit PU- Prediction Unit CU- Coding Unit JCT-VC - Joint Collaborative Team on Video Coding Tmuc – Test Model under Consideration HM- HEVC Test Model DCT- Discrete Cosine Transform MC- Motion Compensation SAO- Sample Adaptive Offset ME- Motion estimation SAP- Sample based Angular Intra-Prediction NLM- Non-Local Means

HEVC High Efficiency Video Coding (HEVC) [1][2] is a Video Compression standard, a successor to H.264/MPEG-4 AVC [22]. HEVC is said to double the data compression ratio compared to H.264/MPEG-4 AVC [1] at the same level of video quality [2]. The design of most video coding standards is primarily aimed at having the highest coding efficiency. The main goal of the HEVC standardization effort is to enable significantly improved compression performance relative to existing standards, in the range of 50% bit rate reduction for equal perceptual video quality [10] [11].

Block Diagram HEVC Encoder Figure 1: HEVC Encoder [2]

Block Diagram HEVC Decoder Figure 2: HEVC Decoder [3]

HEVC Lossless Coding The lossless coding mode of HEVC main profile bypasses transform quantization and in-loop filters as shown in the fig.2 [4] [19]. Comparing it with non-lossless coding mode, it has smallest quantization parameter value. Lossless coding mode provides perfect fidelity and average bit rate reduction. Outperforms the existing lossless compression solution such as JPEG-2000 [22] and JPEG-LS [22]. It can prevent accumulation of quantization errors in repeated encoding and decoding operations of video editing

In this method it is essential to preserve numerical video data with fewer bits. DCT coefficients i.e., float-point numbers have to be quantized instead of DCT. Lossless video coding is used when perfect preservation of video data is required [29]. It employs Sample Angular-based Intra-Prediction (SAP) [4]. Same prediction mode signaling method. Same interpolation method of HEVC. Uses adjacent neighbors as reference shown in fig.8. Prediction residuals are coded with the entropy coder in the spatial domain [29].

Block Diagram HEVC Lossless Coding Figure 3: HEVC lossless Algorithm Block Diagram [4]

Algorithm of Sample Based Angular Intra Prediction The SAP [4] is designed to better exploit the spatial redundancy in the lossless coding mode by generating intra prediction samples from adjacent neighbors. The design principle here is very similar to the sample-based DPCM in [21] [4] H.264/MPEG-4 AVC [20] [4] lossless coding, but SAP is fully harmonized with the HEVC block-based angular intra prediction, and can be applied to all the angular intra prediction modes specified in HEVC [4]. As shown in the fig.7 SAP is performed sample by sample. The adjacent neighboring samples,  of the current sample   in the current PU are used for prediction.

SAP Algorithm Figure 4: Algorithm of SAP [4]

DCT for HEVC lossless compression DCT is applied to prediction residuals. DCT coefficients are quantized. Quantized DCT coefficients and quantization error are coded. Coding of each unit is performed by Adaptive Quantization parameters. Improved HEVC lossless compression using Two-Stage coding Block of residual signal is separated into two parts: Part 1: Quantized DCT coefficients. Part 2: Quantization error. Quantized coefficients are used to reconstruct a lossy decoded block which is subtracted from the residual block. Quantization error is encoded as the spatial block.

Figure 5: Two-stage lossless coding [30]

Table 1: Coded block for two-stage coding [30] Table 2: Performance of two-stage coding [30]

Pixel-based averaging predictor Lossless video compression of noisy video content can be improved if the noise within the video is considering the compression [32]. In HEVC lossless coding block-wise processing is not needed, pixel-wise prediction could be performed for better spatial correlation within the image or a video signal [31]. De-noised intra prediction scheme is used, where de-noising is performed by the predictor instead of removing the noise. Non-Local Means (NLM) algorithm is used for de-noising [33]. Pixel-wise prediction is the combination of linear predictors with exponentially decaying weights from NLM algorithm. Developed predictor results in a weighted average of surrounding pixels.

Figure 6: NLM algorithm for de-noising [32]. Other non-local predictors e.g., forward adaptive scheme where intra-frame motion compensation is performed [34] [32] or a backward adaptive scheme where template matching is performed for prediction [35] [36] [32], are designed for block-wise lossy prediction in H.264/AVC and thus are not efficient for lossless compression. Figure 6: NLM algorithm for de-noising [32].

NLM Algorithm The Non-Local Means (NLM) [31] algorithm has been introduced in [32] for image de-noising. In NLM de-noising, the estimate for a de-noised version of a noisy pixel is established by averaging all the pixels in the local as well as non-local neighborhood. The process of NLM [31] de-noising is illustrated in Fig 9. In the illustrated ex-ample, the pixel g[i] should be de-noised, where i=(x,y) is the two-dimensional coordinate. Therefore all pixels in the support area S are averaged depending on their similarity to g[i]. The similarity between the pixels is measured by a certain mean distance of the pixels in the surrounding area, which is illustrated with the square around the pixels g[j1], g[j2] and g[j3]. The averaging process is described by: ΡNLM[i] = ∑j∊s w[I,j], g[j] [31]

Figure 7: Casual patches for NLM predictor [31]

Low-Complexity Pixel wise Predictor Implementation Run time for prediction is proportional to the neighborhood size or the patch size. If the patch becomes larger, structural complexity of the patch becomes higher, so it becomes harder to find similar patches. Hence patch and neighborhood sizes are reduced in NLM predictor. Results of the investigated parameter constellations a, b, h, dSSE and dSAD are shown in Table 3 [32].

Table 3: Compression results of the proposed pixel-wise prediction [32]

Figure 8: Race horse sequence [29] Test Sequences Sequences are obtained from [29] and experimented to obtain the performance based on various parameters described as follows. Figure 8: Race horse sequence [29]

Figure 9: Basketball drill sequence [29]

BD- % Bit rate reductio n Test sequence 1: Test Sequence Resolution Frame rate (fps) RaceHorses_416x240_30.yuv 416 x 240 30 Test sequence   Intra Profile Random Access Profile BD- % Bit rate reductio n BD- PSNR Race horse sequence PSNR (dB) 34.2442 33.7342 -22.4269 1.483 Bitrate(kbps ) 1842.64 21 371.49 Encoding time(sec) 24.332 119.764 Decoding time(sec) 0.840 4.134

Test sequence 2: Test Sequence Resolution Frame rate (fps) BasketballDrill_832x480_50.yuv 832 x 480 50 Test sequence   Intra Profile Random Access Profile BD- % Bit rate reduction BD- PSNR Basketball drill PSNR (dB) 37.2947 35.7193 -32.8763 1.956 Bitrate(kbps) 5941.77 32 817.87 Encoding time(sec) 97.539 347.891 Decoding time(sec) 1.2 4.28

Project Results  BD-PSNR (dB) for the test sequences

BD-% Bit rate reduction for test sequences

Test sequence 1 [29]: Encoding and Decoding time(secs) for Intra and Random access profiles

Test sequence 2 [29]: Encoding and Decoding time(secs) for Intra and Random access profiles

Future Work Future simulations will be conducted using the HM 16.0 [16] software for other test sequences [29]; results will be plotted and compared for the best results using the parameters of PSNR, Bit rate, encoding and decoding time. Also BD-PSNR and BD-%bit rate reduction values will be plotted. Theoretical analysis will be carried out on sample-based angular prediction for HEVC lossless coding.

References [1] G.J. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Trans. CSVT, vol. 22, pp.1649-1668, Dec. 2012. [2] G.J. Sullivan et al, “Standardized Extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of selected topics in Signal Processing, vol.7, pp.1001-1016, Dec. 2013.   [3] C. Fogg, “Suggested figures for the HEVC specification”, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC) document JCTVC- J0292r1, July. 2012. [4] M. Zhou et al, “HEVC lossless coding and improvements”, IEEE Trans. CSVT, vol.22, pp.1839-1843, Dec. 2013. [5] N. Purnachand et al, "Fast Motion Estimation Algorithm for HEVC", IEEE Second International Conference on Consumer Electronics-Berlin (ICCE-Berlin), vol.11, pp.34-37, Sep. 2012.

[6] P. Hanhart et al, “ Subjective quality evaluation of the upcoming HEVC video compression standard”, SPIE Optical Engineering+ Applications, International Society for Optics and Photonics, vol. 8499, pp.84990v-84990v, Aug. 2012. [7] M. Horowitz et al, “Informal subjective quality comparison of video compression performance of the HEVC and H.264/MPEG - 4 AVC standards for low delay applications”, SPIE Optical Engineering+ Applications. International Society for Optics and Photonics, vol.84990, pp.84990w-84990w, Aug. 2012. [8] A. Abdelazim, W. Masri and B. Noaman, "Motion estimation optimization tools for the emerging high efficiency video coding (HEVC)", IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, vol. 9029, pp. 902905-902905, Feb. 2014. [9] M. Jakubowski and G. Pastuszak, “Block-based motion estimation algorithms-a survey”, Journal of Opto-Electronics Review, vol. 21, pp.86-102, Mar. 2013.

[10] B. Bross et al, “High Efficiency Video Coding (HEVC) Text Specification Draft 10”, Document JCTVC-L1003, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC), Jan. 2013, available on ​http://phenix.it-sudparis.eu/jct/doc_end_user/current_document.php?id=7243 [11] J. Ascenso et al, "Improving Frame Interpolation with Spatial Motion Smoothing for Pixel Domain Distributed Video Coding", 5th EURASIP Conference on Speech and Image Processing, Multimedia Communications and Services, pp.1-6, Smolenice, Slovak Republic, July. 2005.  [12] X. Wang et al, “Paralleling Variable Block Size Motion Estimation of HEVC on Multicore CPU plus GPU platform”, IEEE International Conference on Image Processing (ICIP), vol.22, pp. 1836-1839, Sep. 2013. [13] Introduction to parallel computing https://computing.llnl.gov/tutorials/parallel_comp/#Whatis

[14] L. Zhao et al, “Group-Based Fast mode decision algorithm for intra prediction in HEVC”, IEEE Eighth international Conference on Signal Image Technology and Internet based Systems. Article no.6115979, pp. 225-229, Nov. 2011. [15] V. Sze and M. Budagavi, "High Throughput CABAC Entropy Coding in HEVC", IEEE Transactions on Circuits and Systems for Video Technology, vol.22, no.12, pp.1778-1791, Dec 2012. [16] T.Nguyen et al, "Transform Coding Techniques in HEVC", IEEE Journal of Selected Topics in Signal Processing, vol.7, pp.978–989, Dec. 2013. [17] HEVC tutorial by I.E.G. Richardson: http://www.vcodex.com/h265.html [18] HEVC Reference Software HM16.0. https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.0rc1/

[19] B. Bross et al,“High Efficiency Video Coding (HEVC)Text Specification Draft 8”, JCT-VC document, JCTVC-J1003, Stockholm, Sweden, Jul. 2012. http://phenix.it-sudparis.eu/jct/doc_end_user/current_document.php?id=6465 [20] Joint Video Team, “Advanced Video Coding for Generic Audiovisual Services”, ITU-T Rec. H.264 and ISO/IEC, 14496-10 (MPEG-4) AVC, pp.H.100-H.869, Feb. 2014. [21] Y.L. Lee et al, "Improved lossless intra coding for H.264/MPEG-4 AVC", IEEE Trans. Image Process., vol.15, no.9, pp.2610-2615, Sep 2006. [22]K.R. Rao, D.N Kim and J.J Hwang, “High Efficiency Video Coding (HEVC) Revised/Updated Chapter from the book Video Coding Standards”–Springer 2014. [23] ITU-T website: http://www.itu.int/ITU-T/index.html [24] JCT-VC documents are publicly available at http://ftp3.itu.ch/av-arch/jctvc-site and http://phenix.it-sudparis.eu/jct/

[25] V. Sze, M. Budagavi, and G. J [25] V.Sze, M. Budagavi, and G.J. Sullivan, “High Efficiency Video Coding (HEVC) Algorithms and Architectures” Springer, 2014. [26] Software reference manual for HM: https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/branches/HM-9.2-dev/doc/software-manual.pdf          [27] M. Wien, “High efficiency video coding: Tools and specification”, Springer, 2015. [28] I.E. Richardson, “Coding video: A practical guide to HEVC and beyond”, Wiley, 11 May 2015. [29] Video Sequences: http://ultravideo.cs.tut.fi/ http://forum.doom9.org/archive/index.php/t-135034.html [30] C. Xun and Q. Gu, "Improved HEVC lossless compression using Two-Stage coding with Sub-Frame level optimal quantization values", IEEE International Conference. Image Processing (ICIP), vol.23, pp. 5651-5655, Oct. 2014.

[31] W. Eugen et al, "Pixel-based averaging predictor for HEVC lossless coding", IEEE International Conference. Image Processing (ICIP), vol.23, pp. 1806-1810, Sept 2013.   [32] E. Wige et al, "In-loop denoising of reference frames for lossless coding of noisy image sequences" IEEE International Conference. Image Processing (ICIP), vol.19, pp. 461-464, Sept 2010. [33] A. Buades, B. Coll, and J.M. Morel, "A non-local algorithm for image denoising", IEEE Computer Society Conference. Computer Vision and Pattern Recognition (CVPR), vol.2, pp. 60-65, June 2005. [34] S.L. Yu and C. Chrysafis,” New intra prediction using intra-macroblock motion compensation”, Joint Video Team (JVT) of ISO/IEC MPEG &ITU-T VCEG (ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Std., doc. JVT-C151, Virginia, USA, Apr.2004. [35] T. K. Tan, C. S. Boon, and Y. Suzuki, "Intra prediction by template matching", IEEE International Conference. Image Processing (ICIP), vol.15, pp. 1693-1696, Oct. 2006.

[36] T.K. Tan et al, "Intra Prediction by Averaged Template Matching Predictors", IEEE. Consumer Communications and Networking Conference (CCNC), vol. 16, pp. 405-409, Jan 2007. [37] V. Sze, M. Budagavi, “Design and Implementation of Next Generation Video Coding Systems (H.265/HEVC Tutorial),” IEEE International Symposium on Circuits and Systems (ISCAS), presented on June. 2014. http://www.rle.mit.edu/eems/publications/tutorials/