EE5359 Multimedia Processing Final Presentation SPRING 2015 ADVISOR: Dr. K.R.Rao EE5359 Multimedia Processing1 BY: BHARGAV VELLALAM SRIKANTESWAR UTA ID – Submission date – 11th May 2015

 3D - Three Dimension  AVC - Advanced Video Coding  AMVP - Advanced motion vector prediction  CTU - Coding Tree  CU - Coding Unit  CABAC - Context Adaptive Binary Arithmetic Coding  DCT- Discrete Cosine Transform  FPS – Frames Per Second  GOP - Group Of Pictures  HD - High Definition EE5359 Multimedia Processing2

 HEVC - High Efficiency Video Coding  ISO - International Organization for Standardization  MCTF - Motion Compensated Temporal Filtering  MPEG - Moving Pictures Experts Group  MSE - Mean Square Error  MV - Motion Vector  POC - Phase-Only Correlation  PB – Prediction Block  PU - Prediction Unit  QP - Quantization Parameter EE5359 Multimedia Processing3

 RD – Rate Distortion  SAO - Sample Adaptive Offset  SSIM – Structural Similarity Index Metrics  TB - Transform Block  TU- Transform Unit  VCEG - Video Coding Experts Group EE5359 Multimedia Processing4

1. Introduction to HEVC 2. About Block Artifacts 3. Implementation 4. Comparison metrics 5. Deblocking Parameters used in the Project 6. Profile used and testing platform 7. Configuration of HM Test sequences 9. Results 10. Conclusion and Future work 11. References EE5359 Multimedia Processing5

 HEVC is the latest video coding standard jointly presented by ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group[1].  The best performance improvement of HEVC over H.264 is ~50% bit rate reduction for equal perceptual video quality[1].  The growing popularity of HD video, and the emergence of beyond-HD formats (e.g.4kx2k or 8kx4k resolution) are creating even stronger needs for coding efficiency superior to H.264/MPEG-4 [4] AVC's capabilities[4]. EE5359 Multimedia Processing6

7 Figure 1: Block diagram of HEVC Encoder[9]

 Partitioning each picture into multiple units  Predicting each unit using inter or intra prediction, and subtracting the prediction from the unit  Transforming and quantizing the residual (the difference between the original picture unit and the prediction)  Entropy encoding transform output, prediction information, mode information and headers[17] EE5359 Multimedia Processing8

9 Figure 2: HEVC Decoder[10]

 Entropy decoding and extracting the elements of the coded sequence  Rescaling and inverting the transform stage  Predicting each unit and adding the prediction to the output of the inverse transform  Reconstructing a decoded video image [17] EE5359 Multimedia Processing10

 A distortion that appears in compressed video material as abnormally large pixel blocks[6].  Occurs when the encoder cannot keep up with the allocated bandwidth[6].  Visible with fast motion sequences or quick scene changes[6]. EE5359 Multimedia Processing11

EE5359 Multimedia Processing12 Figure 3 : example showing macro blocks in a image[15]

 In a coding scheme that uses block-based prediction and transform coding, discontinuities can occur in the reconstructed signal at the block boundaries [17].  Visible discontinuities at the block boundaries are blocking artifacts [17]. EE5359 Multimedia Processing13

EE5359 Multimedia Processing14 Figure 4 : Block boundary with blocking artifact[17]

 The HEVC de-blocking filter significantly improves the subjective quality of coded video sequences at lower bitrates[6].  Reference software encoder may produce visible block artifacts on some sequences with content that shows chaotic motion, such as water or fire[6]. EE5359 Multimedia Processing15

 Analyzed the reasons for blocking artifacts in various sequences  Two simple encoder-side methods are implemented that improve the subjective quality.  0 to 1% increase in bit rate. EE5359 Multimedia Processing16

In order to attenuate block artifacts in a picture with higher depth, HEVC can be configured to signal the deblocking filter offsets at the slice/picture level. Higher offsets are sent for the frames, which are at higher depth in the coding hierarchy. The proposed approach relaxes the deblocking decisions thresholds and clipping values for the pictures at higher depth [6]. EE5359 Multimedia Processing17

 When the rate-distortion optimization chooses 32 × 32 intra-predicted CUs at higher depth, the prediction is often coarse.  It might be difficult to conceal a blocking artifact by just applying the deblocking filtering.  By limiting the maximum TU size to 16 × 16 samples for coding of intra CUs in inter-predicted slices.  This will restrict the maximum intra-predicted block size and will increase the bit rate [6]. EE5359 Multimedia Processing18

 PSNR – Peak Signal to Noise Ratio  Computational time  BD- Bit rate  BD - PSNR EE5359 Multimedia Processing19

 tc_offset_div 2 In-loop deblocking filter parameter tc_Offset_Div2 is added to the base parameter LoopFilterTcOffset div2 to set the final tc_offset_div2 parameter for this picture signalled in the slice segment header. The final value of tc_offset_div2 shall be an integer number in the range - 6 to +6 [8]. EE5359 Multimedia Processing20

 beta_offset_div2 In-loop deblocking filter parameter beta_Offset_Div2 is added to the base parameter LoopFilterBetaOffset div2 to set the final beta_offset_div2 parameter for this picture signalled in the slice segment header. The final value of beta_offset_div2 shall be an integer number in the range-6 to +6 [8]. EE5359 Multimedia Processing21

 Rate Distortion (RD) penalty RD-penalty for 32x32 TU for intra in non-intra slices. Enabling this parameter can reduce the visibility of CU boundaries in the coded picture [8].  0 No RD-penalty  1 RD-penalty  2 Maximum RD-penalty (no 32x32 TU). EE5359 Multimedia Processing22

 The HM 16.4 main profile [8], This profile allows for a bit depth of 8-bits per sample with 4:2:0 chroma sampling, which is the most common type of video used with consumer devices.  TESTING PLATFORM Processor : Intel(R) core(TM) i5-4200,2.30 GHz Memory : 6 GB Operating system : 64 bit windows 8 EE5359 Multimedia Processing23

 The project is implemented in Random Access configuration of HM 16.4  Profile : main  IntraPeriod : 1 # Period of I-Frame ( -1 = only first)  GOPSize : 8 # GOP Size (number of B slice = GOPSize-1)  QP : 32 # Quantization parameter(0-51)  FastSearch 1 # 0:Full search 1:TZ search  SearchRange : 64 # (0: Search range is a Full frame) EE5359 Multimedia Processing24

EE5359 Multimedia Processing25 Figure 5: Hierarchical-B coding structure with GOP8 illustrating different depths of the picture in the coding structure [6]

EE5359 Multimedia Processing26 The following test sequences are used for analyzing the project [11]

EE5359 Multimedia Processing27 Figure 6: RaceHorses_416x240_30.yuv [11] Figure 7: ParkScene_1920x1080_24.yuv [11] Figure 8: Kimono_1920x1080_24.yuv [11]

EE5359 Multimedia Processing28 Figure 9: football30_cif_90.yuv [11]Figure 10: BasketballDrill_832x480_50.yuv [11

EE5359 Multimedia Processing29 Figure 11: ParkScene_1920x1080_24.yuv Without reduction of block artifacts Figure 12: ParkScene_1920x1080_24.yuv with reduction of block artifacts Figure 11 and Figure 12 are encoded sequences for a QP of 42 without and with reduction of block artifacts respectively.

EE5359 Multimedia Processing30 Table 1: Results for RaceHorses_416x240_30.yuv sequence in Random Access Configuration

EE5359 Multimedia Processing31 Figure 13 : RD- Plot for RaceHorses_416x240_30.yuv

EE5359 Multimedia Processing32 Figure 14 : QP vs PSNR comparison plot for RaceHorses_416x240_30.yuv

EE5359 Multimedia Processing33 Figure 15 : QP vs Encoding time comparison Plot for RaceHorses_416x240_30.yuv

EE5359 Multimedia Processing34 Table 2: Results for ParkScene_1920x1080_24.yuv sequence in Random Access Configuration

EE5359 Multimedia Processing35 Figure 16 : RD- Plot for ParkScene_1920x1080_24.yuv

EE5359 Multimedia Processing36 Figure 17: QP vs PSNR comparison Plot for ParkScene_1920x1080_24.yuv

EE5359 Multimedia Processing37 Figure 18: QP vs Encoding time comparison Plot for ParkScene_1920x1080_24.yuv

EE5359 Multimedia Processing38 Table 3: Results for Kimono_1920x1080_24.yuv sequence in Random Access Configuration

EE5359 Multimedia Processing39 Figure 19: RD-Plot for Kimono_1920x1080_24.yuv

EE5359 Multimedia Processing40 Figure 20: QP vs PSNR comparison plot for Kimono_1920x1080_24.yuv

EE5359 Multimedia Processing41 Figure 21: QP vs Encoding time comparison plot for Kimono_1920x1080_24.yuv

EE5359 Multimedia Processing42 Table 4: Results for football30_cif_90.yuv sequence in Random Access Configuration

EE5359 Multimedia Processing43 Figure 22: RD-Plot for football30_cif_90.yuv

EE5359 Multimedia Processing44 Figure 23: QP vs PSNR comparison plot for football30_cif_90.yuv

EE5359 Multimedia Processing45 Figure 24: QP vs Encoding time comparison Plot for football30_cif_90.yuv

EE5359 Multimedia Processing46 Table 5: Results for BasketballDrill_832x480_50.yuv sequence in Random Access Configuration

EE5359 Multimedia Processing47 Figure 25: RD-Plot for BasketballDrill_832x480_50.yuv

EE5359 Multimedia Processing48 Figure 26: QP vs PSNR comparison plot for BasketballDrill_832x480_50.yuv

EE5359 Multimedia Processing49 Figure 27: QP vs Encoding time comparison Plot for BasketballDrill_832x480_50.yuv

EE5359 Multimedia Processing50

EE5359 Multimedia Processing51 Figure 28: BD-PSNR Comparison with and without blocking artifacts for QP=32

EE5359 Multimedia Processing52 Figure 29: %BD-Bit rate comparison with and without blocking artifacts for QP=32

 RD plots infer that after the blocking artifacts are reduced in a sequence, PSNR increases where as Bit rate also increases. This is a drawback.  As the Quantization Parameter is increased the encoding time decreases and PSNR also decreases.  BD-Bit rate plot indicates that there is 0.578% increase in average BD-bit rate with reduction of blocking artifacts.  BD-PSNR indicates that there is an average of 1.67dB increase in BD-PSNR with reduction of blocking artifacts.  Future work has to be done to decrease the bit rate and encoding time. EE5359 Multimedia Processing53

1. G.J. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Transactions on CSVT, vol. 22, pp , Dec P. Hanhart et al, “Subjective quality evaluation of the upcoming HEVC video compression standard” SPIE Applications of digital image processing XXXV, vol.8499, pp , Aug G J. Sullivan et al, “Standardized Extensions of HEVC”, IEEE Journal of Selected topics in Signal Processing, Vol.7, no.6, pp , Dec F. Bossen et al, “HEVC Complexity and Implementation Analysis”, IEEE Transactions on CSVT, vol.22, pp , Dec HEVC white paper-Ateme: 6. A. Norkin et al, "Two HEVC encoder methods for block artifact reduction ", IEEE International Conference on Visual Communications and Image Processing (VCIP) article no , pp.1-6,Nov EE5359 Multimedia Processing54

7. L. Zhao et al, “Fast mode decision algorithm for intra prediction in HEVC”, IEEE International Conference on Visual Communications and Image Processing Conference (VCIP) article no , pp. 6-9 Nov HEVC Reference Software HM D. Marpe,T.Weigand and G.J.Sullivan, “The H.264/MPEG4 advanced video coding standard and its applications”, IEEE Communications Magazine, Vol. 44, pp , Aug C. Fogg, “Suggested figures for the HEVC specification”, ITU-T / ISO-IEC Document: JCTVC J0292r1, July Required test sequences V. Sze and M. Budagavi, ”High Throughput CABAC Entropy Coding in HEVC”, IEEE Transactions on CSVT, Vol.22, pp, , Dec T.Nguyen et al, "Transform Coding Techniques in HEVC" IEEE Journal of Selected Topics in Signal Processing,Vol.7, pp. 978–989, Dec EE5359 Multimedia Processing55

14. W.Y.Wei, "Deblocking Algorithms in Video and Image Compression Coding" Graduate Institute of Communication Engineering, National Taiwan University, Taipei, Taiwan, ROC. 15. A. Norkin et al, “HEVC Deblocking Filter”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 22, No. 12, pp , Dec W.Shen et al, “A high-throughput VLSI architecture for deblocking filter in HEVC” IEEE International Symposium on Circuits and Systems (ISCAS), pp , May Access to HEVC tutorial by I.E.G. Richardson: I. E.G. Richardson, “The H.264 Advanced Video Compression Standard”, 2nd Edition, Wiley N.Ahmed, T.Natarajan and K.R.Rao, " Discrete Cosine Transform ", IEEE Transactions on Computers, Vol 23, PP , Jan EE5359 Multimedia Processing56

20. Access to HM 16.4 software manual: manual.pdfhttp://hevc.kw.bbc.co.uk/svn/jctvc-a124/tags/HM-16.4/doc/software- manual.pdf 21. V.Sze, M,Budagavi and G.J.Sullivan, "High Efficiency Video Coding (HEVC): Algorithms and Architectures", Springer I.E.G. Richardson, “Video Codec Design: Developing Image and Video Compression Systems”, Wiley, M.Wein, "High Efficiency Video Coding, Coding Tools and Specification”, Springer X. Li et al, “Rate-complexity-distortion evaluation for hybrid video coding”, IEEE International Conference on Multimedia and Expo (ICME), pp , July G. Bjontegaard, “Calculation of Average PSNR Differences between RD Curves”, document VCEGM33, ITU-T SG 16/Q 6, Austin, TX, Apr I.E.G. Richardson, "Coding Video: A Practical Guide to HEVC and Beyond" Wiley, May EE5359 Multimedia Processing57

27. J. Vanne et al, “Comparative Rate-Distortion-Complexity Analysis of HEVC and AVC Video Codecs”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 22, No. 12, pp , Dec HM Encoder Description: coding/n14703-high-efficiency-video-coding-hevc-encoderhttp://mpeg.chiariglione.org/standards/mpeg-h/high-efficiency-video- coding/n14703-high-efficiency-video-coding-hevc-encoder 29. White paper on PSNR-NI: HEVC Tutorial by V.Sze and M.Budagavi, ”Design and Implementation of Next Generation video coding systems”, IEEE International Symposium on Circuits and Systems (ISCAS), Melbourne, Australia, June Available on : A.Hore and D.Ziou, "Image Quality Metrics : PSNR vs SSIM", IEEE International Conference on Pattern Recognition (ICPR), pp , Aug G. Correa et al, “Fast HEVC encoding decisions using data mining”, IEEE Transactions on CSVT, vol.25, pp , Apr EE5359 Multimedia Processing58

33. Intel VTune Amplifier XE Software profiler website : H.Schwarz, D.Marpe and T.Weigand," Analysis of Hierarchical B pictures and MCTF", IEEE International Conference on Multimedia and Expo (ICME), pp. 9-12, July EE5359 Multimedia Processing59

Thank You. EE5359 Multimedia Processing60