1. Presentation on:
Performance and Computational Complexity Assessment of High-Efficiency Video Encoders
Presented by:
MANU RAJENDRA SHEELVANT
Under the guidance of:
Dr. K. R. RAO

2. ACRONYNMS ALF : Adaptive Loop Filter. JM: H.264 Test Model.
AMP : Asymmetric Motion estimator.
JPEG: Joint Photographic Experts Group.
AVC : Advanced Video Coding.
MV : Motion Vector.
CABAC: Context Adaptive Binary Arithmetic Coding.
MC: Motion Compensation.
CTB: Coding Tree Block.
ME: Motion Estimation.
CTU: Coding Tree Unit.
MPEG: Motion Picture Experts Group.
CU: Coding Unit.
PC : Prediction Chunking.
CB : Coding Block
PCS: Professional Communication Society.
DCT : Discrete Cosine Transform.
PU : Prediction Unit.
DBF: De-blocking Filter.
PB: Prediction Block.
FDM : Fast Merge Decision.
QP: Quantization Parameter.
HEVC: High Efficiency Video Coding.
SAO: Sample Adaptive Offset.
HM: HEVC Test Model.
TB: Transform Block.
HP : Hierarchical Prediction.
TU: Transform Unit.
JCT: Joint Collaborative Team.
VCEG: Video Coding Experts Group.
JCT-VC: Joint Collaborative Team on Video Coding.

3. MOTIVATION
Study about the computational complexity and encoding performance of the HEVC encoder.
Different cases of encoding configurations will be tested on wide variety of video contents and test sequences.
The relation/trade-off between HEVC complexity and coding efficiency cost will be studied.
Different types of coding tools will be combined and/or left out and then check the how the HEVC complexity and coding efficiency cost will be effected.
P Correa et al [2] have investigated the coding efficiency and computational complexity of HEVC encoders. Implement this analysis by considering the set of 16 different encoding configurations.

4. Figure 1: Block Diagram of HEVC CODEC[10]
HEVC VIDEO CODEC
Figure 1: Block Diagram of HEVC CODEC[10]

5. Figure 2: Block Diagram of the HEVC Encoder[7]

6. Figure 2: Block Diagram of the HEVC Decoder[11]

7. HEVC ENCODER
Six mandatory configurations for normalized experiments have been proposed by JCT-VC [37].
These are combinations of two possible complexity or efficiency settings, defined as high efficiency 10 (HE10) [36] and Main, and three basic coding or access settings, named as
Intra Only
Random Access
Low Delay.

8. POSSIBLE MANDATORY CONFIGURATIONS
Complexity/Efficiency Settings
Coding/Access Settings 1.
High efficiency 10 (HE10)
Intra Only 2.
Random Access 3.
Low Delay. 4.
Main 5. 6.
Table 1: Possible Encoding Configurations and Combinations

9. COMPLEXITY/EFFICIENCY SETTINGS
High Efficiency 10:- The HE10 setting is used for high coding efficiency (and high complexity).
Main:- Main defines a coding configuration with low computational complexity, which does not provide the best coding efficiency.

10. CODING/ACCESS SETTINGS
The Intra Only mode uses no temporal prediction.
Random Access allows the use of future frames as reference in temporal prediction.
Low Delay uses only previous frames as reference temporal prediction.

11. DIFFERENT CONFIGURATION TOOLS
Hadamard ME :- Enable(1) or Disable(0) Hadamard Transform for fractional Motion Estimator(ME). DCT is used for fractional Motion Estimation. Hadamard ME reduces the complexity also giving away a little Coding Efficiency.
Deblocking Filter Disable :- Disable(0) or Enable(1) Deblocking Filter during encoding process.
AMP (Asymmetric Motion Partitioning) :- Enable(1) or Disable(0) AMP.
AMP improves the coding efficiency, since irregular image patterns, which otherwise would be constrained to being represented by a smaller symmetric partition, can now be more efficiently represented without requiring further splitting.

12. DIFFERENT CONFIGURATION TOOLS
Search Range :- Specifies the search range used for motion estimation. (Default = 96) (To change we use --sr) (If sr = 0 full frame search).
Bi- Prediction Refinement (BiPredSearchRange) :- Specifies the search range used for bi-prediction refinement in motion estimation. (Default = 4).

13. DIFFERENT CONFIGURATION TOOLS
Fast Encoding (FEN) :- Enable(1) or Disable(0) fast encoding. When enabled, the following occurs:
In the SAD computation for blocks having size larger than 8, only the lines of even rows in the block are considered.
The number of iterations used in the bi-directional motion vector refinement in the motion estimation process is reduced from 4 to 1.
Fast Merge Decision (FDM) :- Enable(1) or disable(0) the use of fast encoder decisions for 2Nx2N merge mode.

14. DIFFERENT CONFIGURATION TOOLS
Sample Adaptive Offset (SAO) :- Enable(1) or Disable(0) Sample Adaptive Offset Filter. It reduces sample distortion by first classifying reconstructed samples into different categories, obtaining an offset for each category, and then adding the offset to each sample of the category.
Adaptive Loop Filter (ALF) :- Enable(1) or Disable(0) ALF. It is used minimize the mean square error between original pixels and decoded pixels using Wiener-based adaptive filter coefficients.

15. EXECUTION
Table 1: Encoder Configurations Used for Complexity and Performance Analysis [2]

16. EXECUTION(contd..) Table 2: Diffrenet Tests and their modifications.
Test Cases
Configurations tools modified
Test 1
Default test conditions used in encoder_randomaccess_main.
Test 2
Hadamard Motion Estimation enabled.
Test 3
Deblocking filter disabled.
Test 4
Assymetric Motion Partition enabled.
Test 5
Search Range modified 64 -> 96.
Test 6
Search Range modified 96 -> 128.
Test 7
Search Range for Bi-Prediction Refinement modified 4 ->8.
Test 8
Fast Encoding disabled.
Test 9
Fast Merge Decision disabled.
Test 10
Sample Adaptive Offset enabled.
Test 11
Adaptive Loop Filter enabled.
Test 12
Transform Skip enabled.
Test 13
Transform Skip Fast enabled.
Test 14
Wavefront Sychronisation enabled.
Test 15
Deblocking Filter enabled, Fast Encoding enabled, Fast Merge Decision enabled, Search Range for ME modified 128->64 and Search range for Bi-Prediction Refinement modified 8->4. Basically enabling all the tools available.
Table 2: Diffrenet Tests and their modifications.

17. NEW CONFIGURATION TOOLS HM7.0 -> HM.16.9
Transform Skip :- Enable(1) or Disable(0) transform-skipping mode decision for 4x4 TUs.
Transform Skip Fast :- Enable(1) or Disable(0) reduced testing of the transform-skipping mode decision for chroma TUs.
When enabled, no RDO search is performed for chroma TUs, instead they are transform-skipped if the four corresponding luma TUs are also skipped.
This option has no effect if Transform Skip is disabled.

18. YUV-PSNR = [6(PSNR)Y + (PSNR)U + (PSNR)V]/8
COMPLEXITY ANALYSIS
Encode different test sequences and compare their YUV-PSNR, Bit rates, and Time taken to encode the video sequence.
Normalized plot (w.r.t Configuration case 16) of time taken in different cases.
Normalized plot (w.r.t Configuration case 16) of YUV-PSNR in different cases.
YUV-PSNR = [6(PSNR)Y + (PSNR)U + (PSNR)V]/8
Normalized plot (w.r.t Configuration case 16) of BD- Bit Rates in different cases.
Plot BD-Bit Rate vs Time and BD-PSNR vs Time for different configurations. (New)

19. COMPLEXITY ANALYSIS(contd..)
Normalized Parameters:
Since we are dealing with different test sequences with different motion and different resolution, I am normalizing PSNR, Bit- Rate and Time Taken with respect to Default Test configurations. This will help put the all the graphs on the same plot.
Generalized Parameters:
Since we need to know PSNRs and Bit Rates for 4 Quantization Parameters(QP) to compute and plot BD-PSNR and BD-Bit Rate, I working on so many configurations, I could only have the test results for 1 QP, I thought comparing PSNRs and Bit Rates for different Test Sequences for same Test configuration and generate BD-PSNR and BD- Bit Rate.

20. RUNNING DIFFERENT CONFIGURATIONS

21. TEST SEQUENCES:
Test Sequence
Resolution
Frame Rate (in fps)
Total No. of frames
Pace (Motion)
BQ Mall
832x480 60
601
Medium
Kirsten and Sara
1280x720
600
Slow
Park Scene
1920x1080 24
240
People On Street
2560x1600 30
150
Race Horses
416x240
300
Most of the mentioned test sequences have been downloaded from [3].

22. EXPERIMENTAL SETUP Processor Intel(R) Core(TM) i3-2330M CPU @ 2.20GHz
Number of cores
Memory GB
Operating System Windows(TM) Home OS
HM Software version used HM 16.9
HM mode Release Mode
System Type bit OS

23. PEOPLE ON STREET

24. PEOPLE ON STREET(contd..)

25. PEOPLE ON STREET(contd..)

26. PEOPLE ON STREET(contd..)

27. PEOPLE ON STREET(contd..)

28. RACE HORSES

29. RACE HORSES(contd..)

30. RACE HORSES(contd..)

31. RACE HORSES(contd..)

32. RACE HORSES(contd..)

33. BQ MALL

34. BQ MALL(contd..)

35. BQ MALL(contd..)

36. BQ MALL(contd..)

37. BQ MALL(contd..)

38. KRISTEN AND SARA

39. KRISTEN AND SARA(contd..)

40. KRISTEN AND SARA(contd..)

41. KRISTEN AND SARA(contd..)

42. KRISTEN AND SARA(contd..)

43. PARK SCENE

44. PARK SCENE(contd..)

45. PARK SCENE(contd..)

46. PARK SCENE(contd..)

47. PARK SCENE(contd..)

48. NORMALIZED YUV-PSNR

49. NORMALIZED BIT RATE

50. NORMALIZED COMPLEXITY

51. GENERALIZED BD-PSNR

52. GENERALIZED BD-PSNR

53. COMPUTATIONAL COMPLEXITY ANALYSIS

54. CONCLUSIONS The configuration settings I recommend to use:
Assymmetric Motion Partition to be enabled.
Adaptive Loop Filter to be enabled.
Fast Merge Decision to be disabled.
Transform Skip to be enabled.

55. CONCLUSIONS(contd..)
The configuration settings I DO NOT recommend to use:
Transform Skip Fast to be enabled.
Wavefront Synchronization to be enabled.
Set bi-prediction refinement to 8.

56. REFERENCES
[1] G.J. Sullivan et al, “Overview of the High Efficiency Video Coding (HEVC) Standard”, IEEE Trans. on Circuits and Systems for Video Technology (CSVT), Vol. 22, No. 12, pp , Dec
[2] G. Correa et al, “Performance and computational complexity assessment of high efficiency video encoders”, IEEE Trans. CSVT, vol. 22, pp , Dec
[3] Video test sequences -
[4] HM 16.9 software -
[5] I.E.G. Richardson, “Video Codec Design: Developing Image and Video Compression Systems”, Wiley, 2002.
[6] B. Bross et al, “High Efficiency Video Coding (HEVC) Text Specification Draft 10”, Document JCTVC-L1003, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC), Mar.2013.
[7] D. Marpe et al, “The H.264/MPEG4 advanced video coding standard and its applications”, IEEE Communications Magazine, Vol. 44, pp , Aug

57. REFERENCES (Contd..)
[8] HEVC white paper-Ateme:
[9] G.J. Sullivan et al, “Standardized Extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of selected topics in Signal Processing, Vol. 7, No. 6, pp , Dec
[10] HEVC tutorial by I.E.G. Richardson:
[11] C. Fogg, “Suggested figures for the HEVC specification”, ITU-T / ISO-IEC Document: JCTVC J0292r1, July 2012.
[12] T. Wiegand et al, “Overview of the H.264/AVC Video Coding Standard”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 13, No. 7, pp , July 2003.
[13] U.S.M. Dayananda, “Study and Performance comparison of HEVC and H.264 video codecs” M.S. Thesis, EE Dept., UTA, Arlington, TX, Dec available on
[14] M. Wein, “ High Efficiency Video Coding: Coding Tools and Specification”, Springer, 2014.

58. REFERENCES (Contd..)
[15] T. Vermeir, “Use cases and requirements for lossless and screen content coding”, JCTVC-M0172, 13th JCT-VC meeting, Incheon, KR, Apr
[16] J. Sole et al, “Requirements for wireless display applications”, JCTVC-M0315, 13th JCT-VC meeting, Incheon, Korea, Apr
[17] A. Gabriellini et al, “Combined Intra-Prediction for High-Efficiency Video Coding”, IEEE Journal of selected topics in Signal Processing. Vol. 5, no. 7, pp , Nov
[18] L. Younghoon, K. Jungsoo, and K. Chong-Min, “Energy-aware video encoding for image quality improvement in battery-operated surveillance camera,” IEEE Trans. Very Large Scale Integration (VLSI) Syst., vol. 20, no. 2, pp. 310–318, Feb
[19] W. Kim, J. You, and J. Jeong, “Complexity control strategy for real-time H.264/AVC encoder,” IEEE Trans. Consumer Electronics, vol. 56, no. 2, pp. 1137–1143, May 2010.

59. REFERENCES (Contd..)
[20] D. N. Kwon, et al, “Performance and computational complexity optimization in configurable hybrid video coding system,” IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 1, pp. 31–42, Jan
[21] O. Lehtoranta and T. D. Hamalainen, “Complexity analysis of spatially scalable MPEG-4 encoder,” in Proc. Int. Symp. System-on-Chip, pp. 57–60 , 2003.
[22] J. Ostermann, et al, “Video coding with H.264/AVC: Tools, performance, and complexity,” IEEE Circuits Syst. Mag., vol. 4, no. 1, pp. 7–28, Jan.–Mar
[23] S. Saponara, et al, “Performance and complexity co-evaluation of the advanced video coding standard for cost-effective multimedia communications,” EURASIP J. Appl. Signal Process., pp. 220–235, Jan
[24] P. Li, Y. Chen, and W. Ji, “Rate-distortion-complexity analysis on AVS encoder,” presented at the 11th Pacific Rim Conf. Multimedia, Shanghai, China, 2011.

60. REFERENCES (Contd..)
[25] A. Hallapuro, V. Lappalainen, and T. D. Hamalainen, “Performance analysis of low bit rate H.26L video encoder,” presented at the Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol
[26] M. Horowitz, et al, “H.264/AVC baseline profile decoder complexity analysis,” IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7, pp. 704–716, July 2003.
[27] L. Szu-Wei and C. C. J. Kuo, “Complexity modeling of spatial and temporal compensations in H.264/AVC decoding,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 5, pp. 706–720, May 2010.
[28] M. Zhan, H. Hao, and W. Yao, “On complexity modeling of H.264/AVC video decoding and its application for energy efficient decoding,” IEEE Trans. Multimedia, vol. 13, no. 6, pp. 1240–1255, Dec
[29] V. Lappalainen, A. Hallapuro, and T. D. Hamalainen, “Complexity of optimized H.26L video decoder implementation,” IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7, pp. 717–725, July 2003.

61. REFERENCES (Contd..)
[30] On Software Complexity, JCTVC-G757, ISO/IEC-JCT1/SC29/WG11, Geneva, Switzerland, 2011.
[31] HM Decoder Complexity Assessment on ARM, JCTVC-G262, ISO/IECJCT1/SC29/WG11, Geneva, Switzerland, 2011.
[32] Z. Chang-Guo et al, “MPEG video decoding with the UltraSPARC visual instruction set,” in Proc. Compcon ’95: Technol. Inform. Superhighway Dig. Papers, pp. 470–477, 1995.
[33] L. Xiang, M. Wien, and J. R. Ohm, “Rate-complexity-distortion optimization for hybrid video coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 7, pp. 957–970, Jul
[34] OProfile: A System Profiler for Linux [Online]. Available:
[35] Accessing JCT-VC Documents [Online]. Available:

62. REFERENCES(Contd..)
[36] Common Test Conditions and Software Reference Configurations, JCTVC-I1100, ISO/IEC-JCT1/SC29/WG11, Geneva, Switzerland, 2012.
[37] G. Correa, et al, “Complexity control of high efficiency video encoders for power-constrained devices,” IEEE Trans. Consumer Electron., vol. 57, no. 4, pp. 1866– 1874, Nov
[38] G. Laroche, J. Jung, and B. Pesquet-Popescu, “RD optimized coding for motion vector predictor selection,” IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 9, pp. 1247–1257, Sep
[39] S. Oudin et al, “Block merging for quadtree-based video coding,” in Proc. IEEE Int. Conf.
Multimedia Expo, Jul. 2011, pp. 1–6 , July 2011.
[40] V. Sze and M. Budagavi, “Parallelization of CABAC transform coefficient coding for HEVC,” in Proc. PCS, pp. 509–512 , 2012.
[41] HM Software Manual:

63. REFERENCES(Contd..)
[42] N. Ling, “High efficiency video coding and its 3D extension: A research perspective,” Keynote Speech, ICIEA, Singapore, July [43] X. Wang et al, “Paralleling variable block size motion estimation of HEVC on CPU plus GPU platform”, IEEE ICME workshop, [44] H.R. Tohidpour, M.T. Pourazad and P. Nasiopoulos, “Content adaptive complexity reduction scheme for quality/fidelity scalable HEVC”, IEEE ICASSP 2013, pp , June [45] M. Wien, “HEVC – coding tools and specifications”, Tutorial, IEEE ICME, San Jose, CA, July [46] D. Grois, B. Bross and D. Marpe, “HEVC/H.265 Video Coding Standard (Version 2) including the Range Extensions, Scalable Extensions, and Multiview Extensions,” (Tutorial) Monday 29 June am-3:00pm , IEEE ICME 2015, Torino, Italy, 29 June – 3 July, [47] D. Grois, B. Bross and D. Marpe, “HEVC/H.265 Video Coding Standard including the Range Extensions, Scalable Extensions, and Multiview Extensions,” (Tutorial), IEEE ICCE , Berlin, Germany, 6 – 9 Sept [48] D. Grois, B. Bross and D. Marpe, “HEVC/H.265 Video Coding Standard (Version 2) including the Range Extensions, Scalable Extensions, and Multiview Extensions,” (Tutorial) Sunday 27 Sept 2015, 9:00 am to 12:30 pm), IEEE ICIP, Quebec City, Canada, 27 – 30 Sept The tutorial below is for personal use only. Password: a2FazmgNK

64. Any Suggestions/Questions are welcome.
Thank You.