1. Santosh Kumar Muniyappa
IMPLEMENTATION OF COMPLEXITY REDUCTION ALGORITHM FOR INTRA MODE SELECTION IN H.264/AVC
Santosh Kumar Muniyappa
Guided by
- Dr. Rao

2. Contents Need for video compression Motivation
Video coding standards, video formats and quality
Overview of H.264
Complexity reduction algorithm for intra mode selection
Experimental results
Conclusions and future work
References

3. Need for Video Compression
Growing market for high bit-rate connections, large storage capacity
Insufficient resource to handle uncompressed video
Better proposition is to send high- resolution compressed video than a low- resolution, uncompressed stream over a high bit-rate transmission channel

4. Motivation [2] Removing redundancy in a video clip
Only a small percentage of any particular frame is new information
Highly complex process
Reduce the overall complexity suitable for handheld devices

5. Timeline of Video Development [10]
Inter-operability between encoders and decoders from different manufacturers
Build a video platform which helps to interact with video codecs, audio codecs, transport protocols, security and rights management in well defined and consistent ways

6. Video Format [19] Format Video Applications Resolution Sub-QCIF
128 × 96
Mobile applications
Quarter CIF (QCIF)
176 × 144
SIF(525)
352 x 240
Video conferencing
CIF/SIF(625)
352 × 288
4CIF/4SIF(625)
704 × 576
SDTV, DVD video
CIF - common intermediate format
SIF - simple intermediate format
SDTV – standard definition television

7. Quality Specify, evaluate and compare
Visual quality is inherently subjective
PSNR - most widely used objective quality measurement
PSNRdB = 10 log10 ((2n − 1)2 / MSE)
where, n = number of bits per pixel
SSIM - represents perceptual image quality based on the structural information

8. OVERVIEW OF H.264 / AVC STANDARD
First published by JVT in 2003
Achieve substantially higher video compression and a network friendly video representation
Built on the concepts of earlier standards such as MPEG-2 and MPEG-4 Visual
50% reduction in bit-rate
Error resilience
Supports various interactive (video telephony) and non-interactive applications (broadcast, streaming, storage, video on demand)

9. H.264 Video Codec [3]

10. Slice Types [3] I (intra) slice – contains reference only to itself.
P (predictive) slice – uses one or more recently decoded slices as a reference (or prediction) for picture construction.
B (bi-predictive) slice – works similar to P slices except that former and future I or P slices may be used as reference pictures
SI and SP or “switching” slices may be used for transitions between two different H.264 video streams.

11. H.264 Encoder [27]

12. H.264 Decoder [27]

13. Intra Prediction
I – pictures usually have a large amount of information present in the frame.
The spatial correlation between adjacent macro-blocks in a given frame is exploited.
H.264 offers nine modes for intra prediction of 4x4 luminance blocks.
H.264 offers four modes of intra prediction for 16x16 luminance block.
H.264 supports four modes similar to 16x16 luminance block for prediction of 8x8 chrominance blocks.

14. Intra Prediction Modes for 4x4 Luminance Blocks [27]

15. Intra Prediction Modes for 16x16 Luminance Blocks and 8x8 Chrominance Blocks [27]

16. Inter Prediction [1]
Takes advantage of the temporal redundancies that exist among successive frames.
Temporal prediction in P frames involves predicting from one or more past frames known as reference frames.

17. Inter Prediction Modes [11]

18. Integer Transform and Quantization
Prediction error block is expressed in the form of transform co- efficients.
H.264 employs a purely integer spatial transform, which is a rough approximation of the DCT.
Quantization:
Significant portion of data compression takes place.
Fifty-two different quantization step sizes can be chosen.
Step sizes are increased at a compounding rate of approximately 12.5%.

19. De-blocking Filter and Entropy Coding
Removes the blocking artifacts due to the block based encoding pattern
In-loop de-blocking filter
Entropy coding:
Assigning shorter code-words to symbols with higher probabilities of occurrence, and longer code-words to symbols with less frequent occurrences.
CAVLC and CABAC

20. H.264 Profiles [27]

21. Complexity Reduction Algorithm For Intra Mode Selection
Existing intra prediction algorithm in H.264 using RDO examines all possible combinations of coding modes.
Mode combinations for each MB sums up to 592.
This research proposes a complexity reduction algorithm using simple directional masks and neighboring modes.
Reduces the number of mode combinations into 132 at the most, with negligible loss of PSNR and SSIM with a slight bit-rate increase.

22. Rate Distortion Optimization
Lagrangian multiplier λMODE = 0.85x 2(QP-12)/3
Cost = D + λ MODE x R
N8 – number of modes of an 8x8 chroma block
N4 – number of modes of a 4x4 luma block
N16 – number of modes of a 16x16 luma block
Computing the best mode for one macro block:
N8x (16xN4 + N16) = 4 x (16 x 9 + 4)
= 592

23. Proposed Intra Mode Selection Algorithm for a 4x4 Luma Block [16]
Diff = |a – m| + |b – n| + |c – o| + |d – p|, for vertical direction
Diff = |a – d| + |e – h| + |i – l| + |m – p|, for horizontal direction
Diff = |c – i| + 2·|d – m| + |h – n|, for diagonal down left direction
Diff = |b – l| + 2·|a – p| + |e – o|, for diagonal down right direction
Diff = |a – n| + 2·|b – o| + |c – p|, for vertical right direction
Diff = |a – h| + 2·|e – l| + |i – p|, for horizontal down direction
Diff = |b – m| + 2·|c – n| + |d – o|, for vertical left direction
Diff = |e – d| + 2·|i – h| + |m – l|, for horizontal up direction

24. Continued… [16] Step 1 - For a 4x4 luma block, obtain avg and S
Step 2a - If S is larger than a threshold, T1, carry out RDO procedure for at most four candidate modes: two modes with minimum and second minimum Diff, and at most two modes from adjacent blocks. In this case, DC mode of adjacent blocks is excluded from RDO procedure.
Step 2b - If S is smaller than a threshold, T1, carry out RDO procedure for at most four candidate modes: one mode with minimum Diff, at most two modes from adjacent blocks, and DC mode.
Note: T1 is set to 32

25. Proposed Intra Mode Selection Algorithm for a 16x16 Luma Block
Step 1 - Examine sizes of adjacent blocks: if both blocks (upper block and left block) are 16x16, go to Step 2, otherwise go to Step 4.
Step 2 - Examine modes of adjacent blocks: if both modes are same, go to Step 3, otherwise select the best mode for a 16x16 luma block, which results in the minimum SATD (sum of absolute transformed differences) between two adjacent modes of modeA and modeB.
Step 3 - If both adjacent modes are DC mode, go to Step 4, and otherwise select the best mode for a 16x16 luma block, which results in the minimum SATD between the adjacent mode and DC mode.
Step 4 - Let ΔV be a vertical difference between upper boundary pixels of the current block and boundary pixels of the upper block, and ΔH be a horizontal difference between left boundary pixels of the current block and boundary pixels of the left block.

26. Continued… [16]

27. Continued…
Step5 - Obtain candidate modes by using two difference values, ΔV and ΔH:
if |ΔV − ΔH | is smaller than 2xT2, candidate modes are DC mode and plane mode.
if (ΔV − ΔH) is larger than T2, candidate modes are DC mode and horizontal mode.
if (ΔV − ΔH) is smaller than T2, candidate modes are DC and vertical mode.
The threshold T2 is set equal to 8.
Finally, select the best mode among all candidate modes by choosing the mode with minimum SATD.

28. Experimental Results I-only type.
Various QP of 10, 20, 28, 34, and 40.
QCIF -100 frames
CIF - 30 frames
The results were compared with exhaustive search of JM in terms of the change of PSNR, bit-rate, SSIM, compression ratio, and encoding time.
Intel Pentium Dual Core processor of 2GHz and 2GB memory.

29. Test Sequences [39]
CIF and QCIF video sequences (a) bridge-close (b) bridge-far (c) coastguard (d) container (e) mobile

30. CR = Complexity reduction algorithm
JM = Joint model

32. CR = Complexity reduction algorithm
JM = Joint model

33. CR = Complexity reduction algorithm
JM = Joint model

34. CR = Complexity reduction algorithm
JM = Joint model

35. CR = Complexity reduction algorithm
JM = Joint model

36. Conclusions
Comparison of encoding time, PSNR, bit-rate, and SSIM between the JM reference software and complexity reduced encoder at QP, 10, 20, 28, 34 and 40.
Average of 24.93% reduction in encoding time in case of complexity reduced algorithm with negligible loss of PSNR and SSIM and a paltry increase in bit- rate at nearly the same compression ratio.
Hence, this approach of reducing the number of mode combinations in spatial domain using simple directional masks can find its application in low complexity devices like mobile or any handheld device.

37. Future Work
The complexity reduction algorithm was implemented for CIF and QCIF format video sequences. The idea can be extended further to other video formats like 4SIF and HD.
The complexity reduction algorithm was integrated in to JM reference software, it can be integrated to other open source H.264 softwares like X264 and performance analysis can be done.
Since the aim is to reduce the overall complexity suitable for handheld devices with limited computing resources, algorithms which reduce the mode combinations in inter-prediction can also be integrated with this complexity reduced intra-prediction algorithm.

38. References
I. E.G. Richardson, “H.264 and MPEG-4 video compression: video coding for next-generation multimedia”, Wiley, 2003.
Interframe coding pictures, Axis Communications,
I. E. Richardson, “The H.264 advanced video compression standard”, 2nd Edition, Wiley, 2010.
Draft ITU-T Recommendation and final draft international standard of joint video specification (ITU-T Rec. H.264/ISO/IEC AVC), Mar
ITU-T Recommendation H.120. Codecs for Videoconferencing using primary digital group transmission. March 1993.
ITU-T Recommendation H.261. Video codec for audiovisual services at px64 kbits. December March 1993 (revised).
ITU-T Recommendation H.262. Information technology – generic coding of moving pictures and associated audio information: video. July 1995
ISO/IEC Information technology - Coding of moving pictures and associated audio for digital storage media at up to about 1.5Mbps. November 1998.
ITU-T Recommendation H.263. Video coding for low bit rate communication. February 1998.
Overview of International Video Coding Standards (preceding H.264/AVC), Gary J. Sullivan, ITU, 2005.
H.264 / MPEG-4 Part 10 White Paper, I. E G Richardson.
M. Jafari and S. Kasaei, “Fast Intra- and Inter-Prediction Mode Decision in H.264 Advanced Video Coding”, International Journal of Computer Science and Network Security, VOL.8 No.5, pp. 1-6, May 2008.

39. F. Pan et al, “Fast intra mode decision algorithm for H
F. Pan et al, “Fast intra mode decision algorithm for H.264/AVC video coding”, in Proc.IEEE Int. Conf. Image Process., pp. 781– 784, Singapore, Oct
F. Fu et al, “Fast intra prediction algorithm in H.264/AVC”, in Proc. 7th Int. Conf. Signal Process., pp. 1191–1194, Beijing, China, Sep
Y. Zhang et al, “Fast 4x4 intra-prediction mode selection for H.264”, in Proc. Int. Conf. Multimedia Expo, pp. 1151–1154, Taipei, Taiwan, Jun
J. Kim et al, “Complexity reduction algorithm for intra mode selection in H.264/AVC video coding” J. Blanc-Talon et al. (Eds.): pp – 465, ACIVS 2006, LNCS 4179, Springer-Verlag Berlin Heidelberg, 2006.
JM reference software, Fraunhofer Institute for Telecommunications Heinrich Hertz Institute.
Recommendation ITU-R BT.601-7, Studio encoding parameters of digital television for standard 4:3 and wide‑screen 16:9 aspect ratios, BT Series, March 2011
Open source article on video formats, Wikipedia foundation.
Z. Wang et al, "Image quality assessment: From error visibility to structural similarity," IEEE Transactions on Image Processing, vol. 13, no. 4, pp , Apr
ITU-T Recommendation H.264. Advanced video coding for generic audiovisual services. November 2007.
ISO/IEC Information technology - Coding of moving pictures and associated audio for digital storage media at up to about 1.5Mbps – Part 4. March 1998.
ITU Telecom. Standardization Sector of ITU, “Video coding for low bitrate communication,” Draft ITU-T Recommendation H.263 Version 2, Sept
ITU-T "Video Coding for low bit rate communication," ITU-T Recommendation H.263; version 1, Nov 1995; version 2, Jan. 1998; version 3, Nov
ISO/IEC Information technology - Coding of audio-visual objects – Part 2. December 2001.
Monash University, Multimedia webpage,

40. S. Kwon, A. Tamhankar, and K. Rao, “Overview of H
S. Kwon, A. Tamhankar, and K. Rao, “Overview of H. 264/MPEG-4 part 10, " Journal of Visual Communication and Image Representation, vol. 17, no. 2, pp , April 2006.
T. Purushotham, “Low complexity H.264 encoder using machine learning," M.S. Thesis, E.E Dept, UTA, 2010.
H. Yadav, “Optimization of the deblocking filter in H.264 codec for real time implementation,” M.S. Thesis, E.E Dept, UTA, 2006
G. Sullivan, P. Topiwala, and A. Luthra, The H.264/AVC advanced video coding standard: overview and introduction to the fidelity range extensions, in: SPIE Conference on Applications of Digital Image Processing XXVII, vol. 5558, pp , 2004.
K.R. Rao and P. Yip, Discrete Cosine Transform, Academic Press, 1990.
Y. Huh, K. Panusopone and K.R. Rao, “Variable block size coding of images with hybrid quantization”, IEEE Trans. Circuits and Systems for Video Technology vol. 6, pp. 679–685, Dec
J. Ribas-Corbera and D.L. Neuhoff, “Optimizing Block Size in Motion Compensation”, Journal of Electronic Imaging, vol. 7, pp , Jan. 1998
M. Wien, “Variable block size transforms for H.264/AVC”, IEEE Trans. For Circuits and Systems for Video Technology, vol. 13, pp. 604–613, July 2003.
M. Ravassi, M. Mattavelli and C. Clerc, “JVT/H.26L decoder complexity analysis”, Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG, doc. JVT-D153, Klagenfurt, Austria, 22–26 July, 2002 (available via anonymous ftp from ftp://ftp.imtc-files.org/jvt-experts/).
M.-T. Sun, T.-D. Wu and J.-N. Hwang, “Dynamic bit allocation in video combining for multipoint conferencing,” IEEE Trans. Circuits and Syst. II, vol. 45, no. 5, pp , May 1998.
O. Werner, “Re-quantization for transcoding of MPEG-2 intra frames,” IEEE Trans. Image Processing, vol. 8, no. 2, pp , Feb

41. A. Luthra, G. Sullivan and T
A. Luthra, G. Sullivan and T. Wiegand, “Introduction to the special issue on the H.264/AVC video coding standard”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 13, issue 7, pp , July 2003.
Repository for freely-redistributable test sequences, media.xiph.org.
D. Marpe, T. Wiegand and G. Sullivan, "The H.264/MPEG4 advanced video coding standard and its applications," IEEE, Communications Magazine, vol.44, no.8, pp , Aug
ITU-T and ISO / IEC JTC 1, “Advanced Video Coding for Generic Audiovisual Services,” ITU-T Rec. H.264 & ISO / IEC , Version 1, May 2003; Version 2, Jan. 2004; Version 3 (with High family of profiles), Sept 2004; Version 4, July 2005 [Online]. Available:
G. J. Sullivan, “The H.264 / MPEG4-AVC video coding standard and its deployment status,” Proc. SPIE Conf. Visual Communications and Image Processing (VCIP), Beijing, China, July
T. Wiegand et al, “Rate-constrained coder control and comparion of video coding standards,” IEEE Trans. Circuits Systems Video Technol., vol. 13, no. 7, pp , July 2003.
A. Puri, X. Chen and A. Luthra, “Video coding using the H.264/MPEG-4 AVC compression standard”, Signal processing: image Communication, vol. 19, pp , Oct
T. Stockhammer, D. Kontopodis, and T. Wiegand, “Rate-distortion optimization for H.26L video coding in packet loss environment,” in Proc. Packet Video Workshop 2002, Pittsburgh, PY, April
K.R. Rao and J.J. Hwang, “Techniques and standards for digital image/video/audio coding,” Englewood Cliffs, NJ: Prentice Hall, 1996.

42. Thank You 