1. Vineeth Shetty Kolkeri University of Texas, Arlington
ERROR CONCEALMENT TECHNIQUES IN H.264/AVC, FOR VIDEO TRANSMISSION OVER WIRELESS NETWORKS
Vineeth Shetty Kolkeri
University of Texas, Arlington

2. Outline Introduction What is Error Concealment?
Error Concealment Architecture
H.264/MPEG-4 AVC – Overview
Error Concealment algorithm
Error Concealment performance analysis test results
Conclusions
Future Work
References

3. Figure 1: Typical Situation of 3G/4G cellular telephony
Introduction
Figure 1: Typical Situation of 3G/4G cellular telephony

4. What is Error Concealment?
The operation adopted to reconstruct lost information for video transmission over wireless networks.
Need: Compatible with all video streaming devices.
Applications: Recovery of lost information which is comparable with encoded video sequence with minimal complexity.

5. Error Concealment Architecture

6. Overview of H.264 / AVC Latest Video coding standard
Basic design architecture similar to MPEG-x or H.26x
Better compression efficiency
Upto 50% bitrate reduction from the preceding video codec standard
Subjective quality is better
Advanced functional element
Wide variety of applications such as video broadcasting, video streaming, video conferencing, D-Cinema, HDTV.
Layered structure - consists of two layers: Network Abstraction Layer (NAL) and Video Coding Layer (VCL); supports 4:2:0 chroma sampling picture format including QCIF and CIF formats

7. Overview of H.264 / AVC (contd.)
Uses hybrid block based video compression techniques such as:
Transformation - reduction of spatial correlation
Quantization - bit-rate control
Motion compensated prediction - reduction of temporal correlation
Entropy coding - reduction in statistical correlation
Includes the following features:
Intra-picture prediction
4x4 integer transform
Multiple reference pictures
Variable block sizes
Quarter pel precision for motion compensation
In-loop de-blocking filter
Improved entropy coding

8. Television Broadcasting
H.264/AVC Profiles
Profiles and Levels for particular applications
Profile : a subset of entire bit stream of syntax,
different decoder design based on the Profile
Four profiles : Baseline, Main, Extended and High
Profile
Applications
Baseline
Video Conferencing
Videophone
Main
Digital Storage Media
Television Broadcasting
Extended
Streaming Video
High
Studio editing

9. Specific coding parts for the Profiles

10. Specific coding parts for the Profiles (contd.)
Common coding parts for the Profiles
I slice (Intra-coded slice) : the coded slice by using prediction only from decoded samples within the same slice
P slice (Predictive-coded slice) : the coded slice by using inter prediction from previously-decoded reference pictures, using more than one motion vector and reference index to predict the sample values of each block
CAVLC (Context-based Adaptive Variable Length Coding) for entropy coding

11. Specific coding parts for the Profiles (contd.)
Coding parts for Baseline Profile
Common parts : I slice, P slice, CAVLC
FMO Flexible macroblock order : macroblocks may not necessarily be in the raster scan order. The map assigns macroblocks to a slice group
ASO Arbitrary slice order : the macroblock address of the first macroblock of a slice of a picture may be smaller than the macroblock address of the first macroblock of some other preceding slice of the same coded picture
RS Redundant slice : This slice belongs to the redundant coded data obtained by same or different coding rate, in comparison with previous coded data of same slice

12. H.264 Encoder (contd.)

13. H.264 - Transform 4x4 multiplier-free integer transform
Transform coefficients perfectly invertible
Hierarchical structure - 4 x 4 Integer DCT, Hadamard transform
Hadamard transform applied when (16x16) intra prediction mode is used with (4x4) integer DCT
MB size for chroma depends on 4:2:0, 4:2:2 and 4:4:4 formats

14. H.264 – Transform (contd.) YUV different formats
4:4:4 is full bandwidth YUV video, and each macroblock consists of 4 Y blocks, and 4 U/V blocks. Being full bandwidth, this format contains as much as data would if it were in the RGB color space.
4:2:2 contains half as much chrominance information as 4:4:4 and 4:2:0 contains one quarter of the chrominance information.

15. H.264 – Transform (contd.)
Codec - A video codec is software that can compress a video source (encoding) as well as play compressed video (decompress).
CIF - Common Intermediate Format - a set of standard video formats used in videoconferencing, defined by their resolution. The original CIF is also known as Full CIF (FCIF).
QCIF - Quarter CIF (resolution 176x144)
SQCIF - Sub quarter CIF (resolution 128x96)
4CIF - 4 x CIF (resolution 704x576)
16CIF - 16 x CIF (resolution 1408x1152

16. H.264 - Scaling and Quantization
Multiplication operation for exact transform combined with multiplication of scalar quantization
Scale factor for each element in each sub-block varies as a function of quantization parameter associated with macro-block that contains sub block; position of element within sub-block
Rate-control algorithm in encoder controls value of quantization parameter
Encoder performs post-scaling and quantization

17. Deblocking filter Adaptive
To reduce the blocking artifacts in the block boundary and prevent the propagation of accumulated coded noise.
Filtering is applied to horizontal or vertical edges of 4 x 4 blocks in a macroblock, adaptively on the several levels (slice, block-edge, sample).

18. H.264 – Motion Compensation
Predicted rectangular arrays of pixels: 4x4, 4x8, 8x4, 8x8, 16x8, 8x16, and 16x16.
Translation from other array positions in reference picture specified with quarter pixel precision.
In case of 4:2:0 format, the chroma MVs have a resolution of 1/8 of a pixel, derived from transmitted luma MVs of 1/4 pixel resolution.

19. Prediction of variable block size
A MB can be partitioned into smaller block sizes
4 cases for 16 x 16 MB, 4 cases for 8 x 8 Sub-MB
Large partition size : homogeneous areas, small : detailed areas

20. Prediction of variable block size (contd.)
Frame divided into multiple macroblocks of 16 x 16, 8 x 8, 4 x 4 variable size to represent coding profiles
No. of bits in I and P frames
I P I
Graph shows the size of the different I and P frames obtained after encoding 19 frames of the Foreman QCIF video sequence. Green line shows the average values of the bit lost when it is passed through the lossy algorithm after encoding in a video sequence

21. H.264 – Entropy Coding
CAVLC (Context-based Adaptive Variable Length Coding).
CABAC (Context-based Adaptive Binary Arithmetic Coding).
CAVLC makes use of run-length encoding.
CABAC utilizes arithmetic coding; codes both MV and residual transform coefficients.
Typically CABAC provides % reduction in bit rate compared to CAVLC, for the same PSNR.
All syntax elements are encoded by Exp-Golomb codes (Universal Variable Length Codes (UVLC)).

22. Error Concealment Algorithm
1.Spatial Concealment – weighted averaging:
Estimate missing pixels by smoothly extrapolating surrounding pixels
Correctly recovering missing pixels is extremely difficult, however correctly estimating the DC (average) value is very helpful
2.Temporal Concealment – copy algorithm:
Copy the pixels at the same spatial location in the previous frame
Effective when there is no motion, potential problems when there is motion
3.Motion compensated temporal Concealment–motion vector interpolation:
Estimate missing block as motion-compensated block from previous frame
Can use coded motion vector, neighboring motion vector, or compute new motion vector

23. Spatial Concealment – weighted averaging
Block based weighted averaging

24. Spatial Concealment–weighted averaging (contd.)
Macroblock based weighted averaging

25. Spatial Concealment – weighted averaging (contd.)
Recovery of the damaged macroblock in Foreman and Akiyo video sequence (a) distorted image lying within a smooth area; b) macroblock based weighted averaging applied on a white smooth area; c) block based weighted averaging applied on a white smooth area.

26. Spatial Concealment – weighted averaging (contd.)
Recovery of the damaged macroblock in Foreman and Akiyo video sequence (a) distorted image lying within a smooth area; b) macroblock based weighted averaging applied on a white smooth area; c) block based weighted averaging applied on a white smooth area.

27. Temporal Concealment – Frame Copy
Frames# 5, 6 and 7 are the output of H.264 encoded frames after it is transmitted in the error prone wireless medium
Frame# 5 is the decoded frame. Here Frame# 6 successfully copied lost information from Frame 5 by copy algorithm; Frame #7 is degraded (Because Frame#7 is reconstructed bycollecting the information from previous reference frames)

28. Temporal Concealment – MV Interpolation
Motion vector recovery by a) Using the motion vectors from the surrounding macroblocks after frame decoding b) Using the motion vectors from the surrounding macroblocks during macroblock decoding

29. Temporal Concealment – MV Interpolation (contd.)
Four Prior-decoded pictures Current Picture
as references

30. Frames# 5, 6 and 7 of the Original Sequence
Motion Vector Interpolation
Frames# 5, 6 and 7 of the Original Sequence
Frame# 5 of the decoded frame, Successfully decoded lost Frame # 6. Frame# 6 was reconstructed by Motion Copy algorithm. Frame #7 is degraded.

31. Motion Vector Interpolation (contd.)
Recovery of the damaged macroblock in Foreman video sequence (a) original sequence b) Distorted Sequence c) Concealed Output using Motion Estimation.

32. Error Concealment performance analysis test results (Foreman Sequence)

33. Error Concealment performance analysis test results (contd
Error Concealment performance analysis test results (contd.) (Foreman Sequence)

34. Error Concealment performance analysis test results (contd
Error Concealment performance analysis test results (contd.) (Foreman Sequence)

35. Error Concealment performance analysis test results (contd
Error Concealment performance analysis test results (contd.) (Foreman Sequence)

36. Error Concealment performance analysis test results (contd.)
Simulation results of different error concealment algorithms for Foreman QCIF176x144 video sequence.

37. Error Concealment performance analysis test results (contd.)
Simulation results of different error concealment algorithms for Foreman QCIF176x144 video sequence.

38. Conclusions
Implementation of spatial concealment performed better in a video with constant background.
Implementation of temporal concealment performed better in a video with linear motion between consecutive frames.
Implementation of temporal concealment performed better in a video with dynamic motion between consecutive frames.
At higher bit rates: spatial and temporal concealments achieved better results.
Complexity of implementation is negligible and does not degrade in the processor performance.

39. Future Research
Implementing Error Concealment algorithm in H.264/SVC video codec as it doesn’t support error concealment in current implementation.
Use forward and backward MVs.
Implementing Error Concealment algorithm in H.264 extended and High profiles.
Find a block which has uniform motion within small duration.
Implementing Error Concealment model for real time application like video surveillance.

40. References
T. Stockhammer, M. M. Hannuksela and T. Wiegand, “H.264/AVC in Wireless Environments”, IEEE Trans. Circuits and Systems for Video Technology, Vol. 13, pp , July 2003.
2. Soon-kak Kwon, A. Tamhankar and K.R. Rao, ”Overview of H.264 / MPEG-4 Part 10”, J. Visual Communication and Image Representation, vol. 17, pp , April 2006.
3. S. Wenger, “H.264/AVC over IP” IEEE Trans. Circuits and Systems for Video Technology, vol. 13, pp , July 2003.
4. M. Wada, “Selective Recovery of Video Packet Loss using Error Concealment,” IEEE Journal on Selected Areas in Communication, vol. 7, pp , June 1989.
5. I.C.Todoli “Performance of Error Concealment Methods for Wireless Video”, Diploma Thesis, Vienna University of Technology,
6. Video Trace research group at ASU, “YUV video sequences,”
7. A.B. Watson, "Toward a perceptual video quality metric", SPIE Human Vision, Visual Processing, and Digital Display VIII, vol. 3299, pp , 1998.
8. F. Xiao, “DCT-based video quality evaluation,” Final Project for EE392J Stanford Univ
Z. Wang, “The SSIM index for image quality assessment,”

41. References (contd.)
10. Z. Wang, et al, “Image Quality Assessment: From Error Visibility to Structural Similarity”, IEEE Trans. Image Processing, vol. 13, pp , April 2004.
11. ISO/IEC JTC1, “Joint Draft 8 of SVC Amendment,” ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6, Doc. JVT-U201, Oct
12. ISO/IEC JTC1, “Joint Scalable Video Model 8.0,” ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6, Doc. JVT-U202, Oct
13. Yi-Hau Chen, et al, ’’ Bandwidth-efficient encoder framework for H.264/AVC scalable extension” Ninth IEEE International Symposium on Multimedia, pp , Dec 2007.
14. DSP Design Line article on SVC
eInfochips HD Codecs - H.264 SVC - for Digital Media Processors From Texas Instruments
H.264/AVC Reference Software Download:
17. H.264/SVC Reference Software download:
18. AVI to YUV converter:

42. Thanks for your attention!
Q/A