1. Interim Project Presentation Error concealment techniques in H.264 Under the guidance of Dr. K.R. Rao By Moiz Mustafa Zaveri (moiz.mustafazaveri@mavs.uta.edu) (1001115920)

2. Contents Problem statement Project objective Sequence information Error characteristics Spatial domain error concealment Temporal domain error concealment Quality Metrics Generation of errors References

3. Acronyms AVCAdvanced Video Coding AVSAudio Video Standard DSLDigital Subscriber Line HVSHuman Visual System IECInternational Electrotechnical Commission ISOInternational Organization for Standardization ITUInternational Telecommunication Union JMJoint Model LANLocal Area Network

4. Acronyms MMSMultimedia Messaging Service MSUMoscow State University PSNRPeak signal to noise ratio SADSum of absolute differences SSIMStructural similarity index metric

5. Problem Statement Video transmission errors are errors in the video sequence that the decoder cannot decode properly. In real-time applications, retransmission is not an option, therefore the missing parts of the video have to be concealed. To conceal these errors, spatial and temporal correlations of the video sequence can be utilized. As H.264 [3] employs predictive coding, this kind of corruption spreads spatio-temporally to the current and consecutive frames. This is seen in fig.1.

6. Problem Statement Figure 1: Error propagation in H.264 due to predictive coding [1]

7. Objective To implement both the spatial domain and temporal domain categories of error concealment techniques in H.264 [3] with the application of the Joint Model (JM 19.0) reference software [3] and use metrics like the peak signal to noise ratio (PSNR) and mean squared error (MSE), blockiness [1], blurriness [1] and structural similarity index metric (SSIM) [4] in order to compare and evaluate the quality of reconstruction.

8. H.264/AVC Block Diagram The H.264/AVC encoder/decoder block diagram is shown in fig.2. Figure 2: H.264 encoder/decoder block diagram [7]

9. Sequence Information Temporal information Movement -It is easier to conceal linear movements in one direction because we can predict pictures from previous frames (the scene is almost the same). -If there are movements in many directions, finding a part of a previous frame to fit, is going to be more difficult. An example of this can be screen cuts. - In fig.3, five frames of three video sequences (football match, village panorama and music video clip) with different amounts of movement are seen. Speed - The slower the movement of the camera, the easier it will be to conceal an error. - We can see an example of two different video speeds if we compare village sequence with football sequence given in fig.3.

10. Sequence Information Figure 3: Movement and speed [2]

11. Sequence Information Spatial information Smoothness of the neighborhood -The smoothness of the neighborhood of the erroneous macroblock will determine the difficulty of the spatial concealment. -In fig.4, there are three different cases in which the lost macroblock has to be concealed using the neighboring blocks. -In the first one, it is going to be easy to reconstruct the lost macroblock because the neighborhood is very uniform (smooth), with almost no difference between the neighboring macroblocks. -In the second situation, it is going to be a little bit more difficult; we have to look for the edges, and then, recover the wire line. -The third case is an example where the neighbors cannot help us to recover the macroblock because they do not give any information about the lost part (in this case, the eye).

12. Sequence Information Figure 4: Smoothness of the neighborhood [2]

13. Error Characteristics Lost information Size and form of the lost region I/P frame - If the error is situated in an I frame, it is going to affect more critically the sequence because it will affect all the frames until the next I frame and I frames do not have any other reference but themselves. - If the error is situated in a P frame it will affect the rest of the frames until the next I frame but we still have the previous I frame as a reference.

14. Spatial domain error concealment Weighted Averaging The pixel values of the damaged macroblock are obtained by interpolating the neighboring four pixels. Missing pixel values can be recovered by calculating the average pixel values from the four pixels in the four 1-pixel wide boundaries of the damaged macroblock weighted by the distance between the missing pixel and the four macroblocks boundaries (upper, down, left and right boundaries).

15. Temporal domain error concealment Frame Copy algorithm Replaces the missing image part in the current frame with the spatially corresponding part inside a previously decoded frame, which has maximum correlation with the affected frame. It is easier to conceal linear movements in one direction because pictures can be predicted from previous frames (the scene is almost the same). When the current frame ‘n’ is missing (i,j) values, they can be concealed using the (i,j) values from the ‘n-1’ frame. This is shown in fig.5. Figure 5: Copying (i,j) pixel values from previous ‘n-1’ frame into current ‘n’ frame [1] [1]

16. Temporal domain error concealment Frame copy algorithm If there are movements in many directions or scene cuts, finding a part of previous frame that is similar is going to be more difficult, or even impossible. The slower is the movement of the camera, the easier will be to conceal an error. This method only performs well for low motion sequences but its advantage lies in its low complexity.

17. Temporal domain error concealment

18. Motion Vector Interpolation A motion vector of a 4x4 block can be estimated by interpolating its value from the motion vectors in the surrounding macroblocks, as shown in fig.8. The distance between the 4x4-block to be found and the surrounding 4x4 blocks can be used as a weighting factor. This is shown in fig.7. Figure 7: (a) Weighted averaging in Motion Vector Interpolation [1] (b) Motion vector interpolation [1]

19. Quality Metrics Peak Signal to Noise Ratio (PSNR) and Mean Squared Error (MSE) Blockiness and Blurriness Structural Similarity Index Metric (SSIM)

20. PSNR and MSE

22. Distortion artifacts Here measurement of distortion artifacts like blockiness and blurring is done. Blockiness is defined as the distortion of the image characterized by the appearance of an underlying block encoding structure. [1] Blurriness is defined as a global distortion over the entire image, characterized by reduced sharpness of edges and spatial detail. [1] Blockiness measure [1]: This metric was created to measure the visual effect of blocking. If the value of the metric for first picture is greater than the value for the second picture, it means that first picture has more blockiness, than the second picture. An example is seen in fig.8. Blurriness measure [1]: This metric compares the power of blurring of two images. If the value of the metric for first picture is greater than the value for the second picture, it means that second picture is more blurred, than first. An example is seen in fig.9. The blockiness and blurriness value are computed using the Moscow State University Video Quality Measurement Tool [6] shown in fig.10.

23. Distortion artifacts Figure 8: (a) Orignal image (b) Blockiness in original image [1]

24. Distortion artifacts Figure 9: (a) Orignal image (b) Blurriness in original image [1]

25. MSU Video Quality Measurement Tool Figure 10: MSU Video Quality Measurement Tool [1]

26. Structural Similarity Index (SSIM) SSIM compares local patterns of pixel intensities (for similarities) that have been normalized for luminance and contrast. The block diagram for measuring the SSIM is given in fig.11. Figure 11: SSIM measurement [1]

27. Structural Similarity (SSIM)

29. Generation of Errors To generate errors we use the rtp_loss file in the software solution. This program can be used to simulate frame loss on RTP files. The C random number generator is used for selecting the frames to be lost. The number of frames to be lost depends on the ‘loss percent’ that the user provides. Example: rtp_loss.exe infile outfile losspercent where is an optional parameter that specifies the number of RTP frames that are kept at the beginning of the file. The random number generator is not initialized before usage. Therefore subsequent runs of the tool will create identical loss pattern. Figs. 12 and 13 show a single frame of Akiyo and Foreman sequences with and without frame loss.

30. Generation of Errors Figure 12: (a) Original Akiyo frame (b) Akiyo frame with error

31. Generation of Errors Figure 13: (a) Original Foreman frame (b) Foreman frame with error

32. References [1]I. C. Todoli, “Performance of Error Concealment Methods for Wireless Video,” Ph.D. thesis, Vienna University of Technology, 2007. [2]V.S. Kolkeri "Error Concealment Techniques in H.264/AVC, for Video Transmission over Wireless Networks", M.S. Thesis, Department of Electrical Engineering, University of Texas at Arlington, Dec. 2009 Online: http://www.uta.edu/faculty/krrao/dip/Courses/EE5359/index_tem.html. [3]H.264/AVC Reference Software Download: Online: http://iphome.hhi.de/suehring/tml/download/ [4]Z. Wang, “The SSIM index for image quality assessment,” Online: http://www.cns.nyu.edu/zwang/files/research/ssim/.

33. References [5]Video Trace research group at ASU, “YUV video sequences,” Online: http://trace.eas.asu.edu/yuv/index.html. [6] MSU video quality measurement tool: Online: http://compression.ru/video/quality_measure/video_measurement_tool_en.html. [7] S. –K. Kwon, A. Tamhankar and K.R. Rao, “An overview of H.264 / MPEG-4 Part 10,” J. Visual Communication and Image Representation, vol. 17, pp.186-216, Apr. 2006. [8]Y. Xu and Y. Zhou, “H.264 Video Communication Based Refined Error Concealment Schemes,” IEEE Transactions on Consumer Electronics, vol. 50, issue 4, pp. 1135–1141, Nov. 2004.

34. References [9]M. Wada, “Selective Recovery of Video Packet Loss using Error Concealment,” IEEE Journal on Selected Areas in Communication, vol. 7, issue 5, pp. 807-814, June 1989. [10] Y. Chen et al, “An Error Concealment Algorithm for Entire Frame Loss in Video Transmission,” IEEE Picture Coding Symposium, Dec. 2004. [11] S. K. Bandyopadhyay et al, “An error concealment scheme for entire frame losses for H.264/AVC,” IEEE Sarnoff Symposium, pp. 1-4, Mar. 2006. [12]H. Ha, C. Yim and Y. Y. Kim, “Packet Loss Resilience using Unequal Forward Error Correction Assignment for Video Transmission over Communication Networks,” ACM digital library on Computer Communications, vol. 30, pp. 3676-3689, Dec. 2007.

35. References [13] T. Wiegand et al, “Overview of the H.264/AVC video coding standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, July 2003. [14]Z. Wang et al, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004. [15]D. Levine, W. Lynch and T. Le-Ngoc, “Observations on Error Detection in H.264,” 50 th IEEE Midwest Symposium on Circuits and Systems, pp. 815-818, Aug. 2007. [16]B. Hrušovský, J. Mochná and S. Marchevský, “Temporal-spatial Error Concealment Algorithm for Intra-Frames in H.264/AVC Coded Video,” 20th International Conference Radioelektronika, pp. 1-4, Apr. 2010.

36. References [17]W. Kung, C. Kim and C. Kuo “Spatial and Temporal Error Concealment Techniques for Video Transmission Over Noisy Channels,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 16, issue 7, pp. 789-803, July 2006.