1. Error Control and Concealment for Video Communication CMPT820 Summer 2008 Michael Jia

2. “Error Control and Concealment for Video Communication : A Review” YAO WANG, Member, IEEE QIN-FAN ZHU, Member, IEEE Reference

3.  Introduction  Error Detection  Error Concealment at Coder  Error Concealment at Decoder  Interactive Error Concealment  Conclusion Outline

4. 2 Types of Transmission Errors Random Bit Errors Bit inversion, bit insertion, bit deletion Erasure Errors Packet loss, burst errors, system failures More destructive Common usage of VLC makes them no differences in video streaming Introduction

5. Lossless recovery FEC (Forward Error Correction) ECC (Error Control Coding) ARQ (Automatic Retransmission Request) Not necessary for video transmission, human eyes can tolerate a certain degree of distortion Focus on signal-reconstruction and error-concealment techniques Introduction

6. Typical structure of a video communication system Introduction

7. 3 groups of error-concealment techniques Forward error concealment Encoder plays the primary role Error concealment by post-processing Decoder fulfills the task Interactive error concealment Main concerns Effectiveness Required delay Bit-rate overhead Processing complexity Introduction

8.  Introduction  Error Detection  Error Concealment at Coder  Error Concealment at Decoder  Interactive Error Concealment  Conclusion Outline

9. At transport layer Adding header information (sequence number) H.223 FEC (Forward Error Correction) H.223, H.261 More reliable Need more bandwidth Error Detection

10. At decoder Based on characteristics of natural video signals Pixel value differences of neighboring lines (compare to a threshold) Differences between boundary pixels in a block and its four neighbor blocks Obvious false value of quantization step size or DCT coefficients Insert synchronization code word at the end of line of blocks No additional bits or very few Rely on smoothness of signal Error Detection

11.  Introduction  Error Detection  Error Concealment at Coder  Error Concealment at Decoder  Interactive Error Concealment  Conclusion Outline

12. Layered Coding with Transport Prioritization Most popular and effective (MPEG-2) Error Concealment at Coder (1)

13. Layered Coding Base layer – most important layer, with acceptable quality Transport prioritization Deliver base layer with higher degree of error protection High priority channel Re-transmission and/or FEC No explicit bit-rate overhead Complicate structure and coding overhead H.264 AVC/SVC Redundant pictures Data Partitioning Error Concealment at Coder (1)

14. Multiple-Description Coding (MDC) Several parallel channels Independent error events Small probability of “all channels down” Multiple “Descriptions” Several coded bit streams Transmitted over separate channels Any one will work Error Concealment at Coder (2)

15. Multiple-Description Coding (MDC) Error Concealment at Coder (2)

16. Joint Source and Channel Coding Source-channel interaction at a lower level Given channel error characteristics, design quantizer and entropy coder for to minimize the effect of errors For general sources, noisy channel  coarse quantizer is better For image signals, noisy channel  fewer bits to high-frequency coefficients more bits to low-frequency coefficients Error Concealment at Coder (3)

17. Robust Waveform Coding Intentionally keep some redundancy in source-coding Layered coding and MDC both belong to this category Adding auxiliary information in waveform coder MPEG-2: sending motion vectors for microblocks in I-frames Restricting prediction domain H.263/H.264: prediction is confined within each slice Error Concealment at Coder (4)

18. Robust Entropy Coding Add redundancy in entropy-coding To help detect bit errors and prevent error propagation Self-Synchronizing entropy coding Add a synchronization code word H.261, H.263, MPEG-4 Error-Resilient entropy coding (EREC) MPEG-4 uses RVLC (reversible VLC) Error Concealment at Coder (5)

19. FEC Coding Guarantee lossless data delivery Induct a lot of overhead, reduce usable bandwidth Maybe too strong in video services Cases H.261 in ISDN MPEG-2 in wireless ATM local network Error Concealment at Coder (6)

20. Transport-Level Control Add redundancies at transport level Prioritized transport for layered coding Robust packetization Spatial block interleaving Dual transmission of important information H.264 AVC/SVC NAL unit syntax structure Parameter Sets Error Concealment at Coder (7)

21. Error Concealment at Coder (Summary)

22.  Introduction  Error Detection  Error Concealment at Coder  Error Concealment at Decoder  Interactive Error Concealment  Conclusion Outline

23. Preview Perform error concealment at the decoder Can be used in conjunction with the auxiliary information provided by the source coder Low frequency components dominate images of natural scenes Color values of adjacent pixels vary smoothly except sharp edges Human eyes tolerate more distortion to high-frequency components Error Concealment at Decoder

24. Motion-Compensated Temporal Prediction Replace damaged MB with the motion compensated block Very effective when all the motion information in the base layer Widely used (MPEG-2) What if motion information or coding mode is lost? Will discuss in next slide Error Concealment at Decoder (1)

25. Recovery of Motion Vectors and Coding Modes What if we lost motion vectors or coding modes? Interpolated from spatially and temporally adjacent blocks Estimation of coding modes Simple: treat as ‘intracoded’ More sophisticate: MPEG-2 (See tables) Error Concealment at Decoder (2)

26. Estimation of motion vectors Set to zeros (works well for low motion video) Use MV of the corresponding block in the previous frame Use the average of MVs from spatially adjacent blocks Use the median of MVs from spatially adjacent blocks Select one of the above methods depending on least boundary matching error Error Concealment at Decoder (2)

27. Maximally Smooth Recovery A constrained energy minimization approach Minimize a measure of spatial and temporal variation between adjacent pixels in this block and its spatially and temporally neighboring blocks Measure differences in 3 domains Spatial – adjacent blocks Temporal – prediction block in previous frame Frequency – received coefficients for this block Error Concealment at Decoder (3)

28. Maximally Smooth Recovery 2 sample spatial smoothness measures Error Concealment at Decoder (3)

29. Spatial- and Frequency-Domain Interpolation A coefficient in a damaged block is likely to be close to the corresponding coefficients in spatially adjacent blocks Interpolate from four neighbor blocks Maybe not accurate (8-pixel is too far) Interpolate from four 1-pixel-wide boundaries 2 pixels in 2 nearest boundaries 4 pixels in all 4 boundaries See picture in next slide Error Concealment at Decoder (4)

30. Spatial- and Frequency-Domain Interpolation Interpolate from four 1-pixel-wide boundaries Error Concealment at Decoder (4)

31. Projection onto Convex Sets (POCS) The convex sets are derived by requiring the recovered block to have a limited bandwidth either isotropically (for a block in a smooth region) or along a particular direction (for a block containing a straight edge) A combined block is formed by including eight neighboring blocks with the damaged block Error Concealment at Decoder (5)

32. Error Concealment at Decoder (Summary)

33.  Introduction  Error Detection  Error Concealment at Coder  Error Concealment at Decoder  Interactive Error Concealment  Conclusion Outline

34. Preview If a backward channel is available, can achieve better performance by cooperation Based on the feedback At source coder – coding parameters can be adapted At transport level – adjust the portion of bandwidth used for error control Decoding delay issue Interactive Error Concealment

35. Selective Encoding for Error Concealment Simple – code next frame in intramode Error stopped in about one round-trip time Will cause bit-rate increase Send identity info back, perform error concealment at the same time Continue encode without using the affected area Perform same error concealment procedure (need a prediction frame buffer) See picture in next slide Interactive Error Concealment (1)

36. Selective Encoding for Error Concealment Interactive Error Concealment (1)

37. Selective Encoding for Error Concealment Interactive Error Concealment (1)

38. H.263 uses more prediction frame buffers (reference picture selection mode) Interactive Error Concealment (1)

39. Adaptive Transport for Error Concealment Retransmission is unacceptable for real-time video applications? Not always For one-way video broadcast, we may tolerate a few seconds delay For multipoint video conferencing, use MCU (multipoint control unit) If retransmission is controlled properly, end-to-end quality can be improved Both H.323 and H.324 defined such mechanisms Interactive Error Concealment (2)

40. Retransmission Without Waiting Wait for the retransmission data Not good, may freeze the display Cause transit delay and accumulation delay Without waiting Request the retransmission Conceal the error Track the affected pixels Correct them upon the arrival of the retransmission data Interactive Error Concealment (3)

41. Prioritized, Multi-copy Retransmission Effective in very lossy channels Video streaming via PSTN Are you kidding me? Send multiple copies of a lost packet Use layered coding # of retransmission trials and # of copies are proportional to the importance of the layer Interactive Error Concealment (4)

42. Interactive Error Concealment (Summary)

43.  Introduction  Error Detection  Error Concealment at Coder  Error Concealment at Decoder  Interactive Error Concealment  Conclusion Outline

44. Conclusion Real-time video communication doesn’t require lossless delivery; signal-reconstruction and error-concealment techniques are more effective. Add redundancy when encoding or delivering Estimate missing information when decoding Inform sender what is lost Burstiness has a significant impact on the choice of algorithms

45. Questions?

46. Thank You