1. Source-Channel Prediction in Error Resilient Video Coding Hua Yang and Kenneth Rose Signal Compression Laboratory ECE Department University of California, Santa Barbara

2. 7/8/03ICME 20032 Outline  Introduction  Source-channel prediction  Simulation results  Conclusions

3. 7/8/03ICME 20033 Introduction  Existent error resilient approaches on the prediction mechanism Slice coding  limit prediction within certain non-overlapping spatial regions Video redundancy coding  Multiple independently predicted “threads” Multi-frame motion compensation  Multiple reference frames for prediction  Feature in common: assume the same underlying conventional prediction framework Framework: separate source-channel coding Prediction: past encoder reconstructed frames Motion estimation criterion: minimum prediction error

4. 7/8/03ICME 20034 Introduction  Via considering packet loss effects during encoding, joint source-channel coding usually achieves better error resilience than that of separate coding.  Our proposed approach Prediction is based on expected decoder reconstruction of the previous frames. Novelty Unlike all the other existent error resilient prediction schemes and all the other existent source-channel coding schemes, our proposed method is actually a source-channel prediction scheme.

5. 7/8/03ICME 20035 Introduction 0 p1p1 p2p2 p3p3 pipi packet loss rate of video packet i. p1p1 p2p2 p2p2 p3p3 p3p3 p3p3 p3p3 1-p 1 1-p 2 1-p 3 1-p 2 1-p 3 Encoder reconstruction, i.e. “best possible” decoder reconstruction: quantization loss only. Other possible decoder reconstructions: different transmission loss patterns. Expected decoder reconstruction: quantization loss & transmission loss.

6. 7/8/03ICME 20036 Introduction  Expected decoder reconstruction Encoder’s estimate of the decoder reconstruction. Given the packet loss rate, it can be accurately computed with the ROPE method.  Recursive optimal per-pixel estimate (ROPE) Basic idea:  ROPE accurately computes these unknown quantities in a recursive manner for all the pixels of every frame. Accurate & L ow complexity Frequently used to estimate end-to-end distortion in various RD optimization scenarios. Now we use these expectations for source-channel prediction. Random variable unknown

7. 7/8/03ICME 20037 Source-channel Prediction  Conventional prediction Source-channel prediction Prediction residue Original and predicted values For pixel i in frame n : Encoder and decoder reconstruction values of pixel j in frame n-1 to predict pixel i in frame n. Prediction error to be quantized

8. 7/8/03ICME 20038 Source-channel Prediction  Source-channel prediction is the optimal prediction in the sense of minimum MSE end-to-end distortion.  Pending problem: motion estimation criterion ? Criterion in the conventional scheme Criterion I Constant value: Not the actual predictor of the decoder Plug in:

9. 7/8/03ICME 20039 Source-channel Prediction  Pending problem: motion estimation criterion? (cont.) Criterion II Random variable: Actual predictor of the decoder Criterion II is superior than Criterion I in that it explicitly accounts for the randomness of the decoder’s actual predictor.

10. 7/8/03ICME 200310 Source-channel Prediction  Another interpretation of Criterion II While Criterion II considers the properly weighted impacts of both D R and D D, in contrast, Criterion I only considers D R. In this sense, Criterion II is more “comprehensive”.

11. 7/8/03ICME 200311 Simulation Results  Simulation conditions H.263+ video codec System performance: average luminance PSNR 50 different packet loss patterns  Testing scenarios No INTRA Updating Periodic INTRA Updating For packet loss rate p, coding a MB in INTRA mode once for every 1/p frames. R-D optimized INTRA Updating For each MB, select its coding mode as INTER or INTRA with the R-D criterion.

12. 7/8/03ICME 200312 (a) No INTRA updating ( p = 10%) (b) Periodic INTRA updating.(c) RD optimal INTRA updating.

13. 7/8/03ICME 200313 Simulation Results  Observations The proposed “SCP_CII” method consistently offers the best performance, which proves our previous analysis. When INTRA updating is more effectively performed, smaller gains are achieved by “SCP_CII” over “EP”. Hence, the gain depends on how much damage of packet loss is not accounted for in the conventional scheme. Similar results also hold for other testing sequences, e.g., carphone, miss_am, salesman, etc.

14. 7/8/03ICME 200314 Demo Conventional prediction based on encoder reconstruction (PSNR = 25.06dB) Foreman, QCIF, 30f/s, 300kb/s, packet loss rate = 10%, periodic Intra update. Source-channel prediction based on expected decoder reconstruction (PSNR = 26.72dB)

15. 7/8/03ICME 200315 Conclusions  Novelty: the proposal of further enhancement of error resilience via fundamental modification of the conventional prediction structure.  Source-channel prediction based on expected decoder reconstruction, which uses ROPE to get accurate estimate of decoder quantities.  In spite of the loss in source coding gain due to the lower source prediction quality, our scheme achieves better overall R-D tradeoff than the conventional scheme.  We identify the subtle points in selecting the motion estimation criterion, and shows that it is advantageous to use the criterion of minimizing the expected prediction error.

16. 7/8/03ICME 200316 Thanks!