1. Ph.D. Dissertation Defense Modeling and Evaluating Feedback-Based Error Control for Video Transfer PhD Candidate: Yubing Wang - Computer Science, WPI, EMC Corp. Committee: Prof. Mark Claypool - Computer Science, WPI Prof. Robert Kinicki - Computer Science, WPI Prof. Dan Dougherty - Computer Science, WPI Prof. Ketan Mayer-Patel – Computer Science, UNC at Chapel Hill

2. Ph.D. Dissertation Defense 2 Video Transfer 5 Video Frames Internet Client 4321 Frame Loss Capacity Constraint Server 531 Delay Constraint Too Late Error Propagation 43215

3. Ph.D. Dissertation Defense 3 Error Control 5 Video Frames Internet Client 4321 Server Retransmission NACK 3 Change Coding Parameter Local Concealment 3

4. Ph.D. Dissertation Defense 4 Motivation Frame loss degrades video quality Feedback-based error control techniques use information from decoder to repair Feedback indicates damage location. Encoder and decoder cooperate in error control process. Better than error control techniques where no interaction between encoder and decoder Major techniques: RPS, Intra Update, Retransmission Choice and Effectiveness depends on packet loss, RTT, video content and GOP size No systematic exploration and comparison of impact of video and network conditions on the performance of feedback-based error control techniques

5. Ph.D. Dissertation Defense 5 The Dissertation Analyze video quality with feedback based error control Develop analytical models to predict quality of videos streamed with RPS NACK, RPS ACK, Intra Update or Retransmission Conduct systematic study of effects of reference distance on video quality Validate analytical models through simulations Analysis of loss rate, round-trip time, video content, Group Of Pictures (GOP) Determine choice between RPS NACK, RPS ACK, Intra Update or Retransmission Publications Impact of Reference Distance for Motion Compensation Prediction on Video Quality, MMCN07 An Analytic Comparison of RPS Video Repair, MMCN08 Modeling RPS and Evaluating Video Repair with VQM, IEEE Transactions on Multimedia, 2009, (to appear)

6. Ph.D. Dissertation Defense 6 Outline Introduction Background RPS ACK RPS NACK Intra Update Retransmission Impact of Reference Distance on Video Quality Analytical Models and Results Model Validations Conclusions

7. Ph.D. Dissertation Defense 7 Reference Picture Selection (ACK) The decoder acknowledges all correctly received frames Only the acknowledged frames are used as a reference Error propagation is avoided entirely Distance from reference frame is reference distance Reference distance increases with round-trip delay Coding efficiency decreases as reference distance increases Video quality degrades as coding efficiency decreases 1 2 345 67 ACK(1) ACK(2)ACK(3)

8. Ph.D. Dissertation Defense 8 Reference Picture Selection (NACK) The previous frame is used as a reference for encoding during the error-free transmission. Reference distance is always 1 regardless of RTT The decoder sends a NACK for the erroneous frame along with a reference frame number Error propagation Impact of loss increases with RTT NACK(3) 1 2 34 5678

9. Ph.D. Dissertation Defense 9 Intra Update Upon receiving a NACK from the decoder, encodes the current frame with intra mode Frame is independently encoded without using any information from previous frames Coding efficiency is reduced because of intra coding 1 2 3456789 NACK(4) Intra-coded

10. Ph.D. Dissertation Defense 10 Retransmission Retransmission of lost frames needs extra bandwidth Packets arriving after their display times are not discarded but instead are used to reduce error propagation 1 2 34 56789 1 2 34 56789 Encoder Decoder NACK(3) 3

11. Ph.D. Dissertation Defense 11 Outline Introduction Background Impact of Reference Distance on Video Quality Hypothesis Methodology Results and Analysis Analytical Models and Results Model Validations Conclusions

12. Ph.D. Dissertation Defense 12 Impact of Reference Distance on Video Quality RPS selects one of several previous frames as a reference frame during compression Distance from selected frame is reference distance Higher reference distance, lower quality and vice versa How reference distance affects video quality has not been quantified A systematic study of the effects of reference distance on video quality Data is needed for modeling RPS

13. Ph.D. Dissertation Defense 13 Hypothesis The y-intersect is determined by motion and scene complexity. High-motion video sequences starts with low quality, degrade slower. Low-motion video sequence starts with high quality, degrade faster. Low Motion: The similarities among frames are high; More macro-blocks are inter-coded; High motion: The similarities among frames are low; More macro-blocks are intra-coded;

14. Ph.D. Dissertation Defense 14 Methodology Select a set of non-compressed video clips with a variety of motion content. All in YUV 4:2:2, CIF (352x288) Each video sequence contains 300 video frames with a frame rate of 30 fps. Change reference distances for each selected video sequence Encode the video clips using H.264 Measure video quality using Peak-Signal-to-Noise-Ratio (PSNR) Video Quality Metric (VQM) Analyze the results.

15. Ph.D. Dissertation Defense 15 PSNR vs. Reference Distance Video ClipsabR- Squared Akiyo-2.011647.9650.9953 Container-1.902344.8380.9948 News-1.855643.2950.9984 Silent-1.528341.410.9929 Mom & Daughter -1.458141.4420.9904 Foreman-1.168138.5110.9265 Mobile-1.155326.6630.9754 Coastguard-0.862635.5820.9975 The relationship between PSNR and reference distance can be characterized using a logarithmic function:

16. Ph.D. Dissertation Defense 16 VQM vs. Reference Distance Video ClipsabR- Squared Akiyo-0.01130.98470.9869 Container-0.01140.97660.9848 News-0.01150.97320.9931 Silent-0.01240.96060.9937 Mom & Daughter -0.00850.92170.9821 Foreman-0.00680.90590.9779 Mobile-0.00220.80550.9076 Coastguard-0.00140.84230.9671 The relationship between VQM and reference distance can be characterized using a linear function:

17. Ph.D. Dissertation Defense 17 Outline Introduction Background Impact of Ref. Distance on Video Quality Analytical Models and Results Assumptions RPS ACK RPS NACK Intra Update Retransmission Result & Analysis Model Validations Conclusions

18. Ph.D. Dissertation Defense 18 Assumptions Each GOB is independent from other GOBs in the same frame. An independent video sub- sequence is referred to as a reference chain. Each GOB is carried in a single network packet. Reliable transmission of feedback messages are assumed. Erroneously-decoded GOBs are repaired by local concealment. Make no assumption on specific local concealment techniques. 1 2 34567 Assume independent packet loss with a random loss distribution. In this talk, GOB and Frame is exchangeable.

19. Ph.D. Dissertation Defense 19 Model Parameters

20. Ph.D. Dissertation Defense 20 Modeling of RPS ACK The probability of decoding GOB (n) correctly using GOB (n-δ-i) as a reference: The probability of GOB (n) being successfully decoded is: p Packet loss probability Probability of GOB (n-δ-i) being successfully decoded Round-trip time Time-interval between two frames ACK(1) 1 2 345 ACK(2)

21. Ph.D. Dissertation Defense 21 RPS ACK Modeling (cont.) The expected video quality for n-th GOB: Average video quality for a GOB encoded using the GOB that is r GOBs backward. Average video quality for a Intra-Coded GOB Average PSNR value for a GOB that is repaired using local concealment

22. Ph.D. Dissertation Defense 22 RPS NACK -- Model The probability of GOB (n) being successfully decoded: --- the probability of decoding GOB (n) correctly using GOB (n- δ -i) as a reference 1- p p pp p p p [1] (1) [2] (1) [1] (2) (1) (3)[3] GOB 1 GOB 2 GOB 3 GOB 4 (2) p p root [1] A B C D NACK(1) 1 2 345 NACK(2) GOB Dependency Tree

23. Ph.D. Dissertation Defense 23 Intra Update -- Model The probability of GOB (n) being successfully decoded: -- the probability of decoding GOB (n) correctly using Intra coding 1 2 345 NACK Intra-coded 1- p p pp p p p GOB 1 GOB 2 GOB 3 GOB 4 p p A E B root C D F GOB Dependency Tree

24. Ph.D. Dissertation Defense 24 Retransmission C apacity constraint : The n-th GOB in the reference chain being successfully decoded : The expected video quality for GOB (n):

25. Ph.D. Dissertation Defense 25 Outline Introduction Background Impact of Ref. Distance on Video Quality Analytical Models and Results Assumptions RPS ACK RPS NACK Intra Update Retransmission Result & Analysis Model Validations Conclusions

26. Ph.D. Dissertation Defense 26 Analytic Experiments Our analytical models consider a number of factors that may affect feedback-based repair performance: Reference distance change Loss probability Round-trip time Bitrate constraint Video content GOP Size Select a set of video clips with a variety of motion content

27. Ph.D. Dissertation Defense 27 Quality versus Round-Trip Time RPS ACK RPS NACK Quality degrades with round-trip time increase NACK resistant to degradation with round-trip time for low loss ACK degrades uniformly with round-trip time

28. Ph.D. Dissertation Defense 28 Quality versus Loss Rate RPS ACK RPS NACK Quality degrades with loss rate increase NACK degrades faster with high round trip times ACK uniform degradation

29. Ph.D. Dissertation Defense 29 RPS NACK vs. RPS ACK Above trend line, ACK better. Below trend line, NACK better Crossover points for low-motion are higher than for high-motion Error propagation more harmful to quality than reference distance

30. Ph.D. Dissertation Defense 30 Comparison RPS NACK performs best in low loss RPS ACK performs best in high loss RPS ACK performs worst in low loss Retransmission performs worst in high loss Intra Update performs as well as RPS NACK as RTT increases RTT=80 ms RTT=240 ms

31. Ph.D. Dissertation Defense 31 Outline Introduction Background Impact of Ref. Distance on Video Quality Analytical Models and Results Model Validations Methodology Results Conclusions

32. Ph.D. Dissertation Defense 32 Validation -- Methodology Randomly drop controllable number of frames in input sequence based on given loss probability Based on given round-trip time and randomly selected lost frames, regenerate video sequence Encode video sequence generated in step 2 using H.264 Measure average PSNR and VQM for encoded H.264 video sequence Calculate average PSNR and VQM based upon video quality measured in step 4 1(I) 2(P) 5(P) 6(P) 7(P) RPS NACK, round-trip time = 2 frames, frame 3 is lost

33. Ph.D. Dissertation Defense 33 Validation – RPS NACK Error bar represents 95% confidence interval As loss probability or round-trip time increases, the variance is increased Simulation results are consistent with values predicted by analytical model for both PSNR and VQM

34. Ph.D. Dissertation Defense 34 Outline Introduction Background Impact of Ref. Distance on Video Quality Analytical Models and Results Model Validations Conclusions

35. Ph.D. Dissertation Defense 35 Major Contributions 1.Systematic study of effects of reference distance on video quality for a range of video coding conditions 2.Two utility functions that characterize impact of reference distance on video quality based upon study 3.Modeling prediction dependency among GOBs for RPS NACK and Intra Update using binary tree 4.Analytical models for feedback-based error control techniques including Full Retransmission, Partial Retransmission, RPS ACK, RPS NACK and Intra Update 5.Simulations that verify accuracy of our analytical models 6.Analytic experiments over a range of loss rates, round-trip times and video content using our models

36. Ph.D. Dissertation Defense 36 Future Work Explore and incorporate other existing video quality metrics or develop a new quality metric Investigate how local concealment may affect the choice of feedback-based repair techniques Investigate the impact of the extra bandwidth consumed by feedback messages on performance Build a videoconference system that automatically adapts to the best repair techniques

37. Ph.D. Dissertation Defense 37 Conclusions Degree of video quality degradation is affected by video content High-motion video sequences starts with lower quality, degrade slower. Low-motion video sequences starts with higher quality, degrade more rapidly. Mathematical Characterization of the relationship between video quality and reference distance: PSNR: VQM: Analytical models reveal: RPS NACK performs best in low loss RPS ACK performs best in high loss, worst in low loss RPS NACK outperforms RPS ACK over a wider range for low motion videos than for high motion videos Retransmission performs worst in high loss Intra Update performs as well as RPS NACK as RTT increases

38. Ph.D. Dissertation Defense 38 Acknowledge Prof. Claypool and Prof. Kinicki Prof. Dougherty Prof. Mayer-Patel from UNC at Chapel Hill Faculty/Staff of Computer Science Dept., WPI Huahui Wu, Mingze Li, Feng Li, and everyone from PEDS and CC groups Attendees today My Family