2. Bernd Girod. Joint Source-Network Coding for Real-time Media 1 Joint Source-Network Coding for Real-time Media Bernd Girod Information Systems Laboratory Stanford University

3. Bernd Girod. Joint Source-Network Coding for Real-time Media 2 Message from the President of the IEEE Information Theory Society “Dear Source Coders, Don’t compress too much and consequently require error-free transmission. It makes the task for us channel coders very difficult. We might have to add more redundancy for error protection than you removed!” Professor Joachim Hagenauer, 1996

4. Bernd Girod. Joint Source-Network Coding for Real-time Media 3 Outline Joint source-channel coding –Why it’s good idea –Why it’s bad idea Joint source-network coding –FEC and Smart Prefetching –Why joint source-channel coding is a good idea after all –Adaptive media playout –Packet Path Diversity

5. Bernd Girod. Joint Source-Network Coding for Real-time Media 4 Optimal Trade-Off Source Coding vs. Channel Coding 01020304050 10 20 30 40 50 60 70 80 90 100 25 % 9 % 6 % 2.5 % 0.7 % Intra Rate [%] Code Rate [%] Burst Length: Memory less 2 4 8 16 32 > Source coding redundancy > Channel coding redundancy K. Stuhlmüller, N. Färber, M. Link, and B. Girod, "Analysis of Video Transmission over Lossy Channels," IEEE Journal on Selected Areas in Communications, vol. 18, no. 6, pp. 1012-1032, June 2000.

6. Bernd Girod. Joint Source-Network Coding for Real-time Media 5 Shannon and the Internet Appl. #1 Transport Layer Internetworking Layer Appl. #2 Appl. #n Internet Open Architecture: 2000+ Network Access #1 Network Access #2 Network Access #m Channel coder Channel coder Channel decoder Channel decoder Channel Source coder Source coder Source decoder Source decoder Separation Principle Shannon, 1948 Joint Source-Channel Coding Contradicts Open Architecture? Joint Source-Channel Coding Contradicts Open Architecture? How to Build a System that Works, When Shannon’s Separation Principle Doesn’t? I hope I have shown you that it is beneficial to consider source and channel coding jointly, and pay particular attention to source decoding also.

7. Bernd Girod. Joint Source-Network Coding for Real-time Media 6 “Joint Source-Network Coding” From Joint Source-Channel Coding to Joint Source-Network Coding Internet –Many different channels with link-layer error control –Application (= source coder) shielded from and unaware of physical layer –Application “sees” late or lost packets, not bit-errors What can be done nevertheless –Careful network adaptation layer design (Avoid “chop and ship”) –Application specific error control (for packets erasures, not bit errors) –Jointly design and optimize –Source coding/decoding, transcoding & error concealment –Forward error correction, packet scheduling and retransmission, flow control, cacheing, routing

8. Bernd Girod. Joint Source-Network Coding for Real-time Media 7 FEC across Packets Reed-Solomon Codes RS(N,K) with erasure decoding Any K of N packets can used to reconstruct the information packets. U. Horn, K. Stuhlmüller, M. Link, and B. Girod, "Robust Internet Video Transmission Based on Scalable Coding and Unequal Error Protection," Image Communication, vol. 15, no. 1-2, pp. 77-94, Sept. 1999.

9. Bernd Girod. Joint Source-Network Coding for Real-time Media 8 Internet Media Streaming Workstation PC DSL 56K modem Media Server Internet GPRS Receiver buffer & ARQ Today: >10 sec delay

10. Bernd Girod. Joint Source-Network Coding for Real-time Media 9 Media Streaming with Smart Prefetching Basic idea: Retransmission protocol employing prefetching of more important information Server Client Internet Request stream Request stream Rate-distortion preamble Rate-distortion preamble Prefetch times Prefetch times Media data Repeat request Repeat request Repeat request Repeat request Repeat request Repeat request

11. Bernd Girod. Joint Source-Network Coding for Real-time Media 10 Smart Prefetching 2-Layer Video Results Single Layer Scalable: Optimal Scalable: Equal Prefetch 00.20.40.60.81 0 100 200 300 400 500 600 700 Packet loss rate MSE distortion at decoder Single layer Two layers, equal prefetch Two layers, optimal prefetch

12. Bernd Girod. Joint Source-Network Coding for Real-time Media 11 Adaptive Media Playout State-of-the-Art Improved scheme Idea: reduce latency and packet loss simultaneously by continuously adapting playout deadline to network conditions Idea: reduce latency and packet loss simultaneously by continuously adapting playout deadline to network conditions 5% packet loss 2 sec average receiver buffer 5% packet loss 2 sec average receiver buffer E. Steinbach, N. Färber and B. Girod, "Adaptive Playout for Low-Latency Video Streaming," Proc. International Conference on Image Processing, ICIP-2001, Thessaloniki, Greece, October 2001.

13. Bernd Girod. Joint Source-Network Coding for Real-time Media 12 Adaptive Media Playout Initial delay Initial delay Start of playout Throughput Receiver buffer level t t Target level Target rate Reduced playout speed buffer underflow avoided buffer underflow avoided

14. Bernd Girod. Joint Source-Network Coding for Real-time Media 13 Reduced Pre-roll Time for Stored Streams Probability of buffer underflow < 1% Probability of buffer underflow < 1% M. Kalman, E. Steinbach, and B. Girod, "Adaptive Media Playout for Low Delay Video Streaming over Error-Prone Channels," IEEE Transactions on Circuits and Systems for Video Technology, Special Issue on Wireless Video, submitted August 2000.

15. Bernd Girod. Joint Source-Network Coding for Real-time Media 14 Packet Path Diversity Idea: set up multiple connections along different paths Improved congestion resiliency –lower combined latency –better loss characteristics Realization in current Internet, e.g., through relay server D S Relay 1 2 cross traffic cross traffic Media traffic

16. Bernd Girod. Joint Source-Network Coding for Real-time Media 15 Multipath Voice Streaming Experiment Exodus Comm. BBN Planet Netergy Networks 192.84.16.176 MIT 18.184.0.50 (5 ms) (45 ms) (5 ms) Source Destination Qwest Harvard 140.247.62.110 (40 ms) (5 ms) Relay

17. Bernd Girod. Joint Source-Network Coding for Real-time Media 16 Adaptive playout with two streams Yi J. Liang, E. Steinbach and B. Girod, "Multi-stream Voice Transmission over the Internet Using Path Diversity," Proc. ACM Multimedia 2001, Ottawa, Canada, Sept./Oct. 2001.

18. Bernd Girod. Joint Source-Network Coding for Real-time Media 17 Multiple Description Coding for VoIP E Stream 1 Stream 2 O E O E O O E O E O E E O packet ii+1i+2i+3 Even samples: 8 bitOdd samples: 4 bit Odd samples: 8 bit Even samples: 4 bit O E 

19. Bernd Girod. Joint Source-Network Coding for Real-time Media 18 Packet Path Diversity: Loss vs. Delay >45%

20. Bernd Girod. Joint Source-Network Coding for Real-time Media 19 Conclusions Joint source-channel coding –Improved performance, if Separation Principle does not hold –Non-obvious trade-off between source coding and channel coding Joint source-network coding –Jointly optimize source coding/decoding and packet transport –Adaptive media playout: “real-time” more flexible than we thought –FEC across packets and smart prefetching can provide virtual priority mechanisms –Packet Path Diversity circumvents congestion, reduces latency Many good & relevant research problems!