1. CS335 Principles of Multimedia Systems Multimedia Over IP Networks -- II Hao Jiang Computer Science Department Boston College Nov. 8, 2007

2. CS335 Principles of Multimedia Systems Real-time Transport Protocol  Real-time transport protocol (RTP) is an Internet- standard protocol for transmitting real-time multimedia over IP network.  RTP is (often) built on top of UDP.  A RTP packet is embedded in a UDP packet and provides additional information to support real-time multimedia communications –Timestamp. –Sequence number and source id. UDP header RTP header Multimedia Payload

3. CS335 Principles of Multimedia Systems The RTP Packet Header Timestamp is a 32bit integer placed on the RTP packet. It is incremented based on the local system clock. For video it is usually 90kHz. Sequence number is a 16bit integer incremented by 1 for each generated RTP packet. PT: payload type. For example, 7 is for LPC audio, and 34 is for H263 video.

4. CS335 Principles of Multimedia Systems Example: Motion JPEG Payload Packet  Timestamp is incremented at 90KHz. The marker M bit is set at the last packet for each frame. Main JPEG header following the RTP header There are other headers to provide information like quantization table, etc.

5. CS335 Principles of Multimedia Systems RTCP  Real Time Control Protocol (RTCP) packets can be generated by both a sender and a receiver.  RTCP uses a different UDP port from RTP.  RTCP packets can be used to generate reports for network statistics such as round trip time, jitter and to synchronize different information sources. VPICLength Format Specific Information RCTP Packet Format PT

6. CS335 Principles of Multimedia Systems RTCP Receiver Report  Payload type = 201.  RTCP RR packet reports include –Loss fraction: ratio of lost packets # / expect packets #. –Cumulative number of packets lost. –Highest sequence number received. –Inter-arrival jitter D(i) = (R(i) – R(i-1)) – (S(i) - S(i-1)) J(i+1) = J(i) * 15/16 + | D(i) | * 1/16 –Last send report timestamp (LSR) and delay since last send report (DLSR) [NTP time] round trip time = current timestamp - LSR - DLSR sender receiver DLSR LSR Current Timestamp

7. CS335 Principles of Multimedia Systems RTCP Sender Report  Payload type = 200.  RTCP SR packet contains –NTP timestamp. –RTP timestamp. –Reporter SSRC and others.  The most important usage of RTCP SR packet is to synchronize different media streams.

8. CS335 Principles of Multimedia Systems RTP Example Application capturing Audio Codec, Packing, Depacking Playing back encoding buffer Playback buffer RTP Packet buffer Internet Client n Client k Client i Audio Conferencing based on RTP

9. CS335 Principles of Multimedia Systems RTP Media Sender … … Media buffer Compressed media frames with relative time stamps Buffer until send time comes fragmentation Adding sequence number and other fields to make RTP packets …… Send to UDP module System clock

10. CS335 Principles of Multimedia Systems RTP Media Receiver Play back buffer RTP packet with timestamp t > t1and t < t2 (Packet with timestamp t1) (Packet with timestamp t2) decompress Scheduled to play out We usually need another buffer for de-fragmentation. Packet buffer

11. CS335 Principles of Multimedia Systems Timing 20ms audio encoding making a packet Across The network 20ms audio Sender clock Receiver clock 20ms audio What happens if receiver clock and sender clock has different frequency?

12. CS335 Principles of Multimedia Systems Methods to Compensate for Clock Skew  A method is to directly compute the skew: sender timestamp n1 = (t1 – tx)/dtx packet arrive timestamp n2 = (t2 – ty)/dty Assume we have 2 packets a and b, then n1(a) – n1(b) = [t1(a) – t1(b)]/dtx n2(a) – n2(b) = [t2(a) – t2(b)]/dty => (n2(a) – n2(b)) / (n1(a) – n1(b)) = dtx/dty + D(a,b)/(t1(a) – t1(b))*dtx/dty If (t1(a) – t1(b)) is constant, we can do a long term average to remove the noise term.

13. CS335 Principles of Multimedia Systems Compensation for Jitter  The leaky bucket model: … Similar to a leaky bucket, we can use a buffer to shape the network traffic to generate a smooth playback. Buffer

14. CS335 Principles of Multimedia Systems Compensation for Jitter t1t2 N/(t2-t1) # of Accumulated Packets

15. CS335 Principles of Multimedia Systems Compensation for Jitter t1t2 N/(t2-t1) # of Accumulated Packets Production Curve Consumption Curve

16. CS335 Principles of Multimedia Systems Compensation for Jitter t1t2 N/(t2-t1) # of Accumulated Packets Production Curve Consumption Curve

17. CS335 Principles of Multimedia Systems Compensation for Jitter t1t2 # of Accumulated Packets Production Curve Consumption Curve

18. CS335 Principles of Multimedia Systems Media Synchronization  In RTP, different media streams (audio and video) are sent separately.  Media synchronization relies on the correspondence of timestamp and NTP time on the local machine.  All the information sources need to be synchronized so that they have the same NTP time base.  RCTP SR packet is used to align the timestamp with the NTP time in media synchronization.

19. CS335 Principles of Multimedia Systems Useful Links  http://www.realnetworks.com/support/education/sdk.h tml http://www.realnetworks.com/support/education/sdk.h tml  http://www.w3.org/AudioVideo/Activity http://www.w3.org/AudioVideo/Activity  http://www.networksorcery.com/enp/protocol/rtp.htm