1. 1 Multimedia Networking R. Yang

2. 2 Outline r Admin. and review  Introduction to multimedia networking r An architecture of stored multimedia r Network support of multimedia app. r Application adaptation

3. 3 Preview: Multimedia Networking Our goals: r Understand the service requirements of networked multimedia applications r Learn the high level structures designed for multimedia networking r Deal with multimedia in a best-effort network r How the Internet might evolve to better support multimedia? Schedule of the two classes: r Class 1: Multimedia applications; application adaptation in best-effort networks r Class 2: New architectures: IntServ and DiffServ

4. 4 MM Networking Applications Fundamental characteristics: r Typically delay sensitive m end-to-end delay m delay jitter r But loss tolerant: infrequent losses cause minor glitches r Antithesis of data, which are loss intolerant but delay tolerant. Classes of MM applications: 1) Streaming stored audio and video 2) Streaming live audio and video 3) Real-time interactive audio and video

5. 5 Streaming Stored Multimedia Streaming: r media stored at source r transmitted to client r streaming: client playout begins before all data has arrived r timing constraint for still-to-be transmitted data: in time for playout

6. 6 Streaming Stored Multimedia: Interactivity r VCR-like functionality: client can pause, rewind, FF, push slider bar m 10 sec initial delay OK m 1-2 sec until command effect OK

7. 7 Streaming Live Multimedia Examples: r Internet radio talk show r Live sporting event Streaming r play can lag tens of seconds r still have timing constraint (but shorter delay required) Interactivity r fast forward impossible r rewind, pause possible!

8. 8 Interactive, Real-Time Multimedia r end-end delay requirements: m audio: < 150 msec good, < 400 msec OK includes application-level (packetization) and network delays higher delays noticeable, impair interactivity r session initialization m callee be able to advertise its IP address, port number, encoding algorithms. r applications: IP telephony, video conference, distributed interactive worlds

9. 9 Outline r Admin. and review r A brief introduction to the physical layer r Introduction to multimedia networking  An architecture of stored multimedia r Network support of multimedia app. r Application adaptation

10. 10 Streaming from Web Server (1) r Audio and video files stored in Web servers r Browser requests file with HTTP request message r Web server sends file in HTTP response message r Content-type header line indicates an audio/video encoding r Browser launches media player, and passes file to media player r Media player renders file Major drawback: media player interacts with server through intermediary of a Web browser

11. 11 Streaming from Web Server (2) Alternative: set up connection between server and player r Web browser requests and receives a meta file (a file describing the object) instead of receiving the file itself r Content-type header indicates specific audio/video application r Browser launches media player and passes it the meta file r Player sets up a TCP connection with server and sends HTTP request Some concerns: r Media player communicates over HTTP, which is not designed with pause, ff, rewind commands r May want to stream over UDP

12. 12 RTSP/RTCP/RTP HTTP GET media player Web server media server Web browser clientserver presentation desc. RTSP RTCP RTP RTSP: Real-Time Signaling Protocol RTCP: Real-Time Control Protocol RTP: Real-Time Transport Protocol

13. 13 Start: Using HTTP to get Presentation File Twister <track type=audio e="PCMU/8000/1" src = "rtsp://audio.example.com/twister/audio.en/lofi"> <track type=audio e="DVI4/16000/2" pt="90 DVI4/8000/1" src="rtsp://audio.example.com/twister/audio.en/hifi"> <track type="video/jpeg" src="rtsp://video.example.com/twister/video"> Example presentation file

14. 14 Control: RTSP, Out of Band Control RTSP messages are sent out-of-band: r The RTSP control messages use a different port number (554) than the media stream, and are therefore sent out-of- band r RTSP can be sent over UDP or TCP r Each RTSP message can be sent over a separate TCP connection r The media stream, whose packet structure is not defined by RTSP, is considered “in-band” r If the RTSP messages were to use the same port number as the media stream, then RTSP messages would be said to be “interleaved” with the media stream

15. 15 RTSP: Initialization and Control HTTP GET SETUP PLAY media stream (RTP) media player Web server media server Web browser client server presentation desc. TEARDOWN PAUSE RTCP

16. 16 RTSP: Exchange Example C: SETUP rtsp://audio.example.com/twister/audio RTSP/1.0 Transport: rtp/udp; compression; port=3056; mode=PLAY S: RTSP/1.0 200 1 OK Session 4231 C: PLAY rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=0- C: PAUSE rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=37 C: TEARDOWN rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 S: 200 3 OK

17. 17 Data: Real-Time Protocol (RTP) r RTP runs in the end systems r RTP specifies a packet structure for packets carrying audio and video data: RFC 1889 r RTP packets are encapsulated in UDP segments r Interoperability: If two Internet phone applications run RTP, then they may be able to work together

18. 18 RTP Header - Payload Type - e.g. type 0: (Pulse Coded Modulation) PCM mu-law, 64 Kbps; type 3, GSM, 13 Kbps; type 33, MPEG2 video Question: why include payload type in each packet? - Sequence Number - SSRC field - Identifies the source of the RTP stream. Each stream in a RTP session should have a distinct SSRC - Timestamp field - Reflects the sampling instant of the first byte in the RTP data packet 8 16 32

19. 19 RTP Header (2) r Timestamp field m Reflects the sampling instant of the first byte in the RTP data packet m The timestamp is derived from a sampling clock at the sender m Example audio stream sampled every 125 usecs audio application generates chunks consisting of 160 encoded samples then the timestamp increases by 160 for each RTP packet when the source is active the timestamp clock continues to increase at a constant rate even when the source is inactive 8 16 32

20. 20 H.323 Overview r Foundation for audio and video conferencing across IP networks r Targets real-time communication (rather than on- demand) r Umbrella recommendation from the ITU r Broad in scope: m stand-alone devices (e.g., Web phones) m applications in PCs

21. 21 Overview H.323 terminalsTelephone: SS7, Inband InternetPSTN Gateway Gatekeeper

22. 22 H.232 Big Picture H.323 terminals Gatekeeper Router Internet LAN = “Zone” RAS

23. 23 Gateway H.323 terminals Gatekeeper Router Internet LAN = “Zone” RAS Gateway PSTN Bridge between IP Zone and PSTN (or ISDN) network. Terminals communicate with gateways using H.245 and Q.931

24. 24 Gatekeeper The gatekeeper is optional. Can provides to terminals: address translation to IP addresses bandwidth management: can limit amount of bandwidth consumed by real-time conferences Optionally, H.323 calls can be routed through gatekeeper. Useful for billing. RAS protocol (over TCP) for terminal-gatekeeper communication. H.323 terminals Gatekeeper Router Internet LAN = “Zone” RAS

25. 25 H.323 Terminal

26. 26 H.323 Endpoints Must Support: r G.711 - ITU standard for speech compression r RTP - protocol for encapsulating media chunks into packets r H.245 - “Out-of-band” control protocol for controlling media between H.323 endpoints. r Q.931 - A signalling protocol for establishing and terminating calls. r RAS (Registration/Admission /Status) channel protocol - Protocol for communicating with a gatekeeper (if gatekeeper is present)

27. 27 H.323 Encoding Audio: r H.323 endpoints must support G.711 standard for speech compression. G.711 transmits voice at 56/64 kbps. r H.323 is considering requiring G.723 = G.723.1, which operates at 5.3/6.3 kbps. r Optional: G.722, G.728, G.729 Video r Video capabilities for an H.323 endpoint are optional. r Any video-enabled H.323 endpoint must support the QCIF H.261 (176x144 pixels). r Optionally supports other H.261 schemes: CIF, 4CIF and 16CIF. r H.261 is used with communication channels that are multiples of 64 kbps.

28. 28 Generating Audio Packet Stream in H.323 Audio Source Encoding: e.g., G.711 or G.723.1 RTP packet encapsulation UDP socket Internet or Gatekeeper

29. 29 H.245 Control Channel r H.323 stream may contain multiple channels for different media types. r One H.245 control channel per H.323 session. r H.245 control channel is a reliable (TCP) channel that carries control messages. r Principle tasks: m Open and closing media channels. m Capability exchange: before sending media, endpoints agree on encoding algorithm r Note: H.245 for multimedia conferencing is analogous with RTSP for media streaming

30. 30 Information flows

31. 31 Gatekeeper 2/2 r H.323 terminal must register itself with the gatekeeper in its zone. m When the H.323 application is invoked at the terminal, the terminal uses RAS to send its IP address and alias (provided by user) to the gatekeeper. r If gatekeeper is present in a zone, each terminal in zone must contact gatekeeper to ask permission to make a call. r Once it has permission, the terminal can send the gatekeeper an e-mail address, alias string or phone extension. The gatekeeper translates the alias to an IP address. m If necessary, a gatekeeper will poll other gatekeepers in other zones to resolve an IP address. Process varies by vendor.

32. 32 Data Control: Real-Time Control Protocol (RTCP) r Works in conjunction with each real-time RTP session r Each participant in an RTP session periodically transmits RTCP control packets to all other participants, e.g., m Receiver-report packets: fraction of packets lost, last sequence number, average inter-arrival jitter m Sender-report packets: the number of packets sent, the timestamp and the wall-time of the last sent packet r RTCP attempts to limit its traffic to 5% of the session bandwidth

33. 33 Outline r Admin. and review r A brief introduction to the physical layer  Introduction to multimedia networking r An architecture of stored multimedia  Network support of multimedia app. r Application adaptation in best-effort Internet

34. 34 Multimedia in Networks: Fundamental Characteristics r Many have requirements on rate, i.e. long-term average bandwidth requirement (e.g. 13kbps phone conversion) r Many are delay and jitter sensitive m jitter is the variability of packet delays within the same packet stream. r Many are loss tolerant m infrequent losses cause minor glitches that can be concealed m but high loss rate makes the application useless r Antithesis of data (programs, banking info, etc.), which are loss intolerant but delay tolerant; multimedia is also called “continuous media” rate utility

35. 35 Why Best-effort Internet May not Work Well? r Variable bandwidth m dynamic bandwidth sharing r Potentially long delays m Assume simple arrival and service processes (Poisson). Suppose the utilization of a link is  (<1), the delay is r Variable delay jitters

36. 36 Multimedia Networking: Two Types of Approaches Laissez-faire philosophy r No reservations, no change of the services provided by IP r Several tools m As demand increases, provision more bandwidth m Application adaptation m Place stored content at edge of network Integrated/Differentiated services philosophy: r Extends the service provided by the Internet r Two approaches m IntServ New protocols and router mechanisms so that each flow can reserve end-to- end bandwidth m DiffServ Datagrams are marked User pays more to send/receive 1st class packets

37. 37 Outline r Admin. and review r A brief introduction to the physical layer  Introduction to multimedia networking r An architecture of stored multimedia r Network support of multimedia app. r Application adaptation in best-effort Internet  dealing with variable delay m dealing with packet loss m dealing with variable bandwidth

38. 38 Delay Distribution In general, the distribution of delay is Bell-Shaped [MKT98]

39. 39 Playout Buffer r Basic idea: delay playing out packets to accommodate delay jitter r Discussion: For what types of applications will playout buffer be the most effective? packets number time packets generated p p' delay 0 1 r packets received

40. 40 Example: Adaptive Playout Delay Estimation  TCP-like delay and timeout estimation; adjust playout delay at the beginning of each check point

41. 41 Outline r Admin. and review r A brief introduction to the physical layer  Introduction to multimedia networking r An architecture of stored multimedia r Application adaptation in best-effort Internet  dealing with variable delay  dealing with packet loss m dealing with variable bandwidth

42. 42 Forward Error Correction (FEC): Piggyback A Lower Quality Stream on Next Packet r Send a lower resolution audio stream as the redundant information on the next packet, e.g. nominal stream at 64 kbps and redundant stream GSM at 13 kbps 2 3

43. 43 Outline r Admin. and review r A brief introduction to the physical layer  Introduction to multimedia networking r An architecture of stored multimedia r Application adaptation in best-effort Internet  dealing with variable delay m dealing with packet loss  dealing with variable bandwidth

44. 44 Basic Idea: Change the Rate of an MM Application r Video and audio m can generate different encoding rates by controlling the encoding process r Question: how do you change the rate of a multimedia stream? (262 KB)

45. 45 Video Encoding: Block Transform Encoding DCT Zig-zag Quantize Run-length Code Huffman Code 011010001011101...

46. 46 Discrete Cosine Transform F[u,v] = 4C(u)C(v) n2n2  n-1 j=0 k=0 f(j,k) cos (2j+1)u  2n (2k+1)v  2n cos where C(w) = 1  2 1 for w=0 for w=1,2,…,n-1 - Convert from pixel domain to frequency domain - DCT better at reducing redundancy than Discrete Fourier Transform but it is more computationally expensive

47. 47 Basis Functions of DCT

48. 48 Example: Block Encoding original image DCT DC component AC components Quantize zigzag run-length code Huffman code 10011011100011... coded bitstream < 10 bits (0.55 bits/pixel)

49. 49 Result of Coding/Decoding original block reconstructed block errors

50. 50 Examples Uncompressed (262 KB) Compressed (22 KB, 12:1) Compressed (6 KB, 43:1)

51. 51 Encode Rate According to Available Bandwidth r Steps m Estimate available bandwidth (how?) m Encode the stream according to the estimation of available bandwidth (how?)

52. 52 Backup Slides

53. 53 Layering r Put the stream into multiple layers r Each layer is a refinement of the previous layers r Example: m Zigzag of the DCT matrix, put the coefficients into different layers

54. 54 Alternative: Buffering Plus Layering

55. 55 Bandwidth and Buffer Co-allocation