1. Multimedia SystemsCS-502 Fall 20071 Multimedia Systems CS-502 Operating Systems Fall 2007 (Slides include materials from Operating System Concepts, 7 th ed., by Silbershatz, Galvin, & Gagne and from Modern Operating Systems, 2 nd ed., by Tanenbaum)

2. Multimedia SystemsCS-502 Fall 20072 Outline What is multi-media? Requirements and challenges for audio and video in computer systems –CD devices and formats Systems for multimedia Compression and bandwidth If time:– –Processor and disk scheduling –Network streaming and management Silbershatz, Chapter 20 Tanenbaum, Chapter 7

3. Multimedia SystemsCS-502 Fall 20073 What do we mean by “multimedia” Audio and video within a computer system –CD’s & DVD’s –Computer hard drive –Interactive Live broadcast & web casts –Radio stations, Network TV, Webcams, Skype, … Video on demand –Pause, fast forward, reverse, etc. Interactive activities involving audio, video, images –Meetings and presentations with 2-way audio –Teleconferencing Handheld devices –iPod, MP-3 players; personal video; mobile phones, … …

4. Multimedia SystemsCS-502 Fall 20074 Multimedia Data and Delivery Stored in file system like ordinary data. Must be accessed with specific timing requirements. E.g., video must be displayed at 24-30 frames per second. System must guarantee timely delivery Continuous-media data –Data with specific rate requirements

5. Multimedia SystemsCS-502 Fall 20075 Multimedia Characteristics Multimedia files can be very large. Continuous media data may require very high data rates. Multimedia applications are usually sensitive to timing delays during playback. Note: human ear is more sensitive to jitter in audio than eye is to jitter in video!

6. Multimedia SystemsCS-502 Fall 20076 Requirements “Smooth” audio and video –Deterioration in quality  jerky playback Multiple concurrent streams –Video & multimedia servers –TiVo, etc. Wide range of network bandwidths Audio/video while PC is doing something else

7. Multimedia SystemsCS-502 Fall 20077 System and OS Challenges Bandwidths and Compression Jitter Processor Scheduling Disk Scheduling Network Streaming

8. Multimedia SystemsCS-502 Fall 20078 Some System Architectures Simple: –Data paths for audio/video that are separate from computational data paths Modern –Fast system bus, CPU, devices Video server –Disk farm and multiple streams

9. Multimedia SystemsCS-502 Fall 20079 audio stream CD-ROM drive System Organization (a decade ago) Separate data path for audio stream Or headphone jack and volume control on CD drive itself Main system bus and CPU were too busy/slow to handle real-time audio CPU Memory memory bus DeviceSound card

10. Multimedia SystemsCS-502 Fall 200710 video stream via ISA & bridge to graphics card audio stream via ISA bridge to sound card System Organization (typical Pentium PC today) ISA bridge IDE disk Main Memory CPU Level 2 cache Bridge Moni- tor Graphics card USB Key- board Mouse Ether- net SCSI Modem Sound card Printer PCI bus ISA bus AGP Port

11. Multimedia SystemsCS-502 Fall 200711 Digression CD and DVD devices and files

12. Multimedia SystemsCS-502 Fall 200712 Compact Discs (CDs) Not in Silbershatz –See Tanenbaum, pp. 306-310 Audio CD –Molded polycarbonate; easy to manufacture –120 mm diameter with 15 mm hole –One single spiral track Starts in center, spirals outward 22,188 revolutions, approx 5.6 kilometers long Random access –Constant linear velocity under read head Audio playback:– 120 cm/sec Variable speed motor:– 200 – 530 rpm ISO standard IS 10149, aka the Red Book –Audio CD’s

13. Multimedia SystemsCS-502 Fall 200713 CDs for data storage Problem for adapting to data usage –No bad block recovery capability! ISO standard for data: Yellow Book –Three levels of error-correcting schemes: – Symbol, Frame, Sector ~7200 bytes to record 2048 byte payload per sector –Mode 2: less error correction in exchange for more data rate Audio and video data –Sectors linearly numbered from center to edge Read speed –1x ~ 153,000 bytes/sec –40x ~ 5.9 megabytes/sec ISO standard for multi-media: Green Book –Interleaved audio, video, data in same sector

14. Multimedia SystemsCS-502 Fall 200714 CD-ROM File System ISO 9660 — High Sierra Write once  contiguous file allocation Variable length directories Variable length directory entries Points to first sector of file File size and metadata stored in directory entry Variable length names Linux, Windows support iso9660 file system Several extensions to standard for additional features E.g., adding files, replacing directories See Tanenbaum, pp. 310-313

15. Multimedia SystemsCS-502 Fall 200715 DVD devices Similar in concept, different in details More data, higher bandwidth –Track spacing: 0.834μvs.1.6μ –Bit spacing:0.4μvs.0.74μ –Capacity:4.7 Gbytesvs.650 Mbytes Details not provided here –Tanenbaum, pp. 313-315

16. Multimedia SystemsCS-502 Fall 200716 Compression An essential part of audio and video representation as files

17. Multimedia SystemsCS-502 Fall 200717 Why Compression? – CD-quality audio 22,050 Hz  44,100 samples/sec 16 bits per sample Two channels  176,400 bytes/sec  1.4 mbits/sec –Okay for a modern PC –Not okay for 56 kb/sec modem (speed) or iPod (space)! MP-3  0.14 mbits/sec (10:1) –Same audio quality! –Compression ratio varies with type of music

18. Multimedia SystemsCS-502 Fall 200718 Why Compression? – Video “Standard” TV frame = 640  480 pixels @ 25-30 frames/sec (fps)   9,216,000 pixels/sec = 27,648,000 bytes/sec Hollywood “standard” movie  133 minutes  approx. 210 gigabytes (standard resolution)! HDTV = 1280  720 pixels @ 30 fps  82,944,000 bytes/sec (and rising!) DVD holds ~ 4.7 gigabytes  average of ~ 620 kilobytes/sec! “Standard” movie of 133 minutes requires serious compression just to fit onto DVD

19. Multimedia SystemsCS-502 Fall 200719 Video Compression Requirements Compression ratio > 50:1 i.e., 210 gigabytes:4.7 gigabytes Visually indistinguishable from original Even when paused Fast, cheap decoder Slow encoder is okay VCR controls Pause, fast forward, reverse

20. Multimedia SystemsCS-502 Fall 200720 Video Compression Standards MPEG (Motion Picture Experts Group) –Based on JPEG (Joint Photographic Experts Group) –Multi-layer Layer 1 = system and timing information Layer 2 = video stream Layer 3 = audio and text streams Three standards –MPEG-1 – 352  240 frames; < 1.5 mb/sec ( < VHS quality) Layer 3 = MP3 Audio standard Typical uses:– video clips on internet; video for handhelds –MPEG-2 – standard TV & HDTV; 1.5-15 mb/sec DVD encoding –MPEG-4 – combined audio, video, interactive graphics 2D & 3D animations

21. Multimedia SystemsCS-502 Fall 200721 JPEG compression (single frame) 1.Convert RGB into YIQ Y = luminance (i.e., brightness) ~ black-white TV I, Q = chrominance (similar to saturation and hue) Reason: Human eye is more sensitive to luminance than to color (rods vs. cones) 2.Down-sample I, Q channels i.e., average over 2  2 pixels to reduce resolution lossy compression, but barely noticeable to eye 3.Partition each channel into 8  8 blocks 4800 Y blocks, 1200 each I & Q blocks (for 640  480)

22. Multimedia SystemsCS-502 Fall 200722 JPEG (continued)

23. Multimedia SystemsCS-502 Fall 200723 JPEG (continued)

24. Multimedia SystemsCS-502 Fall 200724 JPEG (continued) 4.Calculate Discrete Cosine Transform (DCT) of each 8  8 block What is a Discrete Cosine Transform? 5.Divide 8  8 block of DCT values by 8  8 quantization table Effectively throwing away higher frequencies 6.Linearize 8  8 block, run-length encode, and apply a Huffman code to reduce to a small fraction of original size (in bytes)

25. Multimedia SystemsCS-502 Fall 200725 JPEG (concluded) 7.Store or transmit 8  8 quantization table followed by list of compressed blocks Achieves 20:1 compression with good visual characteristics Higher compression ratios possible with visible degradation JPEG algorithm executed backwards to recover image Visually indistinguishable from original @ 20:1 JPEG algorithm is symmetric Same speed forwards and backwards

26. Multimedia SystemsCS-502 Fall 200726 MPEG JPEG-like encoding of each frame Takes advantage of temporal locality I.e., each frame usually shares similarities with previous frame  encode and transmit only differences Sometimes an object moves relative to background  find object in previous frame, calculate difference, apply motion vector

27. Multimedia SystemsCS-502 Fall 200727 Temporal Locality (example) Consecutive Video Frames

28. Multimedia SystemsCS-502 Fall 200728 MPEG organization Three types of frames –I-frame: Intracoded or Independent. Full JPEG-encoded frame Occurs at intervals of a second or so Also at start of every scene –P-frame: Predictive frame Difference from previous frame –B-frame: Bidirectional frame Like p-frame but difference from both previous and next frame IBBBP BBBPBBBPBBBIBBBPBBBP

29. Multimedia SystemsCS-502 Fall 200729 MPEG Characteristics Non-uniform data rate! Compression ratios of 50:1 – 80:1 are readily obtainable Asymmetric algorithm –Fast decode (like JPEG) –Encode requires image search and analysis to get high quality differences Cheap decoding chips available for –graphics cards –DVD players, etc.

30. Multimedia SystemsCS-502 Fall 200730 MPEG Problem – Fast Forward/Reverse Cannot simply skip frames –Next desired frame might be B or P derived from a skipped frame Options: –Separate fast forward and fast reverse files MPEG encoding of every nth frame Often used in video-on-demand server –Display only I and P frames If B frame is needed, derive from nearest I or P

31. Multimedia SystemsCS-502 Fall 200731 “Movie” File Organization One MPEG-2 video stream Possible fast forward, fast reverse sub-streams Multiple audio streams Multiple languages Multiple text streams Subtitles in multiple languages All interleaved Possibly in multiple files!

32. Multimedia SystemsCS-502 Fall 200732

33. Multimedia SystemsCS-502 Fall 200733 Questions? (skip to reading assignment)

34. Multimedia SystemsCS-502 Fall 200734 Operating System Challenge How to get the contents of a movie file from disk or DVD drive to video screen and speakers. –Fixed frame rate (25 or 30 fps) –Steady audio rate –Bounded jitter Such requirements are known as Quality-of- Service (QoS) guarantees. Classical problem in real-time scheduling –Obscure niche become mainstream! –See Silbershatz, §19.1–19.5

35. Multimedia SystemsCS-502 Fall 200735 QoS Guarantees Building a system to guarantee QoS effects the following:– –CPU processing –Scheduling and interrupt handling –File systems –Network protocols

36. Multimedia SystemsCS-502 Fall 200736 Requirement of Multimedia Operating Systems There are three levels of QoS –Best-effort service: the system makes a best effort with no attempt to guarantee –Soft QoS: allows different traffic streams to be prioritized, but no QoS guarantees are made –Hard QoS: system ensures QoS requirements are always met Prioritization Admission control Bounded latency on interrupt handling, processing, etc.

37. Multimedia SystemsCS-502 Fall 200737 Parameters Defining QoS Throughput: the total amount of work completed during a specific time interval Delay: the elapsed time from when request is first submitted to desired result Jitter: delays that occur during playback of a stream. Reliability: how errors are handled during transmission and processing of continuous media

38. Multimedia SystemsCS-502 Fall 200738 Further QoS Issues QoS may be negotiated between the client and server. Operating systems may use an admission control algorithm –Admits a request for service only if the server has sufficient resources to satisfy the request

39. Multimedia SystemsCS-502 Fall 200739 Two common methods Rate Monotonic Scheduling Earliest Deadline First Many variations Many analytic methods for proving QoS (Quality of Service)

40. Multimedia SystemsCS-502 Fall 200740 Processor Scheduling for Real-Time Rate Monotonic Scheduling (RMS) Assume m periodic processes –Process i requires t i msec of processing time every p i msec. –Equal processing every interval — like clockwork!

41. Multimedia SystemsCS-502 Fall 200741 Example Periodic process i requires the CPU at specified intervals (periods) p i is the duration of the period t i is the processing time d i is the deadline by when the process must be serviced –Often same as end of period

42. Multimedia SystemsCS-502 Fall 200742 Processor Scheduling for Real-Time Rate Monotonic Scheduling (RMS) Assume m periodic processes –Process i requires t i msec of processing time every p i msec. –Equal processing every interval — like clockwork! Assume Assign priority of process i to be –Statically assigned Let priority of non-real-time processes be 0

43. Multimedia SystemsCS-502 Fall 200743 Rate Monotonic Scheduling (continued) Scheduler simply runs highest priority process that is ready –May pre-empt other real-time processes –Real-time processes become ready in time for each frame or sound interval –Non-real-time processes run only when no real- time process needs CPU

44. Multimedia SystemsCS-502 Fall 200744 Example p 1 = 50 msec; t 1 = 20 msec p 2 = 100 msec; t 2 = 35 msec Priority(p 1 ) > Priority(p 2 ) Total compute load is 75 msec per every 100 msec. Both tasks complete within every period 25 msec per 100 msec to spare

45. Multimedia SystemsCS-502 Fall 200745 Example 2 p 1 = 50 msec; t 1 = 25 msec p 2 = 80 msec; t 2 = 35 msec Priority(p 1 ) > Priority(p 2 ) Total compute load is ~ 94% of CPU. Cannot complete both tasks within some periods Even though there is still CPU capacity to spare!

46. Multimedia SystemsCS-502 Fall 200746 Rate Monotonic Theorems (without proof) Theorem 1: using these priorities, scheduler can guarantee the needed Quality of Service (QoS), provided that –Asymptotically approaches ln 2 as m   ln 2 = 0.6931… Theorem 2: If a set of processes can be scheduled by any method of static priorities, then it can be scheduled by Rate Monotonic method.

47. Multimedia SystemsCS-502 Fall 200747 Example 2 again Note that p 1 pre-empts p 2 in second interval, even though p 2 has the earlier deadline!

48. Multimedia SystemsCS-502 Fall 200748 More on Rate Monotonic Scheduling Rate Monotonic assumes periodic processes MPEG-2 playback is not a periodic process!

49. Multimedia SystemsCS-502 Fall 200749 Processor Scheduling for Real-Time Earliest Deadline First (EDF) When each process i become ready, it announces deadline D i for its next task. Scheduler always assigns processor to process with earliest deadline. May pre-empt other real-time processes

50. Multimedia SystemsCS-502 Fall 200750 Earliest Deadline First Scheduling (continued) No assumption of periodicity No assumption of uniform processing times Theorem: If any scheduling policy can satisfy QoS requirement for a sequence of real time tasks, then EDF can also satisfy it. –Proof: If i scheduled before i+1, but D i+1 < D i, then i and i+1 can be interchanged without affecting QoS guarantee to either one.

51. Multimedia SystemsCS-502 Fall 200751 Earliest Deadline First Scheduling (continued) EDF is more complex scheduling algorithm Priorities are dynamically calculated Processes must know deadlines for tasks EDF can make higher use of processor than RMS Up to 100% There is a large body of knowledge and theorems about EDF analysis

52. Multimedia SystemsCS-502 Fall 200752 Example 2 (again) Priorities are assigned according to deadlines: –the earlier the deadline, the higher the priority; –the later the deadline, the lower the priority.

53. Multimedia SystemsCS-502 Fall 200753 Reading Assignments Silbershatz, §19.1–19.5 –Real-time systems & Real-time scheduling Silbershat, Chapter 20 –Multimedia systems

54. Multimedia SystemsCS-502 Fall 200754 Questions?

55. Multimedia SystemsCS-502 Fall 200755 Network Management General methods for delivering content from a server to a client across a network:– –Broadcasting: server delivers content to all clients, whether they want it or not. –Multicasting: server delivers content to group of clients who indicate they wish to receive the content. –Unicasting: server delivers content to a single client.

56. Multimedia SystemsCS-502 Fall 200756 Network Management (continued) Broadcasting Like cable TV Fixed portion of network bandwidth dedicated to set of broadcast streams; inflexible Multicasting Typical webcast Multicast tree set up through internet to reach customers Customers can come and go Unicast Dedicated connection between server and client Streaming version of familiar transport protocols

57. Multimedia SystemsCS-502 Fall 200757 Streaming – Delivery of Multimedia Data over Network Two types of streaming:– –Progressive download: client begins playback of multimedia file during delivery. File is ultimately stored on client computer (Hopefully) download speed > playback speed –Real-time streaming: multimedia file is delivered to, but not stored on, client computer Played back at same speed as delivery Limited amount of buffering to remove jitter

58. Multimedia SystemsCS-502 Fall 200758 Two Types of Real-time Streaming Live streaming: to deliver a live event while it is occurring. Broadcast or multicast On-demand streaming: to deliver archived media streams Movies, lectures, old TV shows, etc. Events not delivered when they occur. Playback with pause, fast forward, reverse, etc Unicast (usually)

59. Multimedia SystemsCS-502 Fall 200759 Network Streaming Traditional HTTP Stateless Server responds to each request independently Real-Time Streaming Protocol (RTSP) Client initiates a “push” request for stream Server provides media stream at frame rate

60. Multimedia SystemsCS-502 Fall 200760 Pull (HTTP) vs. Push (RTSP) server

61. Multimedia SystemsCS-502 Fall 200761 RTSP States SETUP: server allocates resources for client session. PLAY: server delivers stream to a client session. PAUSE: server suspends delivery of a stream. TEARDOWN: server releases resources and breaks down connection.

62. Multimedia SystemsCS-502 Fall 200762 RTSP state machine

63. Multimedia SystemsCS-502 Fall 200763 Bandwidth Negotiation Client application provides feedback to server to adjust bandwidth E.g., –Windows Media Player –RealPlayer –Quicktime

64. Multimedia SystemsCS-502 Fall 200764 Video Server Multiple CPUs Disk farm –1000s of disks Multiple high-bandwidth network links –Cable TV –Video on demand –Internet

65. Multimedia SystemsCS-502 Fall 200765 Multimedia File & Disk Management Single movie or multimedia file on disk –Interleave audio, video, etc. So temporally equivalent blocks are near each other –Attempt contiguous allocation Avoid seeks within a frame Text Frame Audio Frame

66. Multimedia SystemsCS-502 Fall 200766 File organization – Frame vs. Block Frame organization Small disk blocks (4-16 Kbytes) Frame index entries point to starting block for each frame Frames vary in size (MPEG) Advantage: very little storage fragmentation Disadvantage: large frame table in RAM Block organization Large disk block (256 Kbytes or more) Block index entries point to first I-frame of a sequence Multiple frames per block Advantage: much smaller block table in RAM Disadvantage: large storage fragmentation on disk

67. Multimedia SystemsCS-502 Fall 200767 Frame vs. Block organization smaller larger

68. Multimedia SystemsCS-502 Fall 200768 File Placement on Server Random Striped “Organ pipe” allocation –Most popular video in center of disk –Next most popular on either side of it, etc. –Least popular at edges of disk –Minimizes seek distance

69. Multimedia SystemsCS-502 Fall 200769 Resources on a file server

70. Multimedia SystemsCS-502 Fall 200770 Conclusion Multimedia computing is challenging Possible with modern computers Compression is essential, especially for video Real-time computing techniques move into mainstream Processor and disk scheduling There is much more to this subject than fits into one class

71. Multimedia SystemsCS-502 Fall 200771 Reading Assignments Silbershatz, §19.1–19.5 –Real-time systems & Real-time scheduling Silbershatz, Chapter 20 –Multimedia systems

72. Multimedia SystemsCS-502 Fall 200772

73. Multimedia SystemsCS-502 Fall 200773 Digression on Transforms Fourier’s Theorem Every continuous, periodic function can be reduced to the sum of a series of sine waves The Fourier transform is a representation of that function in terms of the frequencies of those sine waves Original function can be recovered from its Fourier transform Fourier transforms occur frequently in nature!

74. Multimedia SystemsCS-502 Fall 200774 Nyquist’s Theorem (1924) If a continuous function is sampled at a frequency 2f, then the function can be recovered from those samples provided that its maximum Fourier frequency is  f.

75. Multimedia SystemsCS-502 Fall 200775 Discrete Cosine Transform A form of the Fourier Transform When applied to an 8  8 block of samples (i.e. pixel values) yields an 8  8 block of spatial frequencies Original 8  8 block of samples can be recovered from its DCT. –Assuming infinite arithmetic precision

76. Multimedia SystemsCS-502 Fall 200776 More on Nyquist If arithmetic precision is not infinite, we get quantization error during sampling Recovered signal has quantization noise i.e., a lossy transform