1. 1 05MSE E13- Multi Media Computing By U.RAHAMATHUNNISA ASSISTANT PROFESSOR SITE

2. 2 Introduction Multimedia Description Why multimedia systems? Classification of Media Multimedia Systems Data Stream Characteristics

3. CIS 1043 Multimedia Description Multimedia: is an integration of continuous media (e.g. audio, video) and discrete media (e.g. text, graphics, images) through which digital information can be conveyed to the user in an appropriate way. Multimedia technology is the computer-based integration of text, sound, still images, animation and digitized motion video. Merges capabilities of computers with televisions, VCRs, CD players, DVD players, video and audio recording equipment, music and gaming technologies.

4. CIS 1044 Why Multimedia Computing? Application driven  e.g. medicine, sports, entertainment, education Information can often be better represented using audio/video/animation rather than using text, images and graphics alone. Information is distributed using computer and telecommunication networks. Integration of multiple media places demands on  computation power storage requirements networking requirements

5. CIS 1045 Multimedia Information Systems Technical challenges  Sheer volume of data Need to manage huge volumes of data  Timing requirements among components of data computation and communication. Must work internally with given timing constraints - real-time performance is required.  Integration requirements need to process traditional media (text, images) as well as continuous media (audio/video). Media are not always independent of each other - synchronization among the media may be required.

6. CIS 1046 High Data Volume of Multimedia Information

7. CIS 1047 Technology Incentive Growth in computational capacity MM workstations with audio/video processing capability Dramatic increase in CPU processing power Dedicated compression engines for audio, video etc. Rise in storage capacity Large capacity disks (several gigabytes) Increase in storage bandwidth,e.g. disk array technology Surge in available network bandwidth high speed fiber optic networks - gigabit networks fast packet switching technology

8. CIS 1048 Application Areas Residential Services video-on-demand video phone/conferencing systems multimedia home shopping (MM catalogs, product demos and presentation) self-paced education Business Services Corporate training Desktop MM conferencing, MM e-mail

9. CIS 1049 Application Areas Education Distance education - MM repository of class videos Access to digital MM libraries over high speed networks Science and Technology computational visualization and prototyping astronomy, environmental science Medicine Diagnosis and treatment - e.g. MM databases that provide support for queries on scanned images, X-rays, assessments, response etc.

10. CIS 10410 Classification of Media  Perception Medium How do humans perceive information in a computer?  Through seeing - text, images, video  Through hearing - music, noise, speech  Representation Medium How is the computer information encoded?  Using formats for representing and information  ASCII(text), JPEG(image), MPEG(video)  Presentation Medium Through which medium is information delivered by the computer or introduced into the computer?  Via I/O tools and devices  paper, screen, speakers (output media)  keyboard, mouse, camera, microphone (input media)

11. CIS 10411 Classification of Media (cont.)  Storage Medium  Where will the information be stored?  Storage media - floppy disk, hard disk, tape, CD- ROM etc.  Transmission Medium  Over what medium will the information be transmitted?  Using information carriers that enable continuous data transmission - networks  wire, coaxial cable, fiber optics  Information Exchange Medium  Which information carrier will be used for information exchange between different places?  Direct transmission using computer networks  Combined use of storage and transmission media (e.g. electronic mail).

12. CIS 10412 Media Concepts Each medium defines Representation values - determine the information representation of different media  Continuous representation values (e.g. electro- magnetic waves)  Discrete representation values(e.g. text characters in digital form) Representation space determines the surrounding where the media are presented.  Visual representation space (e.g. paper, screen)  Acoustic representation space (e.g. stereo)

13. CIS 10413 Media Concepts (cont.) Representation dimensions of a representation space are:  Spatial dimensions: two dimensional (2D graphics) three dimensional (holography)  Temporal dimensions: Time independent (document) - Discrete media  Information consists of a sequence of individual elements without a time component. Time dependent (movie) - Continuous media  Information is expressed not only by its individual value but also by its time of occurrence.

14. CIS 10414 Multimedia Systems Qualitative and quantitative evaluation of multimedia systems  Combination of media continuous and discrete.  Levels of media-independence some media types (audio/video) may be tightly coupled, others may not.  Computer supported integration timing, spatial and semantic synchronization  Communication capability

15. CIS 10415 Data Streams Distributed multimedia communication systems data of discrete and continuous media are broken into individual units (packets) and transmitted. Data Stream sequence of individual packets that are transmitted in a time-dependant fashion. Transmission of information carrying different media leads to data streams with varying features  Asynchronous  Synchronous  Isochronous

16. CIS 10416 Data Stream Characteristics Asynchronous transmission mode  provides for communication with no time restriction  Packets reach receiver as quickly as possible, e.g. protocols for email transmission Synchronous transmission mode  defines a maximum end-to-end delay for each packet of a data stream.  May require intermediate storage  E.g. audio connection established over a network. Isochronous transmission mode  defines a maximum and a minimum end-to-end delay for each packet of a data stream. Delay jitter of individual packets is bounded.  E.g. transmission of video over a network.  Intermediate storage requirements reduced.

17. CIS 10417 Data Stream Characteristics  Data Stream characteristics for continuous media can be based on Time intervals between complete transmission of consecutive packets  Strongly periodic data streams - constant time interval  Weakly periodic data streams - periodic function with finite period.  Aperiodic data streams Data size - amount of consecutive packets  Strongly regular data streams - constant amount of data  Weakly regular data streams - varies periodically with time  Irregular data streams Continuity  Continuous data streams  Discrete data streams

18. CIS 10418 Classification based on time intervals Strongly periodic data stream Weakly periodic data stream Aperiodic data stream T T T1T1 T3T3 T2T2 T1T1 T2T2 T

19. CIS 10419 Classification based on packet size T D1 T D2 D3 D1 D2 D3 D1 D2 D3 Dn Strongly regular data stream Weakly regular data stream Irregular data stream t t t

20. CIS 10420 Classification based on continuity Continuous data stream Discrete data stream D D1D2D3D4 D D1D2D3D4

21. CIS 10421 Logical Data Units  Continuous media consist of a time- dependent sequence of individual information units called Logical Data Units (LDU).  a symphony consists of independent sentences  a sentence consists of notes  notes are sequences of samples  Granularity of LDUs  symphony, sentence, individual notes, grouped samples, individual samples  film, clip, frame, raster, pixel  Duration of LDU:  open LDU - duration not known in advance  closed LDU - predefined duration

22. CIS 10422 Granularity of Logical Data Units Film Clip Frame Blocks Pixels

23. Multimedia from a user‘s point of view Presentation of Information also using Audio and Animation Information is easier conveyed Integration of Media with the Computer allows for  Interactive Presentations  Transmission across computer networks 23CIS 104

24. Multimedia and the Industry Sectors affected by the developments in Multimedia Telecommunications Consumer Electronics TV- and Radiostations Publishing Houses => Convergence of Sectors 24CIS 104

25. Properties of multimedia processing High data troughput Realtime Synchronisation 25CIS 104

26. Global Structure/Areas in Multimedia-Systems DatabasesOperating SystemsCommunication System Computer Technology MemoryNetworks Compression AudioVideo 26CIS 104

27. Areas in Multimedia-Systems Abstractions Documents Applications User Interface 27CIS 104

28. Terminology Multi [lat. much], as a Prefix Medium, mediating element, means to reproduce Information 28CIS 104

29. The term „medium“ Means for the distribution and presentation of information  Examples: Text, Figure, Image, Voice, Music Further differentiation of the term in the MHEG- Standard (1993)  On the following two slides 29CIS 104

30. Different media terms Perception-Medium  To See, to Hear (how?) Representation-Medium  JPEG, MPEG (Structure, Coding) Presentation-Medium  Screen, Speakers, Keyboard (Input/Output) 30CIS 104

31. Different media terms Storage-Medium  Paper, Harddisk, CD-ROM (where from?) Transmission-Medium  Network (on what?) Information-Exchange-Medium  Transmitter / Storage of Information: CD, Network (higher abstraction) 31CIS 104

32. Presentation values and domains Presentation domains  Screen, Paper (visual presentation domains)  Stereophonetics Presentation values  Character, shock wave 32CIS 104

33. Presentation dimension Every presentation domain has presentation dimensions  Screen: 2 spatial dimensions In Addition, Time as a dimension possible  Time as additional dimension fundamental for multimedia systems 33CIS 104

34. Classes of media Classification using the time dimension  Discrete Media Independent from time Examples: Text, Figures  Continuous Media Dependent on time Examples: Sound, Motion Picture 34CIS 104

35. The Term „Multimedia“ Qualitative rather than quantitative understanding What kind of media instead of multiple media  A Combination of text and picture is not sufficient for the term „Multimedia“ „Controversial term“ 35CIS 104

36. Main properties of a multimedia system Combination of Media  Processing of discrete and continuous media Independency  in the layout of the media in space and time Computerized Integration  Computer as a tool (Systems able to communicate)  Distributed Multimedia-Systems 36CIS 104

37. Definition of „Multimedia“ A multimedia system is defined through the computerized, integrated creation, manipulation, presentation, storage and communication of independent information coded in at least one continuous and one discrete medium. 37CIS 104

38. Usage of the term „Multimedia“ In a narrow sense  Continuous Media  Borders in Time In a broader sense  Shared processing of multiple media 38CIS 104

39. Transmission of media data Transmission of data of discrete and continuous media For information exchange Based on data streams 39CIS 104

40. Data stream Units (Packets) Send from source to sink Data stream is a sequence of single packets For discrete and continuous media 40CIS 104

41. Asynchronous Transmission No restrictions in time for the packets of the data stream  Example: IP Normally not suitable for multimedia 41CIS 104

42. Synchronous Transmission Defined Maximum End-To-End Latency For multimedia this means a guarantee on when packets will be arriving 42CIS 104

43. Isochronous Transmission Defined maximum and minimum end-to-end latency Jitter of single packets is restricted Buffers can be calculated more exact Suitable for „Streaming“, transmission of very large amounts of data 43CIS 104

44. Data streams of continuous media Different Properties  Time interval between packets  Amount of data / size of packets  Correlation of packets Possibly because of compression Example: Audio- and video transmission 44CIS 104

45. Strongly periodic data stream Time interval between neighbouring packets is constant Example: Voice in a telephone operation system T 45CIS 104

46. Weakly periodic data stream Time interval between packets follows a function of finite period duration Example: Animation T T1T1 T2T2 T3T3 T1T1 T2T2 46CIS 104

47. Aperiodic data stream Neither strongly nor weakly periodic Example: Transmission of events (e.g. Mouse interactions) T1T1 T2T2 TnTn 47CIS 104

48. Strongly regular Constant amount of data of all packets D1D1 DnDn t 48CIS 104

49. Weakly regular Amount of data of packets varies periodicly D1D1 t D2D2 D3D3 D1D1 D2D2 D3D3 T 49CIS 104

50. Not regular Amount of data is neither constant nor periodic D1D1 t D2D2 D3D3 DnDn 50CIS 104

51. Correlation of successive packets Coherent data stream  Packets are transmitted one after another without gaps Incoherent data stream – Gaps between the data packets t t 51CIS 104

52. Units of information Medium consists of a sequence of single units of information PDU: protocol data unit LDU: logical data unit Example: Logical units of a movie sequence  Movie, Clip, Frame, Raster, Pixel 52CIS 104

53. Characteristics of multimedia applications Large quantities of continuous data Timely and smooth delivery is critical  deadlines  throughput and response time guarantees Interactive MM applications require low round-trip delays Need to co-exist with other applications  must not hog resources Reconfiguration is a common occurrence  varying resource requirements Resources required:  Processor cycles in workstations  and servers  Network bandwidth (+ latency)  Dedicated memory  Disk bandwidth (for stored media) At the right time and in the right quantities 53CIS 104

54. Application requirements Network phone and audio conferencing  relatively low bandwidth (~ 64 Kbits/sec), but delay times must be short ( < 250 ms round-trip) Video on demand services  High bandwidth (~ 10 Mbits/s), critical deadlines, latency not critical Simple video conference  Many high-bandwidth streams to each node (~1.5 Mbits/s each), high bandwidth, low latency ( < 100 ms round-trip), synchronised states. Music rehearsal and performance facility  high bandwidth (~1.4 Mbits/s), very low latency (< 100 ms round trip), highly synchronised media (sound and video < 50 ms). 54CIS 104

55. System support issues and requirements Scheduling and resource allocation in most current OS’s divides the resources equally amongst all comers (processes)  no limit on load   can’t guarantee throughput or response time MM and other time-critical applications require resource allocation and scheduling to meet deadlines  Quality of Service (QoS) management Admission control:controls demand QoS negotiation:enables applications to negotiate admission and reconfigurations Resource management: guarantees availability of resources for admitted applications  real-time processor and other resource scheduling 55CIS 104

56. Characteristics of typical multimedia streams Data rate (approximate) Sample or frame frequency size Telephone speech64 kbps8 bits8000/sec CD-quality sound1.4 Mbps16 bits44,000/sec Standard TV video (uncompressed) 120 Mbpsup to 640x 480 pixelsx 16 bits 24/sec Standard TV video (MPEG-1 compressed) 1.5 Mbpsvariable24/sec HDTV video (uncompressed) 1000–3000 Mbpsup to 1920x 1080 pixelsx 24 bits 24–60/sec HDTV video MPEG-2 compressed) 10–30 Mbpsvariable24–60/sec Figure 15.3 56CIS 104

57. QoS Parameters Bandwidth  rate of flow of multimedia data Latency  time required for the end-to-end transmission of a single data element Jitter variation in latency :– dL/dt Loss rate  the proportion of data elements that can be dropped or delivered late 57CIS 104

58. What is a Multimedia System? A system that involves:  Generation: production/authoring tools  Representation: compression and formats  Storage: file system design  Transmission: networking issues, QoS  Search and retrieval: database management  Delivery: service design, QoS of multimedia information 58