1 05MSE E13- Multi Media Computing By U.RAHAMATHUNNISA ASSISTANT PROFESSOR SITE

2 Introduction Multimedia Description Why multimedia systems? Classification of Media Multimedia Systems Data Stream Characteristics

CIS 1043 Multimedia Description Multimedia: is an integration of continuous media (e.g. audio, video) and discrete media (e.g. text, graphics, images) through which digital information can be conveyed to the user in an appropriate way. Multimedia technology is the computer-based integration of text, sound, still images, animation and digitized motion video. Merges capabilities of computers with televisions, VCRs, CD players, DVD players, video and audio recording equipment, music and gaming technologies.

CIS 1044 Why Multimedia Computing? Application driven  e.g. medicine, sports, entertainment, education Information can often be better represented using audio/video/animation rather than using text, images and graphics alone. Information is distributed using computer and telecommunication networks. Integration of multiple media places demands on  computation power storage requirements networking requirements

CIS 1045 Multimedia Information Systems Technical challenges  Sheer volume of data Need to manage huge volumes of data  Timing requirements among components of data computation and communication. Must work internally with given timing constraints - real-time performance is required.  Integration requirements need to process traditional media (text, images) as well as continuous media (audio/video). Media are not always independent of each other - synchronization among the media may be required.

CIS 1046 High Data Volume of Multimedia Information

CIS 1047 Technology Incentive Growth in computational capacity MM workstations with audio/video processing capability Dramatic increase in CPU processing power Dedicated compression engines for audio, video etc. Rise in storage capacity Large capacity disks (several gigabytes) Increase in storage bandwidth,e.g. disk array technology Surge in available network bandwidth high speed fiber optic networks - gigabit networks fast packet switching technology

CIS 1048 Application Areas Residential Services video-on-demand video phone/conferencing systems multimedia home shopping (MM catalogs, product demos and presentation) self-paced education Business Services Corporate training Desktop MM conferencing, MM

CIS 1049 Application Areas Education Distance education - MM repository of class videos Access to digital MM libraries over high speed networks Science and Technology computational visualization and prototyping astronomy, environmental science Medicine Diagnosis and treatment - e.g. MM databases that provide support for queries on scanned images, X-rays, assessments, response etc.

CIS Classification of Media  Perception Medium How do humans perceive information in a computer?  Through seeing - text, images, video  Through hearing - music, noise, speech  Representation Medium How is the computer information encoded?  Using formats for representing and information  ASCII(text), JPEG(image), MPEG(video)  Presentation Medium Through which medium is information delivered by the computer or introduced into the computer?  Via I/O tools and devices  paper, screen, speakers (output media)  keyboard, mouse, camera, microphone (input media)

CIS Classification of Media (cont.)  Storage Medium  Where will the information be stored?  Storage media - floppy disk, hard disk, tape, CD- ROM etc.  Transmission Medium  Over what medium will the information be transmitted?  Using information carriers that enable continuous data transmission - networks  wire, coaxial cable, fiber optics  Information Exchange Medium  Which information carrier will be used for information exchange between different places?  Direct transmission using computer networks  Combined use of storage and transmission media (e.g. electronic mail).

CIS Media Concepts Each medium defines Representation values - determine the information representation of different media  Continuous representation values (e.g. electro- magnetic waves)  Discrete representation values(e.g. text characters in digital form) Representation space determines the surrounding where the media are presented.  Visual representation space (e.g. paper, screen)  Acoustic representation space (e.g. stereo)

CIS Media Concepts (cont.) Representation dimensions of a representation space are:  Spatial dimensions: two dimensional (2D graphics) three dimensional (holography)  Temporal dimensions: Time independent (document) - Discrete media  Information consists of a sequence of individual elements without a time component. Time dependent (movie) - Continuous media  Information is expressed not only by its individual value but also by its time of occurrence.

CIS Multimedia Systems Qualitative and quantitative evaluation of multimedia systems  Combination of media continuous and discrete.  Levels of media-independence some media types (audio/video) may be tightly coupled, others may not.  Computer supported integration timing, spatial and semantic synchronization  Communication capability

CIS Data Streams Distributed multimedia communication systems data of discrete and continuous media are broken into individual units (packets) and transmitted. Data Stream sequence of individual packets that are transmitted in a time-dependant fashion. Transmission of information carrying different media leads to data streams with varying features  Asynchronous  Synchronous  Isochronous

CIS Data Stream Characteristics Asynchronous transmission mode  provides for communication with no time restriction  Packets reach receiver as quickly as possible, e.g. protocols for transmission Synchronous transmission mode  defines a maximum end-to-end delay for each packet of a data stream.  May require intermediate storage  E.g. audio connection established over a network. Isochronous transmission mode  defines a maximum and a minimum end-to-end delay for each packet of a data stream. Delay jitter of individual packets is bounded.  E.g. transmission of video over a network.  Intermediate storage requirements reduced.

CIS Data Stream Characteristics  Data Stream characteristics for continuous media can be based on Time intervals between complete transmission of consecutive packets  Strongly periodic data streams - constant time interval  Weakly periodic data streams - periodic function with finite period.  Aperiodic data streams Data size - amount of consecutive packets  Strongly regular data streams - constant amount of data  Weakly regular data streams - varies periodically with time  Irregular data streams Continuity  Continuous data streams  Discrete data streams

CIS Classification based on time intervals Strongly periodic data stream Weakly periodic data stream Aperiodic data stream T T T1T1 T3T3 T2T2 T1T1 T2T2 T

CIS Classification based on packet size T D1 T D2 D3 D1 D2 D3 D1 D2 D3 Dn Strongly regular data stream Weakly regular data stream Irregular data stream t t t

CIS Classification based on continuity Continuous data stream Discrete data stream D D1D2D3D4 D D1D2D3D4

CIS Logical Data Units  Continuous media consist of a time- dependent sequence of individual information units called Logical Data Units (LDU).  a symphony consists of independent sentences  a sentence consists of notes  notes are sequences of samples  Granularity of LDUs  symphony, sentence, individual notes, grouped samples, individual samples  film, clip, frame, raster, pixel  Duration of LDU:  open LDU - duration not known in advance  closed LDU - predefined duration

CIS Granularity of Logical Data Units Film Clip Frame Blocks Pixels

Multimedia from a user‘s point of view Presentation of Information also using Audio and Animation Information is easier conveyed Integration of Media with the Computer allows for  Interactive Presentations  Transmission across computer networks 23CIS 104

Multimedia and the Industry Sectors affected by the developments in Multimedia Telecommunications Consumer Electronics TV- and Radiostations Publishing Houses => Convergence of Sectors 24CIS 104

Properties of multimedia processing High data troughput Realtime Synchronisation 25CIS 104

Global Structure/Areas in Multimedia-Systems DatabasesOperating SystemsCommunication System Computer Technology MemoryNetworks Compression AudioVideo 26CIS 104

Areas in Multimedia-Systems Abstractions Documents Applications User Interface 27CIS 104

Terminology Multi [lat. much], as a Prefix Medium, mediating element, means to reproduce Information 28CIS 104

The term „medium“ Means for the distribution and presentation of information  Examples: Text, Figure, Image, Voice, Music Further differentiation of the term in the MHEG- Standard (1993)  On the following two slides 29CIS 104

Different media terms Perception-Medium  To See, to Hear (how?) Representation-Medium  JPEG, MPEG (Structure, Coding) Presentation-Medium  Screen, Speakers, Keyboard (Input/Output) 30CIS 104

Different media terms Storage-Medium  Paper, Harddisk, CD-ROM (where from?) Transmission-Medium  Network (on what?) Information-Exchange-Medium  Transmitter / Storage of Information: CD, Network (higher abstraction) 31CIS 104

Presentation values and domains Presentation domains  Screen, Paper (visual presentation domains)  Stereophonetics Presentation values  Character, shock wave 32CIS 104

Presentation dimension Every presentation domain has presentation dimensions  Screen: 2 spatial dimensions In Addition, Time as a dimension possible  Time as additional dimension fundamental for multimedia systems 33CIS 104

Classes of media Classification using the time dimension  Discrete Media Independent from time Examples: Text, Figures  Continuous Media Dependent on time Examples: Sound, Motion Picture 34CIS 104

The Term „Multimedia“ Qualitative rather than quantitative understanding What kind of media instead of multiple media  A Combination of text and picture is not sufficient for the term „Multimedia“ „Controversial term“ 35CIS 104

Main properties of a multimedia system Combination of Media  Processing of discrete and continuous media Independency  in the layout of the media in space and time Computerized Integration  Computer as a tool (Systems able to communicate)  Distributed Multimedia-Systems 36CIS 104

Definition of „Multimedia“ A multimedia system is defined through the computerized, integrated creation, manipulation, presentation, storage and communication of independent information coded in at least one continuous and one discrete medium. 37CIS 104

Usage of the term „Multimedia“ In a narrow sense  Continuous Media  Borders in Time In a broader sense  Shared processing of multiple media 38CIS 104

Transmission of media data Transmission of data of discrete and continuous media For information exchange Based on data streams 39CIS 104

Data stream Units (Packets) Send from source to sink Data stream is a sequence of single packets For discrete and continuous media 40CIS 104

Asynchronous Transmission No restrictions in time for the packets of the data stream  Example: IP Normally not suitable for multimedia 41CIS 104

Synchronous Transmission Defined Maximum End-To-End Latency For multimedia this means a guarantee on when packets will be arriving 42CIS 104

Isochronous Transmission Defined maximum and minimum end-to-end latency Jitter of single packets is restricted Buffers can be calculated more exact Suitable for „Streaming“, transmission of very large amounts of data 43CIS 104

Data streams of continuous media Different Properties  Time interval between packets  Amount of data / size of packets  Correlation of packets Possibly because of compression Example: Audio- and video transmission 44CIS 104

Strongly periodic data stream Time interval between neighbouring packets is constant Example: Voice in a telephone operation system T 45CIS 104

Weakly periodic data stream Time interval between packets follows a function of finite period duration Example: Animation T T1T1 T2T2 T3T3 T1T1 T2T2 46CIS 104

Aperiodic data stream Neither strongly nor weakly periodic Example: Transmission of events (e.g. Mouse interactions) T1T1 T2T2 TnTn 47CIS 104

Strongly regular Constant amount of data of all packets D1D1 DnDn t 48CIS 104

Weakly regular Amount of data of packets varies periodicly D1D1 t D2D2 D3D3 D1D1 D2D2 D3D3 T 49CIS 104

Not regular Amount of data is neither constant nor periodic D1D1 t D2D2 D3D3 DnDn 50CIS 104

Correlation of successive packets Coherent data stream  Packets are transmitted one after another without gaps Incoherent data stream – Gaps between the data packets t t 51CIS 104

Units of information Medium consists of a sequence of single units of information PDU: protocol data unit LDU: logical data unit Example: Logical units of a movie sequence  Movie, Clip, Frame, Raster, Pixel 52CIS 104

Characteristics of multimedia applications Large quantities of continuous data Timely and smooth delivery is critical  deadlines  throughput and response time guarantees Interactive MM applications require low round-trip delays Need to co-exist with other applications  must not hog resources Reconfiguration is a common occurrence  varying resource requirements Resources required:  Processor cycles in workstations  and servers  Network bandwidth (+ latency)  Dedicated memory  Disk bandwidth (for stored media) At the right time and in the right quantities 53CIS 104

Application requirements Network phone and audio conferencing  relatively low bandwidth (~ 64 Kbits/sec), but delay times must be short ( < 250 ms round-trip) Video on demand services  High bandwidth (~ 10 Mbits/s), critical deadlines, latency not critical Simple video conference  Many high-bandwidth streams to each node (~1.5 Mbits/s each), high bandwidth, low latency ( < 100 ms round-trip), synchronised states. Music rehearsal and performance facility  high bandwidth (~1.4 Mbits/s), very low latency (< 100 ms round trip), highly synchronised media (sound and video < 50 ms). 54CIS 104

System support issues and requirements Scheduling and resource allocation in most current OS’s divides the resources equally amongst all comers (processes)  no limit on load   can’t guarantee throughput or response time MM and other time-critical applications require resource allocation and scheduling to meet deadlines  Quality of Service (QoS) management Admission control:controls demand QoS negotiation:enables applications to negotiate admission and reconfigurations Resource management: guarantees availability of resources for admitted applications  real-time processor and other resource scheduling 55CIS 104

Characteristics of typical multimedia streams Data rate (approximate) Sample or frame frequency size Telephone speech64 kbps8 bits8000/sec CD-quality sound1.4 Mbps16 bits44,000/sec Standard TV video (uncompressed) 120 Mbpsup to 640x 480 pixelsx 16 bits 24/sec Standard TV video (MPEG-1 compressed) 1.5 Mbpsvariable24/sec HDTV video (uncompressed) 1000–3000 Mbpsup to 1920x 1080 pixelsx 24 bits 24–60/sec HDTV video MPEG-2 compressed) 10–30 Mbpsvariable24–60/sec Figure CIS 104

QoS Parameters Bandwidth  rate of flow of multimedia data Latency  time required for the end-to-end transmission of a single data element Jitter variation in latency :– dL/dt Loss rate  the proportion of data elements that can be dropped or delivered late 57CIS 104

What is a Multimedia System? A system that involves:  Generation: production/authoring tools  Representation: compression and formats  Storage: file system design  Transmission: networking issues, QoS  Search and retrieval: database management  Delivery: service design, QoS of multimedia information 58