1. Multimedia Systems

2. Course Contents Introduction Chapter 1 Media and Data StreamsChapter 2 Optical Storage MediaChapter 8 Audio TechnologyChapter 3 Graphics and ImagesChapter 4 Video TechnologyChapter 5 Computer-based Animation Chapter 6 Data CompressionChapter 7 Content AnalysisChapter 9

3. Grading Policy Homeworks & Assignments15% Pop Quizzes and/or Class Work5% Two Major Tests10% Final Exam70% Absences with no reasonable justification -2.5%

4. Media and Data Streams

5. Multimedia: content that uses a combination of different content forms. This contrasts with media that use only rudimentary computer displays such as text-only or traditional forms of printed or hand-produced material. Multimedia includes a combination of text, audio, still images, animation, video, or interactivity content forms.

6. Multimedia is usually recorded and played, displayed, or accessed by information content processing devices, such as computerized and electronic devices, but can also be part of a live performance. Multimedia devices are electronic media devices used to store and experience multimedia content. Multimedia is distinguished from mixed media in fine art; by including audio, for example, it has a broader scope. The term "rich media" is synonymous for interactive multimedia. Hypermedia can be considered one particular multimedia application.

7. Media classification With computer processing: – Medium –a means of distribution and presentation of information: text, graphics, speech, music…… Media can be classified with different criteria. We can classify media according to: – Perception, – Representation, – Presentation, – Storage, – Transmission, – Information exchange

8. Perception medium Perception media help the humans to sense their environment; perception mostly occurs through seeing or hearing the information. For the perception through seeing the visual media such as text, image and video are used. For the perception of information through hearing, auditory media such as music, noise and speech are relevant.

9. The Representation Medium Representation media are characterized by internal computer representations of information. The central question is: * How is computer information coded? There are various formats used to represent media information in a computer, For example: Text is coded in ASCII code (America Standard Code for Information Interchange) or EBCDIC (Extended Binary Coded Decimal Interchange Code) is an 8-bit character encoding used on IBM mainframe. Graphics are coded according to CEPT or CAPTAIN videotext standard (Character And Pattern Telephone Access Information Network, or the graphics standard GKS(Graphic Kernel System).

10. Audio stream can be represented using a simple PCM (Pulse Code Modulation) method. Image can be coded as a fax or in JPEG format. Video sequences can be coded in different TV standard formats (e.g. PAL, SECAM, NTSC) and can be stored in MPEG format (Motion Picture Experts Group) standard for video and associated audio

11. The Presentation Medium Presentation media refer to tools and devices for the input and output of information. The central question is: Through which medium is information delivered by the computer, or introduced into the computer? The media to deliver information by computer (output media) includes paper, screen, speaker, whereas the input media are keyboard, mouse, camera, and microphone.

12. The Presentation Medium The Storage Medium Storage media refer to a data carrier which enables storage of information. The central question is: Where will the information be stored? Examples of storage media are Floppy disk, Hard disk, CD-ROM, Magnetic Tapes, Removable Disk, Microfilm and even paper.

13. The Transmission Medium The transmission medium characterizes different information carriers that enable continuous data transmission. The central question is: Over what will the information be transmitted? The answer is that information is transmitted over networks, which use wire and cable transmission such as coaxial and fiber optics, as well as free air space transmission for wireless transmissions( Radio tower,satellite.

14. The Transmission Medium The Information Exchange Medium The information exchange medium includes all information carriers for transmissions, i.e. all storage and transmission media. The central question is: Which information carrier will be used for information exchange between different places? Information can flow through intermediate storage media, where storage media is transported outside through computer networks to the destination, through direct transmission using computer networks, or through combined usage of storage and transmission media ( e.g. e-mail).

15. 4- Representation of Media The media is characterized as information to be processed in computer systems. Each medium defines the following: 1- Representation Values 2- Representation Space The user will perceive the content through these representations

16. 1- Representation Values Representation values are the actual information representation of the media. A sentence can be represented by text, which is a sequence of characters, or by speech, which is in a waveform. Representation values can be considered either as a continuum or a sequence of discrete values. 2- Representation Space Paper or computer monitors are examples of a visual presentation spaces. Each representation space consists of one or more representation dimensions. A computer screen has 2 spatial dimensions; holography and stereophony require an additional spatial dimension. Time can occur in each representation space as an additional dimension, as it has central meaning to multimedia systems.

17. Representation Dimensions Media can be divided into two types with respect to time in their representation space: Time-independent (discrete) Some media such as text, pictures and graphics, are time-independent. Information in these media consist exclusively of a sequence of individual elements or of a continuum without a time component.(indeed, they may be displayed according to a wide variety of timing and still remain meaningful).

18. Representation Dimensions Time-dependent (continuous) Other media, such as sound and motion video(means pure moving images), change over time. Information is expressed not only on the individual value but also the time of its occurrence (requires a continuous playout as a time passes). The meaning depend on the level of the relative change of the discrete values or of the continuum. Representation caused by tactile or temperature sensors with threshold detectors are also time-dependent and falls under this category. Processing these media is time critical because the validity of the data depends on a time condition.

19. As a conclusion the Classification of Media as follows: – Perception Medium How do humans perceive information in a computer? – Through seeing - text, images, video – Through hearing - music, noise, speech – Representation Medium How is the computer information encoded? – Using formats for representing and information – ASCII(text), JPEG(image), MPEG(video) – Presentation Medium Through which medium is information delivered by the computer or introduced into the computer? – Via I/O tools and devices – paper, screen, speakers (output media) – keyboard, mouse, camera, microphone (input media)

20. Classification of Media (cont.) – Storage Medium – Where will the information be stored? – Storage media - floppy disk, hard disk, tape, CD-ROM etc. – Transmission Medium – Over what medium will the information be transmitted? – Using information carriers that enable continuous data transmission - networks – wire, coaxial cable, fiber optics – Information Exchange Medium – Which information carrier will be used for information exchange between different places? – Direct transmission using computer networks – Combined use of storage and transmission media (e.g. electronic mail).

21. Media Concepts Each medium defines Representation values - determine the information representation of different media – Continuous representation values (e.g. electro- magnetic waves) – Discrete representation values(e.g. text characters in digital form) Representation space determines the surrounding where the media are presented. – Visual representation space (e.g. paper, screen) – Acoustic representation space (e.g. stereo)

22. Representation dimensions of a representation space are: – Spatial dimensions: two dimensional (2D graphics) three dimensional (holography) – Temporal dimensions: Time independent (document) - Discrete media – Information consists of a sequence of individual elements without a time component. Time dependent (movie) - Continuous media – Information is expressed not only by its individual value but also by its time of occurrence.

23. Multimedia Systems Qualitative and quantitative evaluation of multimedia systems – Combination of media continuous and discrete. – Levels of media-independence some media types (audio/video) may be tightly coupled, others may not. – Computer supported integration timing, spatial and semantic synchronization – Communication capability

24. 5- Main Properties of Multimedia Systems Combination of Media Not just any arbitrary combination of media deserves the name multimedia. Many people call a simple word processor that handles embedded graphics a multimedia application because it uses two media. By our definition, we talk about multimedia if the application uses both discrete and continuous media. This means that a multimedia application should process at least one discrete and one continuous medium. A word processor with embedded graphics is not a multimedia application by our definition.

25. Independence An important aspect is that the media used in a multimedia system should be independent. Although a computer-controlled video recorder handles audio and moving image information, there is a temporal dependence between the audio part and the video part. In contrast, a system that combines signals recorded on a DAT (Digital Audio Tape) recorder with some text stored in a computer to create a presentation meets the independence criterion. Other examples are combined text and graphics blocks, which can be in an arbitrary space arrangement in relation to one another.

26. Computer-supported Integration Computer-Controlled Systems The independence of media creates a way to combine media in an arbitrary form for presentation. For this purpose, the computer is the ideal tool. That is, we need a system capable of processing media in a computer- controlled way. The system can be optionally programmed by a system programmer and/or by a user (within certain limits). The simple recording or playout of various media in a system, such as a video recorder, is not sufficient to meet the computer-control criterion..

27. Integration Computer-controlled independent media streams can be integrated to form a global system so that, together, they provide a certain function. To this end, synchronic relationships of time, space, and content are created between them. A word processor that supports text, spreadsheets, and graphics does not meet the integration criterion unless it allows program- supported references between the data. We achieve a high degree of integration only if the application is capable of, for example, updating graphics and text elements automatically as soon as the contents of the related spreadsheet cell changes.

28. 6- Definition of Multimedia Systems Based on the details above, the following definition can be used to describe multimedia systems: “A multimedia system is characterized by computer-controlled, integrated production,manipulation, presentation, storage and communication of independent information, which is encoded at least through a continuous (time-dependent) and a discrete (time independent)medium.” A few examples of multimedia systems are: Video on demand, Electronic book, Interactive cinema, Internet Bookshop, and Games. so Multimedia systems are computer-based systems that support the delivery of multimedia content.

29. 7- Multimedia-Related Technologies Multimedia systems is the integration of many components of different technologies. The development of multimedia technology is related to the evolution of the following technologies: Computer Technology With faster processing power, larger memory and storage capacities, the multimedia capabilities of the computer has increased significantly in the last decade. This has enabled the development of multimedia intensive applications in many areas. Telecommunications Digital networks with higher bandwidths increased the development of distributed multimedia applications, especially through the internet. Consumer Electronics Improvements in devices such as display units and optical storage contributed to new and better multimedia facilities. In addition, mass production reduced the prices significantly for consumers.

30. Entertainment and Broadcasting Professional audio and video production has significantly improved the technical capabilities and multimedia contents. These are adapted to multimedia systems for better quality products. Publishing Many large publishing houses already offer their publications in electronic form. More and more multimedia information can be offered.

31. 8-Key properties of a MM system: They must be computer control controlled thus, a computer must be involved at least in the presentation to the user They are integrated, that is they use.Minimum number of different devices example :single computer screen is used to display all type of visual information They must be support media independence They need to handle discrete and continuous media

32. 9-Application Areas Where to Use MM: Residential Services video-on-demand video phone/conferencing systems multimedia home shopping (MM catalogs, product demos and presentation) self-paced education Business Services Corporate training Desktop MM conferencing, MM e-mail

33. Education Distance education - MM repository of class videos Access to digital MM libraries over high speed networks Science and Technology computational visualization and prototyping astronomy, environmental science Medicine Diagnosis and treatment - e.g. MM databases that provide support for queries on scanned images, X-rays, assessments, response etc.

34. MM in Business: Presentation, Training, Marketing, Advertising, DB, Catalog, Network communication voice mail, Video conference will be soon provided on many local and wide area network. MM is School : connect every Class room,Library,Clinic and Hospital MM at Home : TV set or monitor with built in interactive user input or use computer with CD-ROM device, or set top players(Kodak photo CD player, Philips CD-1, or Panasonic 3DO Player MM in public place : Hotels, Train station, shopping malls,….to provide information and help.

35. 10 - MM Hardware & Software: Digital media device: Keyboard, Scanner, Display driver. Analog media device : We can divided into 3 parts : – Source produce analog signal Video, Camera, microphone – Sink consumer convert of analog signal : speaker, video display – Filter :convert form One analog to another audio mixer, video overlay device. General purpose devices Ex: 1- Sorting device CD-ROM, Magnatic disk … 2-network interface Interaction devices : Mouse, Joystick, electronics pens,Keyboard

36. Data streaming Is the transfer of data at a steady high-speed rate sufficient to support such applications as high- definition television (HDTV) or the continuous backup copying to a storage medium of the data flow within a computer. Data streaming requires some combination of bandwidth sufficiency and, for real-time human perception of the data, the ability to make sure that enough data is being continuously received without any noticeable time lag.

37. Characterizing Data Streams When we transmit information originating from various media, we obtain data streams that have very different characteristics. The attributes asynchronous, synchronous, and isochronous are traditionally used in the field of telecommunications to describe the characteristics of a data transmission.

38. Asynchronous Transmission The sender and the receiver do not need to coordinate before data can be transmitted. – Two independent clocks (one at the sender and the other at the receiver) determine the start of each bit (bit synchronization)

39. Synchronous Transmission The beginning of transmission may only take place at well-defined times, matching a clocking signal that runs the synchronism with that of the receiver. If audio from a digitized phone call is delayed, however, the human listening to the call will hear the delay as annoying interference or noise. Once a receiver starts to play digitized samples that arrive late, the receiver cannot speed up the playback to catch up with the rest of the stream.

40. Isochronous Transmission The term isochronous refers to a periodic signal, pertaining to transmission in which the time interval separating any two corresponding transitions is equal to the unit interval or to a multiple of the unit interval. The time interval separating any two corresponding transitions is equal to the unit interval or to a multiple of the unit interval. Secondly, it refers to data transmission in which corresponding significant instants of two or more sequential signals have a constant phase relationship.

41. Isochronous: cont… an end-to-end network connection is said to be isochronous if the bit rate over the connection is guaranteed and if the value of the delay jitter is also guaranteed and small. The notion of isochronism serves to describe what the performance of a network should be in order to satisfactorily transport continuous media streams, such as real-time audio or motion video. What is required to transport audio and video in real time? If the source transmits bits at a certain rate, the network should be able to meet that rate in a sustained way.

42. – Transmission rate “guaranteed” and jitter is minimized. Significant instants of two or more sequential signals have a constant phase relationship. Those three attributes are a simplified classification of different types of data streams.

43. Characterizing Continuous Data Streams This section provides a summary of the characteristics for data streams that occur in multimedia systems in relation to audio and video transmissions. The description includes effects of compression methods applied to the data streams before they are transmitted. This classification applies to distributed and local environments. This is in relation to audio and video transmission 2.5.1Strongly and Weakly Periodic Data Streams 2.5.2 Variation of the data volume of consecutive information units. 2.5.3 Interrelationship of consecutive packets.

44. Strongly Periodic Data Stream The first property of data streams relates to the time intervals between fully completed transmissions of consecutive information units or packets. Based on the moment in which the packets become ready, we distinguish between the following variants:

45. Strongly Periodic Data Stream When the time interval between neighboring packets is constant, then this data stream is called a strongly periodic data stream. This also means that there is minimal jitter-ideally zero. An example for this type is PCM-encoded (Pulse Code Modulation) voice in telephone systems. So: Constant time interval and Minimum jitter

46. PCM To obtain PCM from an analog waveform at the source (transmitter end) of a communications circuit, the analog signal amplitude is sampled (measured) at regular time intervals. The sampling rate, or number of samples per second, is several times the maximum frequency of the analog waveform in cycles per second or hertz.

47. Weakly Periodic Data Stream Duration of time intervals between neighboring packets is a function with finite period duration. Time intervals between neighboring packets is not constant.

48. Non-Periodic Data Streams Other possibilities – e.g. mouse movement on white boards of the screen in video conferences An example of an aperiodic data stream is a multimedia conference application with a common screen window. Often, the status (left button pressed) and the current coordinates of the mouse moved by another user have to be transmitted to other participants. If this information were transmitted periodically, it would cause a high data rate and an extremely high redundancy. The ideal system should transmit only data within the active session that reflect a change in either position or status.

49. 2.5.2 Variation of the Data Volume of Consecutive Information Units. A second characteristic to qualify data streams concerns how the data quantity of consecutive information units or packets varies. If the quantity of data remains constant during the entire lifetime of a data stream, then we speak of a strongly regular data stream

50. Strongly regular data stream This characteristic is typical for an uncompressed digital audio-video stream. Practical examples are a full-image encoded data stream delivered by camera or an audio sequence originating from an audio CD

51. weakly regular data stream If the quantity of data varies periodically (over time), then this is a weakly regular data stream

52. Variation of the Data Volume of Consecutive Information Units. Strongly Regular e.g. uncompressed audio/video Periodically Regular e.g. MPEG

53. Variation of the Data Volume of Consecutive Information Units. cont… Some video compression methods use a concept that encodes and compresses full images individually. The resulting information unit is a relatively large data packet in the data stream.

54. Variation of the Data Volume of Consecutive Information Units. Irregular e.g. JPEG Data streams are called irregular when the data quantity is neither constant, nor changing by a periodic function. This data stream is more difficult to transmit and process compared to the variants described earlier

55. JPEG (Joint Photographic Experts Group) is an ISO/IEC group of experts that develops and maintains standards for a suite of compression algorithms for computer image files A JPEG file is created by choosing from a range of compression qualities (actually, from one of a suite of compression algorithms). When you create a JPEG or convert an image from another format to a JPEG, you are asked to specify the quality of image you want ISO compressionalgorithm

56. Since the highest quality results in the largest file, you can make a trade-off between image quality and file size. Together with the Graphic Interchange Format (GIF) and Portable Network Graphics (PNG) file formats, the JPEG is one of the image file formats supported on the World Wide Web, usually with the file suffix of ".jpg".

57. Connection between consecutive packets: Interrelated No gaps in between Maximal throughput and use of reseources

58. Connection between consecutive packets: Non-Interrelated Gaps between some packets

59. Comparison

60. Information Units Logical Data Units – E.g. Video Data Clips Scenes Images Regions Pixels