Chapter Two: Multimedia Basics and Representation Kotebe Metropolitan University Computer Science & Technology Department Multimedia (CoSc4151) Chapter Two: Multimedia Basics and Representation

Outlines 2.1. Digital Multimedia Characteristics Multimedia Elements and Characteristics Text Graphics/Image Animation Audio Video Virtual Reality 2.2. Audio Formats and MIDI Basics of Digital Audio Digitization of Sound Nyquist Theorem Signal to Noise Ratio (SNR) Sampling and Quantization

Multimedia Elements & Characteristics Text Graphics/Image Animation Audio Video Virtual Reality A woman using virtual reality product

Multimedia Elements: Text Text should be: Appropriate for the target audience. Easy to read. Serif typefaces are preferred for printed material. Sans serif typefaces are preferred for on-screen display. Formatted consistently throughout the presentation.

Multimedia Elements: Graphics/Image Graphics are an important part of the communication process. They can be used to: Highlight information Set a mood or tone Provide examples Serve as backgrounds The two types of graphic used in multimedia are raster and vector. Vector graphics are made up of arcs and lines.. Raster graphics are made of dots.

Multimedia Elements: Graphics When using graphics, the multimedia designer must: Determine the best balance between the size and quality. Use appropriate graphics for the intended purpose and audience. Choose appropriate file formats Standard for the internet: JPEG (Joint Photographer Experts Group) GIF (Graphics Interchange file format) PNG (Portable Network Graphics) Most popular TIFF - Tagged Image File Format BMP – Bitmap PCX - Windows Paint PICT - Macintosh Teacher Note: This would be a good opportunity to review the image file formats that students learned in 1.00.

Multimedia Elements: Animation 2-D and 3-D animations are useful in multimedia in the areas of entertainment, education, and training. They can be used to create simplified illustrations of a simulation or dramatization. They can be much easier to understand because they are less complex than video. 2-D animations have smaller file sizes than video files which means quicker loading or downloading of the files. Teacher Note: You can use the links below as an example or find examples of your own. Example of a 2-D animation video (flash) used to explain human respiration. This animation includes text. http://www.metacafe.com/watch/2342235/human_respiration_3d_demonstration/ Examples of 3-D industrial animations from Claro Industrial Animations. http://www.claroanimations.com/portfolio.php

Multimedia Elements: Sound Sounds in multimedia titles could include: Music. Narrations. Sound effects. Sound waves are vibrations that are created when we speak. Sound waves are analog signals because they are continuous, fluctuating waves with no interruptions.

Multimedia Elements: Sound Computers are digital machines, meaning that they represent data with 1s and 0s. To use sound on the computer, the sound waves must be converted from analog to digital form, or digitized. This conversion process is called sampling.

Sampling Sampling is a means of reproducing a continuous event, such as sound or motion, by recording many fragments of it. It involves taking “snapshots” of a sound wave in rapid intervals. These samples, or bits of information, are saved as numbers to allow the computer to process them. Teacher Note: How Stuff Works has a good article with excellent pictures depicting the sampling process. You could let your students read the article or you could just show them the images as you explain the sampling process. Entire Article: http://communication.howstuffworks.com/analog-digital.htm Images illustrating sampling: http://communication.howstuffworks.com/analog-digital3.htm

Audio File Formats AU – (Audio) file created by Sun Microsystems and used on computers running the UNIX operating system. MP3 – (Mpeg-1 Audio Layer 3) very compressed file that is popular for music stored on portable players and on the Internet because it can reproduce near-CD quality audio in small file sizes. MIDI – (Musical Instrument Digital Interface) file format for creating and/or playing music with instruments using synthesizers and sound cards. Teacher Note: Native format refers to the default format for a specific software package.

Audio File Formats (continued) WAV – (Waveform) file format developed jointly by IBM and Microsoft as the native format for Windows sound files. Produces high-quality sound. Generates large file sizes because it is uncompressed. Commonly used to edit sound which is then saved in a different compressed format for distribution. WMA (Windows Media Audio) proprietary file format developed by Microsoft originally to compete with the .MP3 format. More compressed than .WAV files.

Multimedia Elements: Video Videos allow the audience to view actual events instead of just reading about or listening to them. Sources for videos include web sites and stock film companies.

Multimedia Elements: Video Videos can be used in: CD-ROMS Games Presentations Video simulations Video conferences Websites. Videos vary in quality.

The Video Format The file format of the video determines: Which programs can open and play it. How much storage space it occupies. How fast it travels over an Internet connection.

Video File Formats AVI (Audio Video Interleave) MOV (Movie) Windows format, plays in Windows Media Player Very good quality, even at smaller resolutions Large file size – not recommended for delivering video over the Internet. Popular format for videos stored on a computer. MOV (Movie) Apple format, plays in the QuickTime Player Very good quality Popular format for videos downloaded from the Internet. Teacher Note: The most common player has been listed. Other players may play the video files. For example, QuickTime will play a large variety of files besides .mov files.

Video File Formats MPEG (Moving Pictures Expert Group) RM (RealMedia) The standard for compression and storage of audio and motion video for use on the World Wide Web. Creates video small file sizes. Popular format for videos downloaded from the Internet. Its biggest advantage is that It will play in many different media players. RM (RealMedia) Plays in the RealPlayer player. Typically contains a movie clip. Popular format for streaming video viewed over the Internet. Real Player is generally supported by many different computers and operating systems.

Video File Formats WMV (Windows Media Video) FLV (Flash Video) Proprietary video format developed by Microsoft. Plays in Windows Media Player. Popular format for streaming video viewed over the Internet. FLV (Flash Video) New file format widely used on the Internet. Plays in Adobe Flash Player. Very small file size. Popular format for streaming video viewed over the Internet

Multimedia Element: Virtual Reality In multimedia, it can be used to show an environment that surrounds users so that they become part of the experience OR To provide a 3-D representation of an environment or scene for demonstration. Teacher Note: The following links provide optional links to materials using virtual reality in presentations. QuickTime’s Virtual Reality: http://en.wikipedia.org/wiki/QuickTime_VR Website using virtual reality in a presentation to advertise their services: http://www.archiform3d.com/ (takes a few minutes to load) http://www.archiform3d.com/meander/

Review Multimedia Elements: Text Graphics/Image Animation Audio Video Virtual Reality

Basics of Digital Audio Audio information is crucial for multimedia presentations and, in a sense, is the simplest type of multimedia data. However, some important differences between audio and image information cannot be ignored.

Basics of Digital Audio We introduce basic concepts for sound in multimedia The digitization of sound necessarily implies sampling and quantization of signals, so we introduce these topics here.

Basics of Digital Audio We begin with a discussion of just what makes up sound information, then we go on to examine the use of MIDI as an enabling technology to capture, store, and play back musical notes. We go on to look at some details of audio quantization, and give some introductory information on how digital audio is dealt with for storage or transmission. This entails a first discussion of how subtraction of signals from predicted values yield numbers that are close to zero, and hence easier to deal with.

What is Sound? Sound is a wave phenomenon like light, but is macroscopic and involves molecules of air being compressed and expanded under the action of some physical device Since sound is a pressure wave, it takes on continuous values, as opposed to digitized ones

6.1 Digitization of sound Sound is a wave phenomenon like light. The rate at which sound vibrates is known as frequency. Frequency is similar to the musical term 'pitch’ Digitization means conversion to a stream of numbers, and preferably these numbers should be integers for efficiency. Fig. 6.1 shows the 1-dimensional nature of sound: amplitude values depend on a 1D variable, time. Images depend instead on a 2D set of variables, x and y.

Digitization Fig. 6.1: An analog signal: continuous measurement of pressure wave.

Digitization To digitize, the signal must be sampled in each dimension: in time, and in amplitude. The first kind of sampling, using measurements only at evenly spaced time intervals, is simply called, sampling. The rate at which it is performed is called the sampling frequency. For audio, typical sampling rates are from 8 kHz (8,000 samples per second) to 48 kHz. This range is determined by Nyquist theorem. Sampling in the amplitude dimension is called quantization.

Nyquist theorem The Nyquist theorem states how frequently we must sample in time to be able to recover the original sound. For correct sampling we must use a sampling rate equal to at least twice the maximum frequency content in the signal. This rate is called the Nyquist rate. Nyquist Theorem: If a signal is band-limited, i.e., there is a lower limit f1 and an upper limit f2 of frequency components in the signal, then the sampling rate should be at least 2(f2 − f1)

Nyquist theorem Suppose we have a fixed sampling rate. Since it would be impossible to recover frequencies higher than half the sampling rate in any event, most systems have an anti-aliasing filter that restricts the frequency content of the sampler’s input to a range at or below half the sampling frequency. Confusingly, the frequency equal to half the Nyquist rate is called the Nyquist frequency. Then for our fixed sampling rate, the Nyquist frequency is half the sampling rate. The highest possible signal frequency component has frequency equal to that of the sampling itself.

Signal-to-Noise Ratio (SNR) In any analog system, random fluctuations produce noise added to the signal, and the measured voltage is thus incorrect. The ratio of the power of the correct signal to the noise is called the signal-to-noise ratio (SNR). Therefore, the SNR is a measure of the quality of the signal. The SNR is usually measured in decibels (dB), where 1 dB is a tenth of a bel. The SNR value, in units of dB, is defined in terms of base-10 logarithms of squared voltages.

Magnitudes of common sounds, in decibels

Signal-to-Noise Ratio (SNR)

Signal-to-Noise Ratio (SNR) The power in a signal is proportional to the square of the voltage. For example, if the signal voltage Vsignal is 10 times the noise, the SNR is 20 × log10(10) = 20 dB. In terms of power, if the squeaking we hear from ten violins playing is ten times the squeaking we hear from one violin playing, then the ratio of power is given in terms of decibels as 10 dB, or, in other words, 1 Bel. Notice that decibels are always defined in terms of a ratio.

Signal-to-Quantization-Noise Ratio (SQNR) For digital signals, we must take into account the fact that only quantized values are stored. For a digital audio signal, the precision of each sample is determined by the number of bits per sample, typically 8 or 16.

Signal-to-Quantization-Noise Ratio (SQNR) Aside from any noise that may have been present in the original analog signal, additional error results from quantization. That is, if voltages are in the range of 0 to 1 but we have only 8 bits in which to store values, we effectively force all continuous values of voltage into only 256 different values. Inevitably, this introduces a round off error. Although it is not really “noise,” it is called quantization noise (or quantization error). The association with the concept of noise is that such errors will essentially occur randomly from sample to sample.

Signal-to-Quantization-Noise Ratio (SQNR) The quality of the quantization is characterized by the signal-to-quantization-noise ratio (SQNR). Quantization noise is defined as the difference between the value of the analog signal, for the particular sampling time, and the nearest quantization interval value. At most, this error can be as much as half of the interval.

6.1.6 Linear and Nonlinear Quantization We mentioned above that samples are typically stored as uniformly quantized values. This is called linear format. However, with a limited number of bits available, it may be more sensible to try to take into account the properties of human perception and set up nonuniform quantization levels that pay more attention to the frequency range over which humans hear best.

6.1.6 Linear and Nonlinear Quantization Non-uniform quantization: set up more finely-spaced levels where humans hear with the most acuity Weber’s Law stated formally says that equally perceived differences have values proportional to absolute levels: ΔResponse ∝ ΔStimulus/Stimulus

Synthetic Sounds Digitized sound must still be converted to analog, for us to hear it. There are two fundamentally different approaches to handling stored sampled audio. The first is termed FM, for frequency modulation. The second is called Wave Table, or just Wave,sound.

MIDI:Musical Instrument Digital Interface MIDI, which dates from the early 1980s, is an acronym that stands for Musical Instrument Digital Interface. It forms a protocol adopted by the electronic music industry that enables computers, synthesizers, keyboards, and other musical devices to communicate with each other. A synthesizer produces synthetic music and is included on sound cards, using one of the two methods discussed above.

MIDI:Musical Instrument Digital Interface The MIDI standard is supported by most synthesizers, so sounds created on one can be played and manipulated on another and sound reasonably close. Computers must have a special MIDI interface, but this is incorporated into most sound cards. The sound card must also have both DA and AD converters.