Continuous Media: Audio, Animation, Video Multimedia Continuous Media: Audio, Animation, Video T.Sharon-A.Frank

Sound Conversion of energy into vibrations in the air. Most sound sources vibrate in complex ways leading to sounds with components at several different frequencies. Frequency spectrum – relative amplitudes of the frequency components. Range of human hearing: roughly 20Hz–20KHz, falling off with age.

Provide personal contact Tell a story Use of Audio Convey Information Set a mood Capture attention Explain a process Provide personal contact Tell a story T.Sharon-A.Frank

Nature of Sound Sound is produced by the vibration of matter. Pressure variations are created in air surrounding matter. Pattern of oscillation is called a waveform. T.Sharon-A.Frank

Waveforms Sounds change over time e.g. musical note has attack and decay, speech changes constantly. Frequency spectrum alters as sound changes. Waveform is a plot of amplitude against time: Provides a graphical view of characteristics of a changing sound. Can identify syllables of speech, rhythm of music, quiet and loud passages, etc.

Computer Representation of Sound: Basic Sound Concepts Waveform: Period Frequency Amplitude Computer Representation of Sound: Sampling Quantization T.Sharon-A.Frank

Period Repeating portion of the sound wave, although sound waves are never perfectly smooth or uniformly periodic. T.Sharon-A.Frank

Frequency Reciprocal of the period, that is, number of periods in a second; measured in hertz (Hz) or cycles per second. Human hearing frequency range is from 20Hz to 20KHz: Infrasound: 0 - 20Hz Ultrasound: 20KHz - 1GHz Hypersound: 1GHz - 10THz T.Sharon-A.Frank

Amplitude Measure of displacement of the air pressure wave from its mean. Subjectively heard as loudness. amplitude T.Sharon-A.Frank

Sound Wave Example T.Sharon-A.Frank

Idea of Audio Digitization Smooth continuous curve of waveform is not directly represented in computer. Amplitude of waveform is measured and recorded at regular time intervals - process is called encoding. Each interval is called a sample. Need Analog-to-Digital Converter (ADC). T.Sharon-A.Frank

Audio Digitization Converting signal from analogue to digital form Analogue signal can vary continuously; digital is restricted to discrete values. Two-stage process: Sampling – measure the value at discrete intervals. Quantization – restrict the value to a fixed set of quantization levels.

Digital Sampling Sample Height Samples Time Samples T.Sharon-A.Frank

Example: 3 bit quantization Digital Quantization Sample Height Example: 3 bit quantization Samples Samples T.Sharon-A.Frank

Audio Encoding (a) A sine wave. (b) Sampling the sine wave. (c) Quantizing the samples to 4 bits. T.Sharon-A.Frank

Process of Sampling Audio T.Sharon-A.Frank

Number of samples per second measured in hertz (Hz). Sampling Rate Number of samples per second measured in hertz (Hz). More samples = higher quality = bigger files. Example: CD quality = 44,100 Hz (44.1 KHz) Others: 8,000 Hz, 11,025 Hz, 22,050 Hz T.Sharon-A.Frank

Sampling Size Number of bits used to represent each sample. Called also Bit Depth and Sound Resolution. 8-bit - telephone quality (typical of Web). 16-bit - CD quality. Examples: Web quality (11.025 KHz, 8bit) 16 bit, 44.1kHz 8 bit , 44.1KHz T.Sharon-A.Frank

Channels Audio files can support from 1 to 6 channels of audio information. Most common: mono (1 channel) stereo (2 channels) 4-6 channels are usually used for surround sound. T.Sharon-A.Frank

Record Settings Menu for Audio T.Sharon-A.Frank

Sample Audio File Formats aif/aiff wav au mov MPEG/MP3 (MPEG 1 Level 3) RA/RAM MIDI (Musical Instrument Digital Interface) T.Sharon-A.Frank

Audio File Formats (platform specific) aif (aiff) Audio Interchange File Format original audio format for Mac supported by Windows and other platforms wav Waveform Audio File Format original Windows audio file format supported on Mac au u-law (myoo-law) Unix standard audio format T.Sharon-A.Frank

Audio File Formats (container formats) mov (qt) QuickTime, not just Apple/MAC mpg (mpe, mpeg) Moving Pictures Expert Group High quality, excellent compression file format Files tend to be small and sound good ra (ram) RealAudio T.Sharon-A.Frank

Real Audio Format First example of streaming audio on the Web. Quality like that of an AM radio station. Best used for narration and not for music or other sounds. Must have RealPlayer on your system to listen to files. T.Sharon-A.Frank

MIDI MIDI = Musical Instruments Digital Interface Instructions about how to produce music, which can be interpreted by suitable hardware and/or software. Standard protocol for communicating between electronic instruments (synthesizers, samplers, drum machines). Allows instruments to be controlled by hardware or software sequencers.

MIDI and Computers MIDI interface allows a computer to send MIDI data to instruments. Store MIDI sequences in files, exchange them between computers, incorporate them into multimedia. Computer can synthesize sounds on a sound card, or play back samples from disk in response to MIDI instructions Computer becomes primitive musical instrument (quality of sound inferior to dedicated instruments).

Uncompressed Audio Digital Telephony Sampled at 8KHz (8K cycles per second) Quantized to 8 bits per sample (1 Byte) => 64,000bits/s Bandwidth = 64,000/1,024 = Total: 62.5Kb/s T.Sharon-A.Frank

Uncompressed Stereo Audio CD Quality Sampled at 44.1KHz (44.1K cycles per second) Quantized to 16 bits per sample (2 bytes) Two channels for stereo Bandwidth = 2 * 44,100 * 16 = 1,411,200b/s 1,411,200/1024= 1,378Kb/s Total: ~1.4Mb/s T.Sharon-A.Frank

Animation The creation of moving pictures one frame at a time: Literally 'to bring to life‘. e.g. make a sequence of drawings on paper, in which a character's position changes slightly in each drawing. Photograph drawings in sequence, using movie camera that advances one frame at a time. Play back resulting film – character will move.

Use of Animation Capture the viewer’s attention, entertain, or promote. Explain a system or process. Set a mood or create an on-screen environment. T.Sharon-A.Frank

Animation Principles Persistence of vision: object seen by human eye remains mapped on retina for a brief time after viewing. display series of images rapidly and they blend together to create illusion of movement. requires at least 16 frames per second (fps) to look seamless. Television: displayed at 30 frames per second. Movies: displayed at 48 frames per second. T.Sharon-A.Frank

Cell Animation (Frame Animation) Start with first and last image in motion (called keyframes). Draw images between keyframes (process called tweening/morphing) - small changes between images. Images are then layered onto background scene. Images displayed rapidly 15/24/30 frames per second. T.Sharon-A.Frank

Captured Animation Logic and procedural concepts are same as in cell animation (keyframe, tweening, layering techniques). But use computer and video camera. Frame grabbing – record each frame to disk. Edit non-linearly like for video. Can also use scanner or digital still camera, or create each frame in a graphics program (e.g., Painter).

Cell-based Techniques Morphing: uses frames to create illusion of one object changing into another more keyframes = smoother morph. Need at least 15 frames per second. Professional quality uses 24-30 frames per second. Cell-based animation is done more easily on computer than by hand. Easier to create and process than movies. T.Sharon-A.Frank

Path-based Animation (Vector Animation) Create animated objects by following object’s transition over a line or vector. Artist creates one drawing and a path. Computer program manipulates object by drawing frames as object travels over the path. T.Sharon-A.Frank

Sample Animation File Formats Animated GIF mov (QuickTime) SWF (Flash) PNG (Portable Network Graphics) T.Sharon-A.Frank

Animated GIFs Sequence of images can be stored in a single GIF file, and displayed one after another by a Web browser or other software: No browser plug-in required Can specify looping, delay between frames 256 colour palette No sound.

SWF Popular Web animation format. Usually generated by Macromedia Flash. Vector animation format Motion represented as numerical operations on vector data. Can also include bitmapped images (e.g., as backgrounds).

Video is sequence of images displayed at constant rate. Rate is in frames per second (fps). Needs around 10-15 fps for smooth motion. For example, video is displayed at 24-30 fps. T.Sharon-A.Frank

Explain a complex process Personalize a Multimedia experience Use of Video Tell a story Entertain an audience Explain a complex process Personalize a Multimedia experience T.Sharon-A.Frank

Movie - Characteristics Built from Frames - Each frame is an Image Image characteristics (i.e., resolution, colors) Clip Scene Shot Frame Frame T.Sharon-A.Frank

A movie may consist of several files T.Sharon-A.Frank

Sample Video File Formats MOV AVI RA/RAM ASF WMV MPEG1/2/4 FLV T.Sharon-A.Frank

MPEG MPEG = Moving Picture Experts Group. Family of compression algorithms. High compression rate. Algorithm stores only changes from one frame to another rather than each entire frame. Video info then encoded using DCT. Lossy compression but loss of data is generally imperceptible to human eye. T.Sharon-A.Frank

MPEG Video Format MPEG-1: used for video CD 352x240 @30 frames/second. ISO/IEC MPEG (Moving Picture Experts Group) Series of standards including: MPEG-1: used for video CD 352x240 @30 frames/second. MPEG-2: used for compressing DVD and high-definition television (HDTV). MPEG-4: used to transmit audio, video, and graphics at low bitrate – can be delivered over very slow connections (56 Kbps). T.Sharon-A.Frank

Video Acquisition TV set and PC monitor look similar. TV is analog Gets info from continuously varying broadcasting waves. PC is digital Data displayed is digital. Digitization performed by video capture card. Typical card is combination of hardware and software. Capture card typically creates uncompressed files that then must be passed into a codec chip or software. T.Sharon-A.Frank

Issues Affecting Video File Size Movie Length - simple principle: shorter video clips result in smaller files. Frame Size - full-screen video is 640x480. Typical frame size on WWW = 160x120 although can handle 320x240. Frame Rate: TV quality = 30 fps Web = 15 (or even 10) fps. T.Sharon-A.Frank

TV Computer Display Video Presentation PAL NTSC SECAM HDTV T.Sharon-A.Frank

TV Standards NTSC (National Television Standard Committee) Used by TV and video industries in US and Japan Taiwan, parts of South America, … PAL (Phase Alternating Line) Used in Western Europe, Australia & New Zealand, China , … SECAM (Sequential Color and Memory) France and former Soviet Union Standard only used for transmission Very different technology than PAL.

Video (PAL, HDTV) Bandwidth PAL Resolution: 640 x 480 3 bytes per pixel Frame rate: 25 fps Bandwidth: 640 * 480 * 25 * 24 = 184,320,000b/s Total: ~176Mb/s HDTV bandwidth increased by factor of 5.33: ~936Mb/s T.Sharon-A.Frank

More PAL and NTSC Examples PAL uncompressed 768x576 pixels per frame x 3 bytes per pixel x 25 frames per second ≈ 31 MB per second ≈ 1.85 GB per minute NTSC uncompressed 640x480 pixels per frame x 3 bytes per pixel x 30 frames per second ≈ 26 MB per second ≈ 1.6 GB per minute