Introduction to Interactive Media 10: Audio in Interactive Digital Media

Introduction to Interactive Media What is Sound? All sounds are produced by the conversion of energy into vibrations. Vibration becomes a wave. When it reaches the ear, it causes the eardrum to vibrate at the same frequency

Introduction to Interactive Media Audible Spectrum Humans can hear sounds in range of 20 Hz – 20 kHz. Highest note on piano is approx. 4kHz.

Introduction to Interactive Media Sound Over Time Most sounds change frequency over time. Represented in waveform.

Introduction to Interactive Media Music vs. Speech Music waveform Speech waveform Waveform conveys gross character and dynamics of sound. Can provide cues in syncing media.

Introduction to Interactive Media The Nature of Sound Wave captures three features of sound: –Amplitude Perceived as volume. –Frequency Perceived as pitch. –Duration Length of time sound lasts.

Introduction to Interactive Media Getting Sound in and Out of the Computer Analog to Digital Converter captures separate measures of sound amplitude. –Samples are recorded as digital numbers. Digital values are used to recreate the analog form using a Digital to Analog Converter.

Introduction to Interactive Media Recording Sound Best to not do directly into computer as noise from within the computer is picked up. Sample rate can be reduced when recording speaking.

Introduction to Interactive Media Recording Sound When capturing sound, adjust levels. –If amplitude is too low, sound quality is reduced; –If amplitude is too high, clipping occurs and produces distortion

Introduction to Interactive Media Sampling Sound Quality of the sampling depends on: –Sample resolution –Sample rate.

Introduction to Interactive Media Sample Resolution Number of bits to encode amplitude. Like images, more bits used to describe information – more accurately it will be represented. Two common sample resolutions are 8-bit and 16-bit. –8-bit resolution captures 256 different amplitude levels. –16-bit sound has 65,000 different levels. CD quality sound. Inadequate sample resolution can distort the sound.

Introduction to Interactive Media Sample Rate Number of samples taken in a fixed interval of time. –Stated in thousands of Hertz, or kilohertz. –CD-quality sound captures 44.1kHz to record frequencies as high as 22.05kHz. (The highest frequency the human ear can detect is 20kHz.) Two measurements capture each cycle of the sound wave: High value or peak Low value or trough.

Introduction to Interactive Media Sound File Formats Common sampled sound file formats: –WAV –AIFF –AU MP3 is most popular compressed format (lossy) –10:1 file size ratio – WAV:MP3

Introduction to Interactive Media Manipulating Sound Common applications that allow for recording and generate effects. –Apple’s GarageBand –Adobe’s Soundbooth –Sony’s Sound Forge

Introduction to Interactive Media Common Audio Edits Hiss Removal –sample the hiss and remove hiss based on what you sampled. Normalization –brings the average or peak amplitude to a target level Time stretching Pitch alteration

Introduction to Interactive Media Use of Audio in Interactive Media Ambience Sound Effects Auditory Feedback to aid Usability Music Speech

Introduction to Interactive Media Sound on the Web Not always needed (can be annoying) –Became popular on the Web when it became possible (late 1990s – early 2000s) –Fell out fashion on the Web –Web has become more cinematic

Introduction to Interactive Media How to Integrate Sound Authoring applications allow for precise syncing of audio within interactive experience.

Introduction to Interactive Media How to Integrate Sound HTML5 –Prior to HTML 5, there was not a standard for playing audio files on a web. –HTML 5 defines a new element which specifies a standard way to embed an audio file on a web page: the element. For more information re: format and browser support:

Introduction to Interactive Media Beyond Sampled Sound Sampled sound –All audio discussed so far: WAV, MP3, AIFF, AU, etc… –Sound is stored as a description of the sound.

Introduction to Interactive Media Synthesized Sound Sound is stored as a series of commands for the computer to reproduce the sounds. Analogous to vector-based graphics. MIDI (Musical Instrument Digital Interface). –Codes provided for: Specific instruments Notes Force and duration of note Routing commands to different instrument channels Specialized control functions.

Introduction to Interactive Media MIDI Simplest system contains: –Digital musical instrument to create messages –Sound synthesizer to interpret the messages –Amplifier/speaker output system.

Introduction to Interactive Media MIDI Videos Composing – Building Your MIDI Studio –