COMPUTER AUDIO CGDD 4003

What is Sound?  Compressions of air or other media (such as water or metal) from something vibrating  Sounds are made up of high frequency and low frequency sounds  Don’t confuse pitch (frequency) with volume!  Volume is measured in decibels (dB)  Frequency in Hertz (Hz) = cycles per second  Humans only hear from 20Hz to 20KHz Frequency Strange Fact! Speed of sound (Air: 340m/s; Water:1,230 m/s; Gold: 3,240m/s):

Spatial Sound  1 Channel – “mono”. Can be split to several speakers; still no direction  2 Channels – “stereo”. Fades from left to right. Can determine direction  5.1 Audio – Common for home theaters  3D Sound? – Video games (PC). Still has time to develop

The Human Side (20Hz-20KHz)

The Equal Loudness Contour Killa Hurts Ouch!

A Note about decibels  A decibel is 1/10 th of a bel  Abbreviated dB  This is the perceived loudness, which increases linearly as power increases exponentially  Something sounds twice as loud?  10·log 10 (2) = 3.01dB  In gaming, volume usually ranges 0.0f-1.0f

Human Perception (InterAural Time Difference)  Sound hits both ears  Difference in time Hits Right Ear First Hasn’t gotten to left yet

How Computers Perceive Sound  Digitization (DAC and ADC)  Computers “listen” to the amplitude a certain number of times per second (sample rate)  44K is CD  22K is good  8K is lame  Computers have to approximate what they heard and assign it a number  4 bits = 16 level to approximate to  16 bits = 2 million levels to approximate to

Original Sound Amplitude (in dB)Frequency

Low Sampling Rate TIME

Low Sampling Rate What the computer hears TIME

High Sampling Rate TIME

High Sampling Rate TIME

2 bits per sample 4 Approximations TIME

StairStep Effect TIME Called “quantization errors”

3 bits per sample 8 Approximations TIME

Less StairStep TIME

Signal to Noise Ratio (SNR)  Represents the quanitization error  8-bits = 128 discrete values (upper-half only)  Sample is rounded up or down  SNR is 256:1  256:1 translates to 48dB (difference in average noise to max signal)  16-bit = 32K discrete values (upper-half)  SNR = 65,536:1, or 96dB

In General  Sampling rate affects range of frequencies you can capture (Nyquist)  Bits per sample affects noise level as well as volume range  What about recording:  Rock?  Mozart (or anything on NPR for that matter)?  Voice/dialog?

Capturing Sounds  Usually done with:  a microphone (such as voice)  Line in  CD  Hollywood Edge®  Computer has sound card  Input types (RCA, MIDI, mini, ¼”, XLR)  Card has quality (plays 16-bit sound)  Need some kind of software  SoundForge/Audacity  Windows SoundRecorder (gag)

Typical Pipeline Permanent Storage Decoding (from mp3, ogg, etc) Memory Buffer Sound Channel Processing (2D/3D effects) Hardware mixing and DAC Individual Channel

Sample Playback  Playback  Loaded entirely into memory (called “sample” as well)  Streamed (pre-buffer data using a circular buffer)  Channel properties  Pan – left/right  Pitch – frequency  Volume

Compressed Audio  Requires a codec (compress/decompress)  Lossless (e.g..zip files)  Lossy  Bit-reduction (ADPCM, reduces bps from 16 to 4) Simple Used on Sony PSP, Wii and Nintendo DS  Physcho-acoustics (.mp3,.ogg,.wma) Discard sound we don’t normally hear anyway Hard to implement CPU intensive PS3, Xbox 360, PCs Note: mp3 format requires licensing fees to Franhofer-Thompson!

ADSR Envelopes  Used for defining the volume of a sound Attack Decay Sustain Release Time Volume

3D Sound  Don’t have 5.1?  Panning is one option  Psycho-acoustic options  Head-Relative Transfer Function (HRTF)  Tweak the frequencies to match your ears  Sounds have position and velocity  There is a listener component (like a camera)  Relationship between the two  Attenuation (with distance)  Occlusion (low-pass filter)  Doppler (relative velocities)

3D Sound  Environmental effects  Reverb (depends on materials in room)  Echo (depends on size of room)  Occlusion (a wall blocking part of the sound)  Obstruction (no direct path to the listener  Competing reverb technologies  I3DL2 (Interactive 3D Audio Rendering Level 2)  EAX (Creative Labs)  Almost identical

MIDI (Musical Instrument Digital Interface)  MIDI – a method for representing sounds electronically  Became popular in the 80’s  Send 16 different channels (tracks) at one time  Have a total of 128 possible instruments

The Keyboard  The MIDI Keyboard  No audible sounds  Generates a series of 1’s a 0’s (on/off)  Signals represent  Note, loudness  Length, type of instrument…  Signals come out of the keyboard and usually go into a sequencer

The Sequencer  Can be a PC  Responsible for recording individual tracks of music  Responsible for playback  Receives input from keyboard  Sends output to synthesizer

The Synthesizer  Receives 1’s and 0’s from the sequencer  Interprets the 1’s and 0’s to produce audible sounds  Piano  Drums…  Saxophone…  Sounds are sent to speakers

Speakers  Like you haven’t seen these before…

MIDI

MIDI vs Digital Recording  MIDI:  Smaller file size (like 10-20K)  Change keys/tempo/looping on the fly!  Song sounds different on every sound card  No singing allowed!  Also a DLS format (DownLoadable Sound)  Digital Recording:  Larger file size (like 5M)  Sound is close approximation to real thing

Sampling  There are two main approaches to synthesis:  Sampling  FM Synthesis  Sampling  A sample is a recording of actual instrument/sound  Samples are taken at certain intervals  Samples are then shifted up or down depending on the note

Sampling

FM Synthesis  Basic waves:  Sine  Square  Saw  Triangle  Noise

FM Synthesis  Start with basic waveform, and have one wave modulate the other  Here’s volume modulation  440 sine wave, control 2Hz:  440 sine wave, control 880Hz:  440 sine wave, control 3KHz:

Interactive Music  Music adapts based on current state of game  Music broken into chunks  Called segments (or cues)  Can be played back to back  Can be smoothly cross-faded  Segments are combined into themes  fmod’s Sound Designer can do this

Themes in fmod

Sound Variations  Sounds can be triggered by events  There’s no reason to play the same sound the same way  Pick a random sample  Change pitch  Change attenuation

Other technology  Lip-synch  Use the amplitude of the wave to control mouth  Analyze phonemes of sample (language neutral)

Common Audio Technology  XAudio (free) – cross-platform  OpenAL (free) – cross-platform  XACT (free) – Xbox/Windows  fmod (commercial) – cross-platform