1. CSCE 552 Spring 2010
Audio
By Jijun Tang

2. Announcements Final demo Homework 4 May 3rd, Monday, 2:00pm
Open to public with pizza and drink
Homework 4
Evaluate your experience with the class, including improvements for the future, engines, lessons learnt, etc.
Individual assignment
6 points total
Due on the demo

3. Multiplayer Modes: Event Timing
Turn-Based
Easy to implement
Any connection type
Real-Time
Difficult to implement
Latency sensitive

4. Split Screen

5. Multiplayer Modes: Connectivity
Non Real-Time
Floppy disk net
Database
Direct Link
Serial, USB, IrD, … (no hops)
Circuit Switched (phones)
Dedicated line with consistent latency
Packet Switched
Internet
Shared pipe

6. Example Packets
IP Packet
TCP Packet

7. Protocol Stack: Open System Interconnect

8. Protocol Stack: Physical Layer
Bandwidth
Width of data pipe
Measured in bps = bits per second
Latency
Travel time from point A to B
Measured in Milliseconds
You cannot beat the physics
The Medium
Fiber, FireWire, IrD , CDMA & other cell
Table: Max Bandwidth Specifications
Serial
USB
1&2
ISDN
DSL
Cable
LAN
10/100/1G
BaseT
Wireless
802.11
a/b/g
Power Line T1
Speed
(bps)
20K
12M
480M
128k
1.5M down
896K up
3M down
256K up
10M
100M 1G
b=11M
a,g=54M
14M
1.5M

9. Protocol Stack: Transport Layer: TCP
Guaranteed Correct In Order Delivery
Acknowledgement system: Ack, Nack, Resend
Checksum
Out of Band
Connection Required
Packet Window
Packet Coalescence (united)
Keep Alive
Streamed Data
User must serialize data

10. Protocol Stack: Transport Layer: UDP
Non Guaranteed Delivery
No Acknowledgement system
May arrive out of order
Checksum
Not Connected
Source not verified
Hop Count Limit = TTL (time to live)
Required for Broadcasting
Datagram
Sent in packets exactly as user sends them

11. Protocol Stack: Session Layer: Sockets
Based on File I/O
File Descriptors
Open/Close
Read/Write
Winsock
Provides standard specification implementation plus more
Extension to spec prefixed with “WSA”
Requires call to WSAStartup() before use
Cleanup with WSAShutdown()

12. Real-Time Communications: Peer to Peer vs. Client/Server
N = Number of players
Broadcast
Peer/Peer
Client/Server
Connections
Client = 1
Server = N
Broadcast
Peer/Peer
Client/Server
Send 1
N-1
Client = 1
Server = N
Receive

13. Audio Programming

14. Audio Programming
Audio in games is more important than ever before

15. Programming Basic Audio
Most gaming hardware has similar capabilities (on similar platforms)
Mostly programming interfaces differ
Learning fundamental concepts of audio programming is important

16. API Choices DirectSound (part of DirectX API) OpenAL Proprietary APIs
Only available on Windows platforms
XACT for XNA
OpenAL
Newer API
Available on multiple platforms
Proprietary APIs
Typically available on consoles
3rd Party Licensable APIs
Can offer broad cross-platform solutions

17. Analog Sound Wave

18. Basic Audio Terminology and Physics
Amplitude
Measurement of a sound wave’s pressure
Frequency
Measurement of the interval between wave cycles, typically measured in Hertz
Pitch
The perception of frequency
Tuning
Musical distribution of frequencies over keys
Decibel
Measures sound amplitude

19. Digital Representation of a Sound Wave

20. Sampling Most common technique
Sampling involves measuring the amplitude of the analog wave file at discrete intervals
The frequency of sampling is known as sampling rate
Each sample is typically stored in a value ranging from 4 to 24 bits in size
The size of the sample value in bits is known as the ‘bit depth’
Music CDs have a sample rate and bit depth of 44.1 kHz (samples/sec) and 16 bits (sample size)

21. Quantization Error in Sampling

22. Bit Depth and Signal Noise
Bit depth of sample data affects signal noise
Signal to noise ratio = number of available bits / 1
For example, 8-bit samples have a 256:1 SNR (~48 dB), and 16-bit samples have a 65,536:1 SNR (~96 dB)
Decibel ratio is calculated using 10 x log10 (ratio) or x log e (ratio)

23. Sampling Frequency and Frequency Reproduction
Sampling frequency affects range and quality of high-frequency reproduction
Nyquist Limit
Frequencies up to one-half the sampling rate can be reproduced
Audio quality degrades as frequency approaches this limit

24. Sampling Errors vs. Sampling Frequency

25. Modern Audio Hardware Samples are piped into sound “channels”
Often a hardware pipeline from this point
Various operations, such as volume, pan, and pitch may be applied
3D sounds may apply HRTF algorithms and/or mix the sound into final output buffers.

26. Sound Playback Techniques
Two basic playback methods:
1. Play sample entirely from memory buffer
2. Stream data in real-time from storage medium
Streaming is more memory efficient for very large audio files, such as music tracks, dialogue, etc
Streaming systems use either a circular buffer with read-write pointers, or a double-buffering algorithm

27. Playback

28. Sample Playback and Manipulation
Three basic operations you should know
Panning is the attenuation of left and right channels of a mixed sound, results in spatial positioning within the aural stereo field
Pitch allows the adjustment of a sample’s playback frequency in real-time
Volume control typically attenuates the volume of a sound

29. Compressed Audio Format
Compressed audio formats allow sound and music to be stored more compactly
Bit reduction codecs generally are lightweight
ADPCM compression is implemented in hardware on all the major current video game console systems
Psycho-acoustic codecs often have better compression
Discard sounds our ears would not typically be able to hear
Require substantially more computational horsepower to decode

30. MP3, Ogg Vorbis, Licensing & Patent Issues
The MP3 format is patented
Any commercial game is subject to licensing terms as determined by Fraunhofer & Thompson Multimedia, the holders of the patents
Ogg Vorbis is similar to MP3 in many ways
Open source and patent-free (royalty-free)
Be aware of patent and license restrictions when using 3rd party software

31. 3D Audio Two sets of data required when working in world coordinates:
Listener Data
Composed of world position and orientation (virtual microphone in the world)
Source Data
Composed of sound position, orientation, velocity, etc (virtual sound source in the world)

32. Listener/Source

33. Environmental Effects
Environmental effects nearly always implemented in hardware
Sound transmission is categorized in three ways
Direct transmission
Early reflections (echo)
Late reflections (reverberation)

34. Sound Transmission Categories

35. Environmental Effects Standards
EAX 2.0 and beyond
EAX 2.0 developed by Creative Labs and released as an open standard
EAX 3.0 and 4.0 remain proprietary Creative Labs standards
I3DL2
Open standard developed by IA-SIG, similar to EAX 2.0 in functionality

36. Programming Music Systems
Two common music systems
MIDI-based systems
(Musical Instrument Digital Interface)
Digital audio streaming systems
(CD audio, MP3 playback, etc)

37. Advantages and Disadvantages of MIDI
Actual music data size is negligible
Easy to control, alter, and even generate in real-time
High quality music is more difficult to compose and program
Only effective if you can guarantee playback of a common instrument set

38. Example
Go online and find MIDI

39. Other MIDI-based technologies to be aware of
DLS (DownLoadable Sound) Format
A standardized format for instrument definition files
iXMF (Interactive eXtensible Music Format)
New proposed standard for a container format for interactive music

40. Advantages / Disadvantages of Digital Audio Streams
Superb musical reproduction is guaranteed
Allows composers to work with any compositional techniques
Some potential interactivity is sacrificed for expediency and musical quality
Generally high storage requirements

41. A Conceptual Interactive Music Playback System
Divide music into small two to eight-bar chunks that we’ll call segments.
A network of transitions from segment to segment (including loops and branches) is called a theme.
Playing music is now as simple as choosing a theme to play. The transition map tracks the details.

42. Advanced Audio Programming
3D Audio Environmental Effects Integration
Audio Scripting and Engine Integration
Lip-sync Technology
Advanced Voice Playback
Voice Recognition

43. 3D Audio Environmental Effects Integration
Environmental effects should be driven by a room’s shape and material composition.
Can determining the optimal effect settings be done automatically?
This may be important as game worlds become larger and more complex

44. 3D Audio Environmental Effects Integration (cont)
Sound occlusion and damping is a particularly difficult problem to solve
This is essentially a pathfinding problem for audio.
Doors can dynamically affect a sound’s properties
Very few titles have even attempted a robust, general-purpose, and automated solution to these problems.

45. Room Acoustics

46. Dynamic Occlusion

47. Audio Scripting and Engine Integration
Very little audio programming should be done by general game programmers
Game Engine should offer robust support for audio triggers and scripts
Engine should deal with audio scripts, not “sound files”
Why is this so important?

48. Audio Scripting
Many situations require much more information than can be embedded in a linear audio file
Sound Variation
Sound Repetition
Complex Sound Looping
Background Ambience

49. Lip-sync Technology
Lip-sync technology is a blending of audio and visual techniques to create realistic-looking speech by in-game actors.
Simple techniques such as waveform amplitude measurement has worked previously, but…
In future titles, it will be considered inadequate.
Much work can still be done in this field.

50. Advanced Voice Playback
Real-time spoken feedback is especially important in sports titles (simulated announcers)
Game are reaching the limits of what current techniques (canned, prerecorded phrases combined in series) can provide.
Again, this is an opportunity for future groundbreaking audio work.

51. Voice Recognition
Spoken commands are much easier to deliver in certain situations.
A great example of this? Squad-based tactical shooters.
Current generation systems are still very error prone. A great opportunity for breakout audio technology.