Does Spatialized Audio Change Everything? Now I’ve been at Unity almost 2 years. I’m focused mostly on VR and AR audio and video. By video, I mean video decoders. So what does that mean? For audio, it basically means focusing on adding more realistic capabilities to Unity audio. We have 3 audio engineers in Copenhagen focused on core audio. And I’m here in Bellevue focused on things like HRTF and the frameworks to support HRTF (spatializers), ambisonics, and environmental audio. That sounds exciting, but so far my role has really been to get audio working on some new VR and AR platforms and to create frameworks to support this stuff. Or to make sure the Unity framework does not prohibit doing these things. On the video side, I’m making sure our Unity video player works efficiently on new XR platforms. Making sure it works with spatialized audio, making sure it works in AR with the re-constructed meshes captured from the room you’re in. Today, I want to focus on the XR audio stuff I’ve been thinking about, mostly the framework or audio engine around HRTF nodes (or spatializers). “Does Spatialized Audio Change Everything?” I want to explore that with you all a little bit today.

What is a traditional audio engine? Tree of nodes Node performs an operation Buffers of data flow throw these nodes Metadata describes buffers Audio source nodes work on mono data Audio mixer nodes work on stereo data (in XR)

What are some audio nodes? File nodes stream in data De-compression nodes produce raw PCM data Low-pass filter nodes reduce high frequencies High-pass filter nodes reduce low frequencies Mixer nodes add all input buffers 3D panner node outputs stereo based on 3D position

What is a spatializer node? Realistic 3D panner node Left channel is sent data for left ear Right channel is sent data for right ear More subtle than traditional 3D panner node

What is unique about a spatializer node? Fundamental for 3D sounds (for XR and headphones) Expensive

Other fundamental, expensive nodes? De-compression nodes Project-specific nodes Nothing else is so fundamental, necessary, and expensive

What’s the problem? Spatializer nodes fit well into the traditional audio engine Problem is performance

What’s the node performance? Performance numbers taken on Samsung S8, Android phone Most nodes are relatively cheap De-compression nodes cost 0.75% CPU per sound (Vorbis) Spatializer nodes cost 1.0% CPU per sound (Oculus)

What is game audio’s budget? 1ms on main thread (30fps game) 0.5ms on main thread (60fps XR game) 5-10% of device’s memory and CPU resources 50% of core on audio thread

What’s the cost of de-compressed, spatialized sounds? 16 sounds cost 1.75% x 16 = 28% 28 sounds cost 1.75% x 28 = 49% 32 sounds cost 1.75% x 32 = 56% 64 sounds cost 1.75% x 64 = 112%

What can we do with mobile XR? We can support 28 sounds, if we do nothing else But we also want: Occlusion and low-pass filtering Play requests Reverb Event systems 64 sounds for AAA audio

Optimization: Prioritize Prioritize sounds Separate virtual and physical sounds Spatialize most important physical sounds 32 physical sounds, 16 spatialized sounds cost 40%

Optimization: Cheap spatialization Use realistic distance attenuation curves Perform ILT Perform ILD Perform simple, low-pass filtering based on location

Optimization: Group nearby sounds Group sounds at the same location together Spatialize the mixed group of sounds Group far away sounds more aggressively Character sounds play from the mouth and foot nearby Character sounds play from the center far away

Optimization: De-compress less Don’t repeatedly de-compress small, frequently played sounds De-compress once Use more memory and less CPU

Optimizations: All of the above Play 32 physical sounds De-compress 24 sounds Spatialize 8 locations (16 sounds) Use low-LOD spatialization on other sounds 26% CPU usage

Is there a different approach? Google Resonance Windows Sonic Dolby Atmos

Google Resonance Convert each sound into ambisonic format Mix ambisonic sounds Decode one, mixed ambisonic sound and spatialize in one step Scales very well Each sound costs 0.75% to de-compress (same) Each sound costs 0.6% to spatialize (instead of 1.0%) 32 de-compressed, spatialized sounds cost 43%.

Google Resonance limitations Mixer part of audio engine does not exist Or mixers are in 16-channel ambisonic format

Negatives to eliminating Audio Mixers? Sound designer workflow (“Lower the player’s foley.”) Apply expensive effects once to many sounds HDR and side-chaining (activity on mixer affects other sound/mixer properties)

Positives to eliminating Audio Mixers? Very simple pipeline Easy, flexible jobification Low-latency because of few dependencies

My personal thoughts (ambisonic) Initially, I was very intrigued with the ambisonic / Google approach But, the performance improvement is limited for 32 sounds / mobile Not good enough to throw away the traditional audio engine design

My personal thoughts (traditional) Need excellent prioritization / culling algorithm Need lower-quality spatializer to pair with high-quality spatializer Optimizations feel more like traditional game engine / audio opts We can do this!

What do you think?