Simulation of High Quality Sound Fields for Interactive Graphics Applications Nicolas TSINGOS iMAGIS - GRAVIR / IMAG - INRIA UMR CNRS C5527

Introduction Explosion of "3D sound" techniques consumer products Multi-modal experiences computer animation, video games simulators, teleconferencing Acoustic simulations room and environmental acoustics

Overview Context Previous approaches Interactive treatment of sound occlusion Integrated sound and graphics rendering system Adaptive simulation technique

Underlying context Modeling and propagation simulation wave theory geometrical acoustics statistical acoustics Human hearing and restitution systems sound perceived in 3D need to reproduce spatial audio Signal processing

Basics of sound rendering * source receiver Variations of the pressure propagation delay (sound speed  340 m/s) All info can be represented by a digital filter impulse response Rendering Convolution

Overview Context Previous approaches Interactive treatment of sound occlusion Integrated sound and graphics rendering system Adaptive simulation technique

Previous work: acoustic simulations Finite element approaches [Jean98, Hothersall+91,Kopuz+95,Kludsuweit91,Wright95] solve the wave equation discretize space and time treat all propagation phenomena high computational cost (2D and steady state) Geometrical acoustics sound rays valid for high frequencies

Previous work: image sources [Allen+79,Borish84, Foster+91, Strauss+95] Specular reflections Straightforward but exponential cost

Previous work: Ray/beam tracing [Martin+93,Dalenbäck96, Funkhouser+98, Monks+96] More general Faster but less flexible updates

Previous work: Radiant exchanges [Kuttruff71, Lewers93, Goral+84, Cohen+85, Nishita+85] Diffuse reflections [Hodgson91]

Previous work: interactive acoustics Artificial reverberators [Schroeder62,Moorer79,Jot+92] limited control Audio and video integration [Takala+92,Hahn+95] post-processing "timbre-trees" Multi-media libraries [SGI Cosmo 3D, Intel RSX, Microsoft Direct Sound] limited propagation effects

Overview Context Previous approaches Interactive treatment of sound occlusion Integrated sound and graphics rendering system Adaptive simulation technique

Sound waves occlusion Obstacles cause diffraction not a 0/1 "visibility" frequency dependent Simulating diffraction finite elements diffracted rays [Keller62,Kouyoumjian+74] difficult and costly from Isaac Newton's Principia (1686)

An interactive geometrical approach Use a 3D polygonal model identify the diffracting objects use graphics hardware Keep the frequency dependent aspect "Ray-tracing" use "thick" rays defined by first Fresnel ellipsoids Extended visibility term between 0 and 1

- Fresnel ellipsoids + + M R S k =1 M S + R - + Alternate constructive and destructive contributions Twice the unoccluded energy in the first ellipsoid

General algorithm occluders Receiver occlusion map Source and receiver positions Receiver For each frequency Render occlusion map Compute attenuation occlusion map Source Build filter

Computing the occlusion term Render the objects in the first Fresnel ellipsoids parallel projection Occlusion factor: (occluded area) (area of the largest Fresnel zone) 400 Hz 4000 Hz

Results [GI97] Results Source v t Receiver t v Receiver Source

The Fresnel-Kirchoff diffraction theory A wave is a sum of "wavelets" Kirchoff integral theorem Contribution of unoccluded "wavelets" Secondary wavelets R s Primary wave

Computing the diffraction integral Compute a depth map of the obstacles read the Z-buffer For each occluded pixel evaluate occluded contribution subtract obstacle R S

Results: diffraction patterns [AES98] sampling a receiving plane 200*200 pixels 0.02 sec. / point avg. (180 MHz SGI O2 workstation)

Diffraction patterns (II) [AES98] Square apertures wide aperture narrow aperture close-up Fresnel diffraction Fraunhofer diffraction

Summary Two methods to compute sound occlusion Fresnel ellipsoids generic and fast use graphics hardware Fresnel ellipsoids Fresnel-Kirchoff integration

Overview Context Previous approaches Interactive treatment of sound occlusion Integrated sound and graphics rendering system Adaptive simulation technique

Integrating sound with 3D graphics Fabule platform Sound path Image-source model Doppler shifting Occlusions Source, receiver and surface characteristics Télémédia project (CNET)

Overview Context Previous approaches Interactive treatment of sound occlusion Integrated sound and graphics rendering system Adaptive simulation technique

Goal Treat both diffuse and specular reflections source receiver Treat both diffuse and specular reflections Listener independent solution Radiant exchanges between patches

Hierarchical radiosity Designed for lighting simulations [Hanrahan+91,Cohen+88]

Extension to temporal phenomena Echoes frequency band intensity (I) arrival time (T) duration Temporal radiosity list of echoes stored on patches (echograms) duration I t T

Temporal transport spreading duration duration+spreading I FF I.FF t t Tmin Tmax T+ Tmin Tmax Pj Pi Tmin

Hierarchical simulation Efficient hierarchical representation Refinement based on energy based on echo spreading

Adding specular reflections Non diffuse radiosity [Sillion+89/91, Immel+86, Aupperle+93,Christensen+96] Image-sources model Hierarchical specular exchanges use patches centroids shooter reflector gatherer

Echo merging Merge echoes within a given temporal threshold Control the time-complexity Take interferences into account use difference in arrival times

Results [Siggraph97(sketch)] Impulse responses listener-independent Visualization wavefront propagation energy mappings Acoustic predictions Collaboration with CSTB

Conclusion Two fast sound occlusion models use graphics hardware Integrated sound rendering system computer animation and virtual reality A model for temporal radiant exchanges hierarchical both diffuse and specular reflections listener-independent solution tunable time/accuracy tradeoff

Extensions Validation tests in progress Treat high reflection orders adaptive echo bucketing statistical approaches [Monks+93,Martin+93] Clustering [Sillion+95,Paquette+98] Dynamic environments fast update of energy transfers [Drettakis+97] Application to radio waves cellular phones and wireless networks

Push-pull Push Pull copy echoes in sons' echograms preserve intensity reduce width Pull copy echoes in father's echogram multiply intensity by area ratio do not combine echoes

Reconstructing an impulse response Energetic response wide echoes (specular+diffuse paths) dirac impulses (pure specular paths) Reconstruct wide echoes render the echoes energetic enveloppe use white noise to get the missing phase Band pass filter and add up

Updating the sound path information Evaluate the delay at each time-step iterative approach [Noser+95] simple interpolation Combination of 4 filters source, receiver, environment, reflection short Finite Impulse Response (128 pts/32 kHz) Calculating the sound occlusion use a mirrored scene for image-sources use the previous semi-quantitative approach