Presentation on theme: "Real-Time Auralization of Sound in Virtual 3D Environments by Scott McDermott"— Presentation transcript:
Real-Time Auralization of Sound in Virtual 3D Environments by Scott McDermott
Overview & Objective Develop an adaptive virtual environment that simulates real-time generation of 3D sound. Design algorithms to efficiently and effectively compute realistic 3D sounds in this environment. Apply these techniques to various applications, including simulations, virtual reality, gaming, and modeling.
Outline Sound Perception Digital Sound and Computers 3D Sound Approximations True 3D Sound Surround Sounds (Stereo Expansion) Approach Head Response Transfer Function (HRTF) Approach Beam Tracing Approach The Graphics Analogy
Sound Perception When we hear a sound, we automatically obtain certain information about the source: DirectionDistanceElevation Environmental conditions Status of source
Sound Perception Interaural Delay Time (direction) delay between time arrives at each ear (0 to 0.63 ms) delay between time arrives at each ear (0 to 0.63 ms) Head Shadow (direction and distance) difference in volume from one ear to the other (up to 9 dB) difference in volume from one ear to the other (up to 9 dB) Pinna Response (direction and elevation) outer ear filters sound, compare between two ears outer ear filters sound, compare between two ears Shoulder Response (elevation and direction) reflections off upper body (1-3 kHz) reflections off upper body (1-3 kHz) 8 types of cues for sound spatialization : Head Motion move head to re-evaluate these filters move head to re-evaluate these filtersVision ignore audio cues if different from visual ignore audio cues if different from visual Early Echo Response (distance and direction) echos from environment (50 to 100 ms) echos from environment (50 to 100 ms) Reverberation (distance and direction) late dense echos from environment (> 100 ms) late dense echos from environment (> 100 ms)
Sound Perception An environment with true 3D sound will need to take all of these into account. It must also be able to perform calculations and apply filters in real-time. The result must be convincing to the listener and enhance the virtual experience.
Sound in the Digital World Sound in the physical world exists as waves of pressure changes. A microphone converts pressure changes to changes in voltage. Blah blah blah… An analog to digital convert changes these voltage signals to discrete digital signals. A computer stores, manipulates, and retransmits these abstractions of sound. The sounds can be stored in various formats and qualities (such as mono or stereo, 8 or 16 bit, 11 or 44 kHz). The reverse of this process allows the computer to re-generate the sound.
3D Sound, The Basics… In a virtual 3D environment, sound can originate from an infinite number of locations relative to the observer. Ideally, when the observer hears the sound it should take into account the environment. Specifically:
Distance: Causes sound to arrive at different times.
Reflection & Reverberation: Causes copies of the sound to arrive at different times.
Diffraction & Refraction: Causes sound to bend around objects or arrive at different times.
Absorption & Attenuation: Causes the sound to be weaker when it arrives.
3D Sound Approximations: Surround Sound Surround sound uses various filters to simulate the effects of sound spatialization. These filters create effects such as reverberation, localization, and attenuation. Sound paths are not calculated. Used in most theaters and home entertainment units.
Surround Sound The user is situated with a set of speakers around him. To simulate 3D localization, sound is played louder, out of phase, and/or at slightly different times from each speaker. Comes in a variety speaker placement setups : Bark bark bark!! Dolby 5.1Two Speaker StereoHeadphonesQuadraphonic
3D Sound Approximations: Head Related Transfer Functions Used in conjunction with surround sound to create better 3D approximations. Microphones record sound from within the ear of a person or a model. Differences between original sound and recordings are used to create filters. These filters are applied to generated sounds to create the illusion of dimensionality.
Surround Sound & Head Related Transfer Functions Pros:Cons: Relatively cheap. Effective. Makes sense. Many different approaches (non-standard). Works only with limited speaker positions. Not entirely generic. Still not pure 3D sound.
True 3D Sound 3D graphical environments already exist. Light paths traverse the scene and surface intensities are calculated. Currently, sound paths are at most superficially computed. Yet, programmers already have a wealth of environmental data. Various possible approaches…
True 3D Sound Beam Tracing Approach: Source: Real-Time Acoustic Modeling for Distributed Virtual Environments  Divide the environment into cells or regions. Precompute and store beam paths from various source locations. Lookup, in real-time, reverberation paths from the avatar to the source. Use these paths to calculate delay and attenuation from the original, anechoic, audio signal for each of the echoes.
Beam Tracing Quick and effective (with a good data structure). Intuitive. Scalable for large environments. Needs offline computations. Assumes sources are stationary. Assumes source locations are finite. Pros:Cons:
True 3D Sound On a basic level, we can determine sound propagation similar to how light travels through a 3D environment. One simple, but computationally intensive method would be similar to ray tracing. Ray tracing algorithms are generally very effective but also extremely slow and prone to sampling errors. Most real-time algorithms for graphical computers make various assumptions:
True 3D Sound The Graphics Analogy Objects are made from geometric primitives composed of points. These vertices are transformed to be relative to the camera. Objects outside of the viewing field are clipped. Rays are sent from the camera, through each point on the projection plane, and into the scene. Corresponding pixel values in the viewport are calculated from these rays. The 3D Graphics Pipeline:
True 3D Sound The Graphics Analogy Objects are made from geometric primitives (triangles, rectangles) composed of points. Light intensities are calculated based on surface normals of these points. These intensities are fed into the graphics pipeline.
True 3D Sound The Graphics Analogy Many of these computations are forwarded to optimized 3D graphics cards. Many of these same techniques could be employed for generating realistic 3D sounds. We would need to develop and design 3D sound cards and appropriate algorithms.
Conclusion 3D graphics and many other components of todays computer systems have been almost thoroughly developed. 3D sound is still in the infancy stage. This field has a great deal of research potential.
References Burgress, David, A. Techniques for Low Cost Spatial Audio. ACM UIST, pages 53-59, Ellis, Sean. Towards More Realistic Sound in VRML. ACM Virtual Reality and Modeling, pages , Flaherty, Nick. 3D audio: new directions in rendering realistic sound. Electronic Engineering, pages 49, 52, 55, & 56, Funkhouser, Thomas, A., Patrick Min, and Ingrid Carlbom. Real-time Acoustic Modeling for Distributed Virtual Environments. SIGGRAPH, pages , Funkhouser, Thomas, A., Ingrid Carlbom, Gary Elko, Gopal Pingali, and Mohan Sondhi. A Beam Tracing Approach to Acoustic Modeling for Interactive Virtual Environments. Funkhouser, Thomas, A., Ingrid Carlbom, Gary Elko, Gopal Pingali, and Mohan Sondhi. Interactive Acoustic Modeling of Complex Environments. Acoustical Society of America, Min, Patrick, and Thomas A. Funkhouser. Priority-Driven Acoustic Modeling for Virtual Environments. EUROGRAPHICS, Tsingos, Nicolas, Thomas A. Funkhouser, Addy Ngan, and Ingrid Carlbom. Modeling Acoustics in Virtual Environments Using the Uniform Theory of Diffraction. Hull, Joseph. Surround Sound Past, Present, and Future. Dolby Laboratories Inc. Suen, An-Nan, Jhing-Fa Wang, and Jia-Ching Wang. VLSI Implementation of 3-D Sound Generator. IEEE Transactions on Consumer Electronics, pages , 1997.
Real-Time Auralization of Sound in Virtual 3D Environments by Scott McDermott