1. Physically Based Sound COMP259Nikunj Raghuvanshi

2. Overview Background FEM Simulation Modal Synthesis (FoleyAutomatic) Comparison/Conclusions

3. Motivation Sounds could in-principle be produced automatically, just like graphics: Sound Rendering Sound Rendering has not received much research effort Main Goal: Automatic generation of non-music, non-dialogue sound

4. Sound Production Today Movies: Foley Artists http://www.marblehead.net/foley/index.html Games: Anyone noticed the huge sound directory in Unreal Tournament?

5. PBS: Sound Production in Nature Collisions/Other interactions lead to surface vibrations Vibrations create pressure waves in air Pressure waves sensed by ear Surface VibrationPressure WaveEar VibrationPropagationPerception

6. Main Aims of PBS Physics simulator gives contact/collision information Assign material properties for sound, Wood, concrete, metal etc. Sound simulator generates sound using this data (in real time?)

7. Challenges Sound must be produced at a minimum of ~44,000 Hz Extremely High Temporal Resolution (timesteps in the range of 10 -6 -10 -8 s) Stiffness of underlying systems (eg. Metallic sounds. K/m~=10 8 ) Stability may require even smaller timesteps

8. Two Approaches FEM deformable simulation O'Brien, J. F. et. al., “Synthesizing Sounds from Physically Based Motion.” SIGGRAPH 2001. FoleyAutomatic (Modal Synthesis) Kees van den Doel et. Al., “FoleyAutomatic: Physically- based Sound Effects for Interactive Simulation and Animation.” SIGGRAPH 2001.

9. Main ideas Deformable Simulation (arguably) much more “physically based” Foley Automatic: Additive Synthesis Component Sinusoids Sound Signal

10. Overview Background FEM Simulation Modal Synthesis (FoleyAutomatic) Comparison/Conclusions

11. Simulation Requirements Temporal Resolution Simulate Vibration as well as Propagation Vibration Modeling: Deformable Model for Objects Propagation Modeling: Explicit Surface Representation Physical/Perceptual Realism

12. System Structure

13. Vibration Modelling FEM with Tetrahedral Elements Linear Basis Functions, green’s strain Explicit Time Integration Typically #nodes = 500, #elements = 1500, dt = 10 -6 -10 -7 s

14. Sound Propagation Modelling Fluid Dynamic FEM simulation of surrounding air? Very expensive. Instead… Employ Huygen’s Principle: Pressure Wave may be seen as sum of pressure wavelets Receiver Pressure Wave Pressure “Wavelets”

15. Acoustic Impedance of Air Surface Vibrations and Sound Pressure contribution of a patch, Velocity Density of Air Sound Propagation Speed in Air Unit Normal

16. Surface Vibrations and Sound Approximate differential elements with surface triangles Apply band pass filters:  Low pass: windowed sinc filter  High pass: DC blocking filter Result: Pressure known for all surface triangles

17. Putting it all together Pressure/Signal at Receiver Filtered Average Pressure Area of Triangle Visibility Term Approximation of Beam Pattern Distance Falloff Receiver Vibration

18. Propagation Delay Accumulation Buffer Receiver d1d1 d2d2 Source t=0 t 1 = d 1 /c t 2 = d 2 /c 1 2 Receiver Distance from Source Sound Propagation Speed

19. Results: Capabilities General models Generated sounds are accurate Stereo Sound Doppler’s Effect

20. Demo

21. Results: Accuracy

22. Results: Speed Scene TimeStep(s) Nodes/Elems Time/Audio Time Bowl 10 -6 387/1081 91.3/4.01 mins Clamped Bar 10 -7 125/265 240.4/1.26 mins Vibraphone 10 -7 539/1484 1309.7/5.31 mins (~1 day) Timings on a 350MHz SGI Origin MIPS R12K processor

23. Overview Background FEM Simulation Modal Synthesis (FoleyAutomatic) Comparison/Conclusions

24. Features Modal resonance model of solids Location dependent sounds Impact, slide, roll excitation models Real-time, low latency Easy integration with simulation/animation Practical Do not model propagation of sound from source to receiver

25. Synthesis Method Force Vibration Emission PropagationListenerSpeakers Sound Samples User

26. Vibration Surface u(x,t) of body responds to external contact force F(x,t) u(x,t) F(x,t) Strain Functional Speed of Sound Under suitable boundary conditions, the solution to the PDE is a sum of sinusoids

27. Emission Sound pressure s(t) linear functional L of surface vibration u(x,t) u(x,t) L s(t) Note that propagation is not modeled in above

28. The Modal Synthesis Model u(x,t) F(p,t) L s(t) Impulse response/modal model “The response u(x,t) of an arbitrary solid object to an external force can be described as a weighted sum of damped sinusoids” Since L is linear, it implies at s(t) must be a sum of damped sinusoids too

29. Example: A 1D string 1 st Mode2 nd Mode Frequency = f 0 …Higher modes Frequency = f 1 = 2*f 0 Frequency = f k = k*f 0 Main Idea: Sum contributions of all the modes The point of impact decides the proportions in which the modes are to be mixed: a k. Therefore, a k is a function of p, the point of impact The frequencies and damping parameters are a property of the object, and independent of how the object is hit + +...+ a0a0 a1a1 akak

30. The Modal Synthesis Model u(x,t) F(p,t) L s(t) Impulse response, modal model Parameters measured experimentally K th mode: Gain Factor Point Damping Vibration of impact Term Frequency

31. Force Modeling Impact Sliding Rolling Wavetable Stochastic At runtime: Find gain parameters given the location, strength and kind of force. Synthesize sound from previous equation.

32. Impact Forces Duration: hardness (T) Magnitude: energy transfer (w) Multiple micro-collisions Example:

33. Sliding/Scraping Micro-collisions lead to noisy audio-force

34. Sliding/Scraping Wavetable approach  Store force parameters  Modulate amplitude with energy transfer  Modulate rate with contact speed Synthesis Approach  Fractal noise represents roughness  Filter through reson filter  Resonance ~ contact speed  Width ~ randomness of surface

35. Rolling No relative surface motion Differences with sliding: Smoother: Use low pass More damping Harder to create Less understood Essential coupling?

36. Rolling: Smooth Surfaces  Polyhedral objects do not lead to smooth rolling forces  Instead use smooth surfaces directly

37. Rolling: Contact Evolution Evolve the contact in Reduced coordinates q = (u,v,s,t, ) q q q... c(u,v) d(s,t)

38. Rolling: Contact Evolution Piecewise parametric surfaces, loop subdivision surfaces Explicit integration, no stabilization Multiple contacts and conforming contacts are not handled Used only when multiple contacts in close spatio-temporal proximity

39. Demo

40. Dynamic Forces Contact force Rolling speed Slipping speed Impulses …and locations Pebble-in-Wok Demo

41. Results 0.1% CPU time per mode Graceful degradation of quality The bell demo is interactive Uses a PHANToM for interaction Authors do not report any real timings State that “sound quality” is perception- based and has no metric as of now

42. Overview Background FEM Simulation Modal Synthesis (FoleyAutomatic) Comparison/Conclusions

43. Discussion FEM: Physically Rigorous and General Too slow for interactive applications Doesn’t scale well Inappropriate to apply a 30fps technique to 44000fps? Maybe too general for the problem domain?

44. Discussion Modal model exploits the vibrational nature Higher Efficiency But, not rigorously physically based Finding the parameters requires experimentation and “earballing” No rigorous correlation between physical and perceptual parameters

45. Discussion For Realtime: Need for a technique to cover the middle ground Extracting modal parameters in general requires solving PDEs Not possible to do in an automated manner Approximate modal parameters and then use modal synthesis?

46. Conclusion PBS involves orders of magnitude smaller temporal and spatial scales Research is sparse, problems are dense Main contributions of the two papers besides vibration modeling:  FEM: Efficient modeling of sound propagation  FoleyAutomatic: Efficient, Approximate models to handle surface properties and contact forces

47. References O'Brien, J. F., Cook, P. R., Essl G., "Synthesizing Sounds from Physically Based Motion." The proceedings of ACM SIGGRAPH 2001, Los Angeles, California, August 11-17, pp. 529- 536. Kees van den Doel, Paul G. Kry and Dinesh K. Pai, “FoleyAutomatic: Physically-based Sound Effects for Interactive Simulation and Animation” Computer Graphics (ACM SIGGRAPH 01 Conference Proceedings), pp. 537-544, 2001.

48. Acknowledgements Some images were taken from the referred papers and the corresponding SIGGRAPH slides