1. New MIDI-Based Game Audio Techniques – Grigg / Savell Chris Grigg Beatnik Inc. Chairman, MMA Tech Board chrisg @ beatnik.com Tom Savell Creative Labs – Advanced Technology Center MMA Tech Board tcs @ atc.creative.com New MIDI-Based Game Audio Techniques

2. New MIDI-Based Game Audio Techniques – Grigg / Savell What’s MIDI? The Technology The Standards Orgs The New MIDI Standards 1.iXMF: Interactive XMF 2.3D MIDI Controllers 3.High Definition Protocol for MIDI Devices Questions Session Overview

3. New MIDI-Based Game Audio Techniques – Grigg / Savell What’s MIDI?

4. New MIDI-Based Game Audio Techniques – Grigg / Savell MIDI 1.0 over 5-pin DIN The Underlying Models The Recommended Practices GM, Show Control, MTC, MMC, GM2, SP-MIDI, etc. The File Formats SMF, Extended RMID, XMF, Mobile XMF Support in Computer OS’s Mac OS, Windows, BeOS, Linux etc. Support in Other Media Standards MPEG-4 etc. Support in Audio Plug-In API’s VST, DX, AU etc. Support in Mobile Phones 3GPP etc. MIDI: The Technology

5. New MIDI-Based Game Audio Techniques – Grigg / Savell MMA: MIDI Manufacturers Association (world except Japan) Tech Board Working Groups: Mobile WGXMF WG DLS WGTransport Layer WG High-Def WG IASIG: Interactive Audio SIG AMEI: Assoc. of Musical Electronic Industry (Japan only) MIDI: The Standards Orgs

6. New MIDI-Based Game Audio Techniques – Grigg / Savell The New MIDI Standards

7. New MIDI-Based Game Audio Techniques – Grigg / Savell iXMF: Interactive XMF 1 iXMF: Interactive XMF

8. New MIDI-Based Game Audio Techniques – Grigg / Savell First open, non-proprietary standard for a file format for interactive audio content Uses MMA’s XMF container technology Allows game developers to more easily create cross-platform titles Makes it easier for audio professionals to develop interactive content Developed by the iXMF-WG inside IASIG V1.0 spec about to be published Prominent early adopter iXMF: Overview

9. New MIDI-Based Game Audio Techniques – Grigg / Savell iXMF: Motivation (1) There Is No Standard Build Proprietary Tool Limited Time & $ for Tool Limited Tool Quality Harder to Make Content Limited Content Quality :-( Do It Right Next Time What’s the Industry Standard? Break the ‘Depressing Circle’!

10. New MIDI-Based Game Audio Techniques – Grigg / Savell Build Proprietary Tool Limited Time & $ Limited Tool Quality Harder to Make Content Limited Content Quality :-( Do It Right Next Time What’s the Industry Standard? iXMF: Motivation (1) There Is No Standard

11. New MIDI-Based Game Audio Techniques – Grigg / Savell Many companies continually re-creating the wheel – in-house / proprietary Much of what they do is the same – only the format is different This is redundant, parallel work –> no synergies User base (+budget) is very small, so tool quality improves slowly (at best) Tool quality tends to limit IA content quality IA work done on one platform doesn’t port to any other platform --> Re-do for every platform --> IA tends to be seen as cost-prohibitive --> Less IA gets used iXMF: Motivation (2)

12. New MIDI-Based Game Audio Techniques – Grigg / Savell A Project BBQ idea An IASIG Working Group Linda Law is Chair Chris Grigg is Technical Lead Long Requirements phase Draft Specification Public Review period ended in May 2008 Final edits to v1.0 spec under way now Release of v1.0 spec expected soon (October?) iXMF: Process (Standardization)

13. New MIDI-Based Game Audio Techniques – Grigg / Savell The iXMF File Self-Contained IA Content Module Includes both Media and Adaptive Behaviors iXMF: Technical Design (1)

14. New MIDI-Based Game Audio Techniques – Grigg / Savell Runtime Architecture iXMF File feeds IA Engine iXMF: Technical Design (2) Platform Provided Middleware Game

15. New MIDI-Based Game Audio Techniques – Grigg / Savell Cues – soundtrack element objects iXMF file contains zero or more Cues Cue = one soundtrack element – a piece of music, a sfx, or a change to music/sfx Game requests sound/music element by starting a Cue: by name: startCue( “mainTitlesMusic” ); by number: startCue( kMainTitlesMusic ); In iXMF file, Cue contains (or references) : zero or more media Chunks some Scripts iXMF: Technical Design (3)

16. New MIDI-Based Game Audio Techniques – Grigg / Savell Scripts – methods / events of Cues Stored in iXMF file in portable binary format Variables – iXMF Global / Cue class / Cue instance Expressions – all the C operators User Subroutines – call another user Script Built-In Statements Generic programming stuff Specialized IA primitives, to minimize the need for programming Extension Statements No user-defined Statements! – Keep it simple Tools can hide Scripts behind GUI iXMF: Technical Design (4)

17. New MIDI-Based Game Audio Techniques – Grigg / Savell Automatic Behaviors (“Engine Model”) Script hooks: IxmfFileLoad, IxmfFileUnload CueStart, CueStop, CueCancel CommitPoint, ChangeOver Markers can call Scripts NextChunk chaining Mix Transitions – crossfade, sync points/rules SyncGroups MixGroups Media LRU tracking Scripts can invoke game Callbacks Game can read/write iXMF variables iXMF: Technical Design (5)

18. New MIDI-Based Game Audio Techniques – Grigg / Savell Custom Extensibility: Add Script commands “Opcode Spaces” Add custom media formats: Platform-specific audio formats Generative music Even video, graphics Extension Area in every iXMF data structure Community Extensibility: Future Profiles of the v1.0 standard Future versions of the iXMF standard iXMF: Technical Design (6)

19. New MIDI-Based Game Audio Techniques – Grigg / Savell Portability Cues & Scripts will work properly on any iXMF player IF: The media formats used are supported AND ( No extensions are used by content OR Any extensions used are supported OR Any extensions used aren’t critical ) Profiles can manage portability expectations Content is often mastered for specific platforms – portability not an issue there A high-res iXMF “source” file can give authoring-time portability Export to platform-specific runtime formats iXMF: Technical Design (7)

20. New MIDI-Based Game Audio Techniques – Grigg / Savell iXMF is non-proprietary iXMF is royalty-free to use or implement So: Expect multiple runtime implementations, platforms, authoring tools --> Ecosystem Commercial --> Competition --> Improvement Expect interoperability for base features / profiles Don’t expect custom extensions to be portable Look for integration with game dev / media mgmt databases Hope that the IA community can start speaking a common language Expect vendors & IASIG to be responsive Expect IA to become the norm, not exotic iXMF: The Ecosystem

21. New MIDI-Based Game Audio Techniques – Grigg / Savell V1.0 Spec releasing soon. Then: New Versions New Features New Media Types Profiles for specialization as needed – e.g.: Advanced Games Simple/Casual Games Mobile Phones/Games Musical Instruments DJ Applications Location-Based Entertainment Little-Endian Join the iXMF Working Group to participate! iXMF: The Future

22. New MIDI-Based Game Audio Techniques – Grigg / Savell 3D MIDI Controllers 2 3D MIDI Controllers

23. New MIDI-Based Game Audio Techniques – Grigg / Savell 3D MIDI: Overview  Provides set of controllers for independent positioning on each MIDI channel  Uses spherical coordinate system: azimuth angle, elevation angle, and distance  Seamless integration with legacy pan control  Egocentric approach describes head-related 3D sources  Well-defined standardized behavior  Extensible through MMA standardization process  V1.0 to be published soon.

24. New MIDI-Based Game Audio Techniques – Grigg / Savell 3D MIDI: Position Controllers  Registered Parameter Number (RPN) Bank 0x3D  Azimuth and Elevation Angular Resolution of 0.02 degrees from -180 to +179.98 degrees  Distance represented as ratio of distance to a maximum distance: e.g. 0.000061 is 6.1 centimeters if max. distance is 1km.

25. New MIDI-Based Game Audio Techniques – Grigg / Savell 3D MIDI: Pan Spread and Roll Angle  Pan Spread defines the 3D angular distance of the MIDI stereo field represented by the legacy pan controller  Roll Angle defines the angle relative to horizontal of the MIDI stereo field represented by the legacy pan controller

26. New MIDI-Based Game Audio Techniques – Grigg / Savell 3D MIDI: Position Model

27. New MIDI-Based Game Audio Techniques – Grigg / Savell 3D MIDI: Gain Parameters  Gain parameters provide content author control  Gain  Independent of CC#7/11  Gain at max distance  Gain when distance ratio is 1.0  Reference distance  No additional gain below this distance ratio

28. New MIDI-Based Game Audio Techniques – Grigg / Savell 3D MIDI: Distance Model attenuation distance 0 dB max_attenuation reference_distancemax_distance

29. New MIDI-Based Game Audio Techniques – Grigg / Savell 3D MIDI: Ex. Distance of a Fly  The buzz of a fly would typically not be heard beyond 10 meters away  maximum_distance = 10  Sounds significantly louder a few centimeters away from your ear  reference_distance_ratio = 0.01 -- between 0 and 10 cm, the fly sound is not attenuated

30. New MIDI-Based Game Audio Techniques – Grigg / Savell 3D MIDI: Future Enhancements  Reverberation Controls  Source / Reverb mix dependency on distance.  Environmental effects (e.g. occlusion)  Could be made compatible with IA-SIG I3DL2 guideline  Subject to MMA Standardization Process

31. New MIDI-Based Game Audio Techniques – Grigg / Savell High Definition Protocol for MIDI Devices 3 HD Protocol for MIDI Devices

32. New MIDI-Based Game Audio Techniques – Grigg / Savell Whole new generation of the basic MIDI message protocol Address shortcomings of the MIDI 1.0 protocol (1983) Grow the worldwide MIDI markets by opening up new functionality, new musical expressiveness HD work started in 2006 by MMA Tech Board Standardization work still under way Overview Only! All Technical Details Subject to Change! HD-PMD: Introduction

33. New MIDI-Based Game Audio Techniques – Grigg / Savell MIDI 1.0 Technology Needs an Update: MIDI 1.0 first released in 1983 HW transport slow / Timing not tight Complicated msg structure / Parsing: Many multi-message functions Message size varies Byte aligned Hardware orientation --> Hard to write SW for Advanced features difficult to access Expansion room exhausted long ago Desired New Functionality Includes: Better support for non-keyboard instrument models (strings, voices, horns, winds etc.) Better support for non-western music Better support for instruments with continuous pitch / very fine pitch nuance HD-PMD: Motivation

34. New MIDI-Based Game Audio Techniques – Grigg / Savell MMA Working Group in US/Europe AMEI Working Group in Japan Joint & Parallel technical work Weekly WG teleconferences WG email reflector MMA/AMEI coordination via email MMA/AMEI Joint Face-to-Face meetings in LA / SF / Tokyo 2-3x per year HD-PMD: Standards Process

35. New MIDI-Based Game Audio Techniques – Grigg / Savell Transports: An ongoing discussion – Traditional wired transports Software-only transports: Network/comm protocols Plug-In APIs OS MIDI services APIs Messages Significantly reconfigured Simplicity Future extensibility New musical functionality HD-PMD: Technical Design

36. New MIDI-Based Game Audio Techniques – Grigg / Savell Uniform Packet 64/128 bits (8/16 bytes) Simplifies transports - Memory efficient (MIDI 1.0 has 1, 2, 3, or N-byte messages) Message = 1 or More Packets Single-Packet Messages (SPM) Multi-Packet Messages (MPM) Fixed # of Packets Variable # of Packets (VLM) HD-PMD: Messages

37. New MIDI-Based Game Audio Techniques – Grigg / Savell MIDI 1.0 Totally full. Every “Status Nybble” either in use, or else officially deprecated Extensions very cumbersome HD is shown at right Lots of expansion room Especially with sub-commands HD-PMD: Opcode Spaces Compared

38. New MIDI-Based Game Audio Techniques – Grigg / Savell Notes: More Velocity resolution (16-bit – 512x MIDI 1.0 depth) Precise Pitch option (1/256 semitone resolution) Trigger Index option (like MIDI 1.0 Note Number) Controllers: More Value resolution (16/32-bit – 2-4x MIDI 1.0 depth) More Controller Numbers available (32x MIDI 1.0 space) No more MSB / LSB / Data Entry! CC is a unified message, including Bank number RPN now just another controller number NRPN now a unified message – all MIDI 1.0 numbers available Per-Channel and Per-Note Controllers, RPNs, and NRPNs Addressing: More Channels (7 bits - 8x MIDI 1.0 space) Groups concept - Add’l dimension of addressing (5 bits - effectively 256x MIDI 1.0 channel space) HD-PMD: New Functionality (Proposed – Under Discussion)

39. New MIDI-Based Game Audio Techniques – Grigg / Savell More joint MMA/AMEI Meetings Draft v1.0 Specification (English) is in progress Experimental implementations of candidate transports ongoing Decide on v1.0 Spec(s) release date(s) Decide on possible additional Recommended Practices for HD After v1.0 Spec Release: Continuing process to standardize new features… forever… and ever… and ever… HD-PMD: What’s Next?

40. New MIDI-Based Game Audio Techniques – Grigg / Savell Questions?

41. New MIDI-Based Game Audio Techniques – Grigg / Savell YES you can join the iXMF Working Group… YES you can get a copy of the v1.0 iXMF Spec soon… …at www.iasig.org YES you can get a copy of the 3D MIDI Spec soon… YES if you are commercially involved in designing or producing MIDI HW or SW you can join the MMA and the MIDI High Def Working Group… …at www.midi.org Answers!

42. New MIDI-Based Game Audio Techniques – Grigg / Savell Chris Grigg Beatnik Inc. Chairman, MMA Tech Board chrisg @ beatnik.com Tom Savell Creative Labs – Advanced Technology Center MMA Tech Board tcs @ atc.creative.com (You just sat through) New MIDI-Based Game Audio Techniques