SE in RT Audio Applications Bert Schiettecatte Promotor: Prof. D. Vermeir Co-promotor: Prof. R. Lauwereins Advisors: S. Himpe & T. Vander Aa

Introduction Sound: infinite discrete sequence of normalized numbers = ‘samples’ RT Audio Synthesis: real-time (RT) computation and playback of synthetic sound Synthetic Sound: sound generated using a mathematical model RT Deadlines: buffer scheduling, event handling

Goals & Limitations Not: –Detailed treatment of digital signal processing (DSP) arithmetic for audio processing –Experiments on the performance of OO in RT applications Instead: –Realistic exposure to application of some SE techniques in a RT application –Roadmap for building a RT audio synthesizer

Case Study Real-time audio synthesizer on handheld device: StrongARM running PocketPC To be used by musicians on the road Goals: –Feasibility –Exposure to realistic RT application –Exposure to DSP –SE principles realistic in RT applications

Implementation Trade-offs: –Floating-point VS fixed-point –Double buffering, tripple buffering or ring buffering of sound –Static VS dynamic networks –OO VS best-fit implementation

Implementation Strategy: bench & correctness of 1.Arithmetic 2.Synthesis Algorithm 3.Buffer Scheduling & OS Timing 4.Event Scheduling

Fixed VS float arithmetic Arithmetic: major impact on overall performance & sound quality (quantization noise) Benchmark for comparing floating-point / fixed-point performance: LINPACK Case study HW: factor 10 speedup (assembly-level analysis reveals float arithmetic implementation: library)

Fixed-point arithmetic and GP Floating-point arithmetic: sometimes regarded as ‘unlimited range’ by developers Fixed-point arithmetic: limited per definition, requires detailed range analysis in some algorithms, resulting in confusing code Solution: use generative programming to derive best precision for a variable Generalized: explicit type for a variable can be substituted for a set of constraints

Synthesis Algorithm Additive synthesis: sum (harmonically related) weighted sine waves Envelope: piecewise linear function changing weight over time Requirements: –Oscillator: efficient sine algorithm (e.g. CORDIC) –Envelopes: linear interpolation

Synthesis Algorithm Use fixed-point arithmetic to implement algorithm Dynamic network: runtime sequencing of DSP blocks, buffer granularity Static network: compile time sequencing of DSP blocks, sample granularity Here: static network because of fixed-point arithmetic & inlining

Buffer Scheduling: OS API In general: 1.Check device capabilities 2.Open device 3.Prepare buffer 4.Queue buffer in the OS for playback 5.Wait until playback done 6.Goto 3 7.Unprepare buffer 8.Close device

Buffer Scheduling Algorithms: –Naive –Double/Triple buffering –Ring buffering

Buffer Scheduling: Naive Algorithm: 1.Prepare buffer 2.Queue buffer in the OS for playback 3.Wait until OS signals playback done 4.Goto 1 Problems: –sound pauses (prepare + queue) –OS event is overhead

Buffer Scheduling: Double/Tripple Algorithm: 1.Prepare buffer 1 2.Queue buffer 1 in the OS for playback 3.Wait until OS signals buffer 2 done 4.Prepare buffer 2 5.Queue buffer 2 6.Wait until OS signals buffer 1 done 7.Goto 1 Problems: –OS event overhead

Buffer Scheduling: Ring Buffer Algorithm: 1.Fill ½ & queue buffer for looping playback 2.Get playing position of the OS in buffer 3.Fill buffer from the last known rendering position up to the OS position 4.Update rendering position 5.Wait ¼ buffer size 6.Goto 2 Problems: –Tricky: sleep accuracy (WinCE 3.0: 1 to 2 ms VS Win98: 20ms!) –Choosing wait time is hard

Buffer Scheduling: Reality Scheduling: very hard –Requires inspiration and experimenting (e.g. buffer size problems in WinCE 3.0) –Buffer underrun detection/avoidance (e.g. MP3)

Event Scheduling Possibilities: –Seperate event thread updating settings –Handle events in the synthesizer class: keep a song pointer & update when necessary Seperate thread: requires additional CPU time for scheduling, synchronization In the synth class: better, keep a song pointer (in samples) and update settings based on the song’s tempo (e.g. 120 BPM = update every 0.5s * Hz = samples)

Event Scheduling Latency: the delay on events caused by the buffer scheduler Latency: –In seperate event thread: because of synchronization, at least 1 sound buffer –In synthesis loop: almost none, since events can be handled almost every sample

Implementation: Best-fit In general: mix paradigms to write clear code OO not always the best choice: depends on levels of re-use (e.g. OS timer) Machine-level inspection of code: crucial when using OO in RT applications OO not per definition a bad thing

Implementation: Object-orientedness Here: OO is the right paradigm when –It doesn’t keep the compiler from optimizing –It entirely disappears after compilation (e.g. inlining): no run-time OO –It simplifies code In general: OO languages without real ‘compile-time’ tend to have performance problems Meta-programming: welcome addition

Contribution This dissertation: one of the few detailed available roadmaps for RT audio synthesis implementation Not much new here (but still large amount of research, everything proprietary) Application of generative programming to audio processing and fixed-point arithmetic

Conclusion Implementing a RT audio synthesizer is hard Series of benchmarks & tests crucial OO design has to be applied with common sense and is not always the best choice Generative programming can be a powerful mechanism to abstract at little or no cost A meta-programming level adds expressive power Compiler features (inlining, templates, …) essential

Synthesis engine demonstration Sets up the additive synthesis engine, starts the sound buffer scheduler, loads events from a text file and feeds them to the additive synthesizer