1. Music Processing Roger B. Dannenberg

2. Overview  Music Representation  MIDI and Synthesizers  Synthesis Techniques  Music Understanding

3. Music Representation  Acoustic Level: sound, samples, spectra  Performance Information: timing, parameters  Notation Information: parts, clefs, stem direction  Compositional Structure: notes, chords, symbolic structure

4. Performance Information  MIDI bandwidth is 3KB/s, or 180KB/min  More typical: 3KB/minute, 180KB/hour Complete Scott Joplin: 1MB Output of 50 Composers (400 days of music): 500MB (1 CD-ROM)  Synthesis of acoustic instruments is a problem

5. Music Notation  Compact, symbolic representation  Does not capture performance information  Expressive “performance” not fully automated

6. Compositional Structure  Example: Nyquist (free software!) (defun melody1 () (seq (stretch q (note a4) (note b4) (note cs5) (note d5)))) (defun counterpoint () …) (defun composition () (sim (melody1) (counterpoint))) (play (transpose 4 (composition)))

7. Overview  Music Representation  MIDI and Synthesizers  Synthesis Techniques  Music Understanding

8. MIDI: Musical Instrument Digital Interface  Musical Performance Information: Piano Keyboard key presses and releases “instrument” selection (by number) sustain pedal, switches continuous controls: volume pedal, pitch bend, aftertouch very compact (human gesture < 100Hz bandwidth)

9. MIDI (cont’d)  Point-to-point connections: MIDI IN, OUT, THRU Channels  No time stamps (almost) everything happens in real time  Asynchronous serial, 8-bit bytes+start+stop bits, 31.25K baud = 1MHz/32

10. MIDI Message Formats 8 chkey#vel Key Up 9 chkey#vel Key Down Program Change Polyphonic Aftertouch System Exclusive A chpresskey# C chindex# B chctrl#value Control Change Channel Aftertouch D chpress E chlo 7hi 7 Pitch Bend F 0 F E … DATA …

11. Standard MIDI Files  Key point: Must encode timing information  =1 or more, =, = midi data or, = FF =1 or more, =, = midi data or, = FF Delta times use variable length encoding, omit for zero. Interleave time differences with MIDI data...

12. Standard MIDI Files (cont’d)  MThd  MTrk  MThd  MTrk header info track data: each with 16 channels

13. Overview  Music Representation  MIDI and Synthesizers  Synthesis Techniques  Music Understanding

14. Music Synthesis Introduction  Primary issue is control No control  Digital Audio (start, stop,...) Complete control  Digital Audio (S[0], S[1], S[2],... ) Parametric control  Synthesis

15. Music Synthesis Introduction (cont’d)  What parameters? pitch loudness timbre (e.g. which instrument) articulation, expression, vibrato, etc. spatial effects (e.g. reverberation)  Why synthesize? high-level representation provides precision of specification and supports interactivity

16. Additive Synthesis  amplitude A[i] and frequency  [i] specified for each partial (sinusoidal component)  potentially 2n more control samples than signal samples!

17. Additive Synthesis (cont’d)  often use piece-wise linear control envelopes to save space  still difficult to control because of so many parameters  and parameters do not match perceptual attributes

18. Table-Lookup Oscillators  If signal is periodic, store one period  Control parameters: pitch, amplitude, waveform Phase + Frequency Amplitude x n Efficient, but... n Spectrum is static n Efficient, but... n Spectrum is static (Note that phase and frequency are fixed point or floating point numbers)

19. FM Synthesis  Usually use sinusoids  “carrier” and “modulator” are both at audio frequencies  If frequencies are simple ratio ( R ), output spectrum is periodic  Output varies from sinusoid to complex signal as MOD increases A F AMPL out = AMPL· sin(2  ·FREQ· t + MOD sin(2  R ·FREQ· t )) + FREQMOD

20. FM Synthesis (cont’d)  Interesting sounds,  Time-varying spectra, and...  Low computation requirements  Often uses more than 2 oscillators … but …  Hard to recreate a specific waveform  No successful analysis procedure

21.  Samplers store waveforms for playback  Sounds are “looped” to extend duration  Spectrum is static (as in table- lookup), so: different samples are used for different pitches simple effects are added: filter, vibrato, amplitude envelope attack portion, where spectrum changes fastest, added to front Sample-based Synthesis AttackLoopLoop again...

22. Physical Models  Additive, FM, and sampling: more-or-less perception-based.  Physical Modeling is source-based: compute the wave equation, simulate attached reeds, bows, etc.  Example: ReedBoreBell

23. Physical Models (cont’d)  Difficult to control, and...  Can be very computationally intensive … but...  Produce “characteristic” acoustic sounds.

24. Overview  Music Representation  MIDI and Synthesizers  Synthesis Techniques  Music Understanding

25. Music Understanding  Introduction  Score Following, Computer Accompaniment  Interactive Performance  Style Recognition  Conclusions

26. What Does Music Mean?  Emotion  Formal structures  Abstract  Physical

27. What is Music Understanding?  Translation?  Recognition? (Of what?)  Parsing?  Pattern forming?  Recognition of themes?  Music Understanding is the recognition of pattern and structure in music.

28. Computer Accompaniment Performance Input Processing Input Processing Matching Score for Performer Score for Accompaniment Performance Accompaniment Performance Music Synthesis Music Synthesis Accompaniment (see Dannenberg ‘84)

29. Interactive Performance  Traditional Western Composition is carefully composed, but the result is static. Composer is central figure.  Jazz and other improvisations are not carefully composed (typically small structures), but the result is dynamic and spontaneous. Performer is central figure.  Can we integrate these two?

30. A New Approach to Music Making  Computers let us put compositional theories into programs.  Music understanding helps us tie programs to live performance.  Result can be carefully composed and structured: Composer-oriented.  At the same time, result can be spontaneous: Performer- oriented.

31. Style Recognition  Everyone recognizes musical style: “I don’t know anything about music, but I know what I like”  What makes something Lyrical? Syncopated?

32. Experimental Setup ? Lyrical Pointilistic Syncopated Frantic

33. Music Understanding Conclusions  Music Understanding: the recognition of pattern or structure in music.  Music Understanding is necessary for high-level interfaces between musicians and computers.

34. Music Summary  Rich in representations  Different representations support different tasks  Active research in: Synthesis Understanding  Hardware to Software