1. 5 March 2005Cory McKay 1 Approaches to Overcoming Problems in Interactive Musical Performance Systems Cory McKay Music Technology Area Faculty of Music McGill University

2. 5 March 2005Cory McKay2 / 17 Topics Introduction to interactive performance systems Computer that performs with human performer(s) Problems that have limited the use of such systems General approaches that address these problems Proposed long-term solution: Standardized control parameter extractor Reliance only on parameters extracted from performers’ audio signal

3. 5 March 2005Cory McKay3 / 17 Interactive performance systems Software / hardware that reacts musically to input from performer(s) Can perform with: A human soloist A human accompanist An ensemble of human performers Other performance systems

4. 5 March 2005Cory McKay4 / 17 Possible roles Use score-following to dynamically timewarp pre- set scores Initiate pre-recorded events triggered by performer actions Basic mappings involving pre-set heuristics e.g. harmonizations, dynamic sound processing, etc. Use of advanced expert systems or non- deterministic artificial intelligence to model sophisticated musical knowledge e.g. “jamming systems”

5. 5 March 2005Cory McKay5 / 17 Advantages Can perform actions that are too technically difficult for humans Take advantage of extended sonic capabilities of computers e.g. precise control of sound synthesis Capitalize on essential expressivity and live dynamism of human performer(s) Useful in an educational context

6. 5 March 2005Cory McKay6 / 17 Problems with interactive systems Lack of reliability during performance Pieces can be too difficult or unintuitive for performers Software can be unstable or too strenuous on hardware Limited longevity Reliance on software and hardware with limited life span Poor interface with little or no documentation Few performances without composer’s direct involvement Hardware expensive and difficult to acquire and transport Systems difficult to set up Proper documentation and instructions rarely provided Lack of effort to make pieces attractive to performers and audiences

7. 5 March 2005Cory McKay7 / 17 Solutions relating to performers Performers should not be expected to perform tasks that are alien to their training Limitations of performer bandwidth must be considered A clear explanation must be made of specifically how the system will respond to their actions Intuitive feedback must be provided Auditory, visual, haptic Performers must be given sufficient time to learn how to use the system Includes significant rehearsal time with functional system

8. 5 March 2005Cory McKay8 / 17 Solutions relating to system robustness Software should be well tested Creative ambition should be tempered by technical realism Simple equipment should be used whenever possible The smaller the amount of electronics the better Systems must be error tolerant Not fooled by minor mistakes and should not fail entirely if major mistakes occur Software interface should be simple Basic overrides should be available

9. 5 March 2005Cory McKay9 / 17 Solutions relating to dissemination and longevity Only commonly available, cheap and easy-to-transport equipment should be used Well-known software that is likely to have a long life-span should be used whenever possible Applies to operating system and implementation language Any need for proprietary or unique hardware or software should be avoided unless absolutely necessary Systems should require only minimal supervision during performance What supervision is needed should involve a simple, intuitive and well-documented interface

10. 5 March 2005Cory McKay10 / 17 Advantages of standardization Standardized performance parameter extractor would be beneficial: Composers would not need to implement their own systems Professional design would cause system to be more robust Concert venues could be pre-equipped with system Technicians at venues would be accustomed to the system Addresses problems with reliability and dissemination of compositions as well as robustness Mapping system should not be standardized Would compromise creative freedom Could use a client-sever architecture to facilitate porting of parameter extraction system to mapping system

11. 5 March 2005Cory McKay11 / 17 Advantages of exclusive use of parameters derived from audio Performers do not need to spend time learning modified instruments or new performance gestures Increased familiarity makes performers less likely to make mistakes Miniaturization and wireless technology can make microphones essentially invisible to performers Rich sound of original instruments is available as well as information previously provided only by MIDI instruments Only hardware needed is microphones and computer Unlikely to become obsolete anytime soon Reliable technology Cheap, portable and easy to set up Could mic audio output of hyper-instruments

12. 5 March 2005Cory McKay12 / 17 Parameters to extract I Want a general-purpose system A large number of parameters provide a diverse palette Too many parameters can be problematic Compromises simplicity Increases latency Implement only the most clearly useful parameters Should be sufficient for most pieces Additional parameters can be added as plug-ins

13. 5 March 2005Cory McKay13 / 17 Parameters to extract II Primary output is MIDI transcription: Note Ons and Note Offs Time, pitch, velocity Stream of onsets detected Frequencies take longer to detect than onsets Note Ons therefore have a greater latency than onset detections Stream of detected frequency values Needed for microtonal pieces and recognition of techniques such as glissandi Overall signal amplitude Gives an indication of overall dynamics

14. 5 March 2005Cory McKay14 / 17 Parameters to extract III Additional features related to timbre This potentially very rich source of information is unavailable from MIDI instruments e.g. relative amounts of energy in different spectral regions More sophisticated features can be extracted from these basic parameters Chord classification, beat tracking, note density, etc.

15. 5 March 2005Cory McKay15 / 17 Inaudible cues Performers often communicate with each other inaudibly e.g. visual gestures Could also be useful to communicate with an interactive system inaudibly e.g. manual state switching if a cue is missed Should avoid sophisticated, expensive or unusual equipment or anything requiring specialized training Cameras and a standardized image processing system could provide a good long-term solution Basic footswitches

16. 5 March 2005Cory McKay16 / 17 Difficulties with real-time transcription Monophonic transcription is known to work well Can still have problems with latency Accurate polyphonic transcription still an unsolved problem Latency an even greater problem here Can use localized microphones in ensembles to help with polyphonic transcription Turns hard polyphonic transcription problem into multiple easier monophonic transcription problems Does not solve problem of polyphonic instruments Piano, guitar, harp, etc. Advances in transcription technology will make real-time transcription more feasible as time goes by

17. 5 March 2005Cory McKay17 / 17 Conclusions There are currently significant problems relating to longevity, dissemination and reliability A variety of fundamental guidelines have been presented to address these issues Promising long-term solution to these problems: A standardized real-time control parameter acquisition system relying only on auditory input