1. Review of MUSIC PERFORMANCE by Caroline Palmer Ann. Rev. Psychol. 1997, 48:115-38 Professor, Dept of Psychology Canada Research Chair Cognitive Neuropsychology of Performance McGill University, Montreal, Quebec, Canada Presented by Elaine Chew On January 11, 2006, as part of ISE 599: Topics in Engineering Approaches to Music Cognition - Computational Models of Expressive Performance

2. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance2 AGENDA  INTRODUCTION  INTERPRETATION  PLANNING  MOVEMENT

3. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance3 Forms of Performance  Sight-reading  Performing well-learned music from memory or notation  Improvising  Playing by ear

4. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance4 Serial Order and Timing Issues  Skilled serial action: speaking, typing, performing music  Activity must be centrally linked Little time for feedback for planning Can be performed w/o kinesthetic feedback  Accurate temporal control: rhythm Basis for dev models of timing mechanisms Consensus on requirements for accuracy

5. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance5 Purpose of psychological studies  Develop theories of performance mechanisms (cognitive/motor constraints)  Explain treatment of structural ambiguities  Understand relationship between performance and perception

6. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance6 Components of performance  Interprete piece conceptually  Retrieve musical structures and units from memory  Prepare for production  Transformed into appropriate movements

7. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance7 Methodological Issues  Wealth of data Separating signal from noise Focus on movement-based information  Judgement of representative piece Recognized level of performer expertise Large samples of data hard to find Rely on converging evidence from both small and large sample studies

8. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance8 Performance Expression  Variations in timing, intensity (dynamics), timbre, and pitch form the microstructure of a performance differentiate it from another of the same pc  Measurements deviation from fixed or regular values as notated in score Relative to performance itself, e.g. pattern of deviation with repect to a unit s.a. a phrase

9. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance9 AGENDA  INTRODUCTION  INTERPRETATION  PLANNING  MOVEMENT

10. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance10 System of Communication  Chain of events …

11. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance11 System of Communication  Composers code musical ideas in notation  Performers recode from notation to acoustical signal Includes performer’s conceptual interpretation of composition  Listeners recode from acoustical signal to ideas

12. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance12 Interpretation  Performer’s individualistic modeling of a piece according to their own ideas or musical intentions  In western music notation: Pitch and duration (clear) Intensity and tone quality (approx) Group boundaries, metrical levels higher than the bar, patterns of motion, tension, and relaxation (unspec, implicit)  Could explain inter- and intra-performer performances of the same piece

13. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance13 Role of Analysis  Every performance involves some kind of interpretation or analysis  Analysis offers explanations for the content of a composition as a Hierarchy of whole/part relations Linear course following harmonic tension Series of moods that result in unity of character  Analysis does not indicate how a performer actually produces a desired interpretation

14. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance14 Goal of Interpretation  Convey the meaning of the music Structure, emotion, and physical movement  Highlight particular structural content  Highlight particular emotional content

15. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance15 Highlighting structural content  Nakamura (1987): Compared musicians’ performances of baroque sonata with notated interpretations of dynamics Perceived dynamics matched intended fairly well, even when underlying acoustic changes were not identifiable  Palmer (1989): Compared pianists’ notated intepretations of phrase structure and melody with expressive timing patterns Melody lead and slowing of tempo at phrase boundaries observed Expressive timing patterns decr when attempting to play w/o interpretation, incr in exaggerated interp  Palmer (1988): Expressive timing patterns incr from novices to experts, during practice of unfamiliar piece, changed in diff interp by same perf

16. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance16 Implication of structural content interp  Palmer (1992): Pitch deletions tend to occur within phrases, and pitches tend to persevere at phrase boundaries Interpretations strengthen phrase boundaries relative to other locations  Palmer & van de Sande (1993, 1995): Melodic events are correctly retrieved and produced relative to nonmelodic events

17. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance17 Goal of Interpretation  Convey the meaning of the music Structure, emotion, and physical movement  Highlight particular structural content  Highlight particular emotional content

18. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance18 Highlighting emotional content  Langner (1953): Music shd sound the way moods feel  Gabrielsson (1995), G. & Juslin (1996): Compared performers’ interp of emotional content with their use of expression Happy/angry - faster, larger dynamic range Soft/sad - slower, smaller dynamic range  Ashkenfelt (1986): Similar results in tender/aggressive experiments  Schmalfeldt (1985), Shaffer (1995): Emotional content as part of narrative, dramatic char, thematic content, conceptions of large-scale structures

19. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance19 Role of experience  Musical experience enhances ability to use and identify interpretations  Nonmusicians can pick up interpretative aspects of performance Discern general differences among mechanical, expressive, exaggerated perf Can hear intended phrase structure Cannot always find melody interpretation  Sufficiency of expressive features to convey intepretations

20. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance20 AGENDA  INTRODUCTION  INTERPRETATION  PLANNING  MOVEMENT

21. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance21 Planning and Memory  Related to melodic, harmonic and diatonic structures Chord errors occur more in homophonic mus Single note errors more in polyphonic music Mistakes originate more from key of piece Mistakes tend to be of same chord type Child singing pitch errors tend to be harmonically related to intended events Pianists’ sight-reading errors in pcs with deliberate pitch alterations indicate tacit melodic/harmonic knowledge

22. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance22 Subsequence Partitioning  Partitioning into phrases Errors originate more from same phrase Interacting errors rarely crossed phrase boundaries (like in speech) Errors increased when melodic, metrical, rhythmic accents unaligned  Planning ahead Eye-hand span 7-8 events, or to phrase end Range of planning affected by serial & struct

23. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance23 Syntax of Musical Structure  Events at most salient levels are commonly emphasized in performance Tactus: foot tapping metric level Phrase: partitioning of melody  More important events are processed at deeper hierarchical (structural) levels Improvisations tend to retain only structurally important events from abstract hierarchical levels of reduction

24. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance24 Structure-Expression Link: Phrases  Decrease of tempo/dynamics at end of phrases  Amt of slowing at a boundary reflects depth of phrase embedding  More important segments have greater phrase- final lengthening  Greatest corr bet expr timing and intensity found at interm phrase level  Performers’ notated/sounded interpretations differ most at levels lower than phrase

25. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance25 Structure-Expression Link: Meter  Events on strong beats often lengthened, have delayed onsets  Events on metrical accents louder, longer, more legato  Listeners’ judgements of metric interpretation aligned best with experienced pianists’ intended meter  Articulation most often used as metric cue, loudness not always present  No one set of necessary and sufficient expressive cues to denote meter exist

26. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance26 Structure-Expression Link: Rhythm  Systematic deviations in Vienese Waltz: Short 1 - long 2 - 3

27. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance27 Expressive Timing Patterns  Structure Meter, accent pattern, simplicity (dur ratios)  Motion Rapidity, tempo, forward movement  Emotion Vitality, excitedness, playfulness

28. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance28 Comments  Melodic/metrical accents sometimes altered by presence of rhythmic accents or each other  Melody lead may serve to separate voices perceptually

29. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance29 Generative model of expr synthesis  Clarke (1993, 1995): Systematic patterns of expression result from transformations of the performer’s internal representation of musical structure  Support for view: the abilities to Replicate same expressive timing profile with little variation across performances Change interpretation and produce different expression with little practice Sight-read with appropriate expression

30. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance30 Rule-based models  Sundberg et al (1983ab): Differentiation, grouping, ensemble rules affect event durations, intensities, pitch tunings, and vibrato  Clynes (1977,1983,1986): Composer-specific inner pulses applied to different levels of musical structure  Piece-specific factors contribute as much as piece-transcendent factors captured by rules

31. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance31 Arguments against generative models  Performers can imitate expressive timing patterns with arbitrary relationships to musical structure  Accuracy worse with more disruptive structure-expression relationship, improved with repeated attempts  Suggests expression not generated solely from structural relationships

32. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance32 Perceptual Functions  Communicate particular interpretations and resolve structural ambiguities  Compensate for perceptual constraints of auditory system

33. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance33 Other explanations for expression  Compensatory explanation: Some notes played louder/longer because they would be heard softer/shorter otherwise  Musical structure elicits expectations: Detection of lengthening more difficult where expected Detection accuracy inversely related to performer’s natural use of lengthening in same piece  Structure constrains both perception and performance

34. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance34 Music Theories  Narmour (1990,1996): Model of melodic expectancy  Lerdahl (1996): Model predicting tonal tension and relaxation  Listeners can apprehend predicted structures  Expressive cues emphasize computed structures  Interpretations constrained by composition

35. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance35 AGENDA  INTRODUCTION  INTERPRETATION  PLANNING  MOVEMENT

36. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance36 Movement  Musical rhythm often defined relative to body movement  Different views on relationship: Motor control - movement generating timing Timekeeper - internal clock for anticipation and coordination of gestures

37. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance37 Timekeeper models  Role: regulate and coordinate complex time series, such as those produced between hands or between performers  Constructs beats at abstract level, providing temporal reference for future movements  Evidence: rhythm reproduction better for integer duration ratios

38. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance38 Internal clocks  Single clock model  Multiple timekeepers (Jones 1990 review)  Attributed to perceptual encoding  Attributed to production mechanisms

39. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance39 Clock operation level  Tactus: most salient metrical level  Preferred tactus ~ 600ms (spontaneous clapping period)  Typical inter-step interval ~ 540ms  Listeners use motion to describe rhythmic patterns when interbeat intervals ~ 650ms  Time periods derived are multiples or fractions of beat periods

40. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance40 Source of temporal variance  Early models: partitioned temporal variance to lack of precision of timekeeper vs. motor response delay  Extended to hierarchical organizations of timekeepers at multiple metrical levels Performed durations at metrical level less variable than durations of residual nested events within that level

41. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance41 Hierarchical clocks  Timekeeping most directly controled at intermediate metrical levels of the sub-beat, the beat, or the bar  Solo piano music: timekeeping controlled at the beat level (hands have independence in coordinating events below beat level)  Separate timekeepers controled timing of individual hands  Duet piano performance: highest precision (least variance) at bar level  (above studies assumed constant global tempo)

42. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance42 Performance timing stability  Not only at tactus / beat / bar level  Exists at level of entire piece. Durations of string quartets over repeat performances highly consistent. Std dev of piece duration ~1% Less than variations in movement lengths  Proportional tempos theory: tempos of successive sections of music form simple integer ratios  Phase synchrony, esp at structural boundaries  May reflect performer’s memory for tempo

43. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance43 Movement  Musical rhythm often defined relative to body movement  Different views on relationship: Motor control - movement generating timing Timekeeper - internal clock for anticipation and coordination of gestures

44. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance44 Motor Programs  Contains representations of internal actions and processes that translate them into movement sequence  Accounts of motor equivalence across contexts  Possible proof: Relational invariance - tempo changes as parameter change

45. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance45 Relational invariance  Relative durations of notes tend to vary across performances played at different tempi  Hypothesis: structural interpretation does not remain constant across performance tempo # group boundaries incr at slower tempo  Practicing at different rate than intended performance might be counterproductive  Lesson: do not draw conclusions from average of performances over diff tempi

46. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance46 Tempo changes perceived structure?  Tempo affects perception of duration patterns Different perceptions may result for same relative expressive timing pattern at different tempo  Repp (1995b): Manipulated degree of expressive timing and global tempo Listeners preferred reduced expression with fast tempo and augmented expression w slow

47. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance47 Kinematic models  View: music performance and perception have origins in kinematic/dynamic characteristics of typical motor actions  E.g. walking -> beat  Aesthetically satisfying performances should satisfy kinematic constraints of biological motion

48. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance48 Kinematic models  Final ritards modeled as variable curve followed be linear decrease in tempo  Feldman et al (1992): cubic polynomial models used to minimize jerk/jumpiness in connecting points of tempo changes  Repp (1992b): used quadratic

49. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance49 Models with dynamics  Studies suggest coupling bet expr timing and dynamics  Todd (1992): proposed model where intensity proportional to square of vel. Used constant acceleration  Todd (1995): proposed auditory model of rhythm performance and perception Temporal segmentation of onsets Periodicity analysis Sensory-motor feedback: tactus, body sway

50. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance50 Arguments agains kinematic models  Physical notions of energy cannot be equated with psychological concepts of musical energy  Tempo changes guided by perceptual rather than kinematic properties: Large tempo changes cannot occur too quickly (perception to rhythmic categories)

51. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance51 AGENDA  INTRODUCTION  INTERPRETATION  PLANNING  MOVEMENT

52. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance52 MUSIC PERFORMANCE  Empirical research review  Sequence planning research review  Motor control  Perceptual consequences

53. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance53 MUSIC PERFORMANCE  Empirical research review Conceptual interpretations Retrieval from memory of musical structures Transformation into motor actions  Sequence planning research review  Motor control  Perceptual consequences

54. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance54 MUSIC PERFORMANCE  Empirical research review  Sequence planning research review Hierarchical and associative retrieval influences Style-specific syntactic influences Constraints on range of planning  Motor control  Perceptual consequences

55. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance55 MUSIC PERFORMANCE  Empirical research review  Sequence planning research review  Motor control Internal timekeeper models Motor programs Kinematic models  Perceptual consequences

56. 2006-01-11ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance56 MUSIC PERFORMANCE  Empirical research review  Sequence planning research review  Motor control  Perceptual consequences Successful communication of interpretations Resolution of structural ambiguities Concordance with listeners’ expectations