1. Machine Discoveries: A few Simple, Robust Local Expression Principles Written by Gerhard Widmer presented by Siao Jer, ISE 575b, Spring 2006

2. Presentation Overview General Overview Introduction Training Data Target Classes Experimental Results Quantitative Validation Conclusion Future Research

3. Gerhard Widmer Head of the Department of Computational Perception at Johannes Kepler University Linz, Austria Head of Machine Learning, Data Mining, and Intelligent Music Processing Group at the Austrian Research Institute for Artificial Intelligence Numerous publication, awards, projects

4. General Overview Discovering rules of expressive music performance Inductive machine learning Experiments with large data sets Simple and general principles Robust with surprisingly high level of accuracy

5. Introduction What do performers do to make music “come alive?” Studies done through a few classical approaches Proposal of inductive machine learning No preconceptions and expectations Huge data sets allowed for more validity

6. Introduction Previous work Success in ability of machine learning (Widmer 2000) Extremely complex Attempt to find a complete model Current goals Testing new learning algorithm based on partial models Learn rules of timing, dynamics, articulation Testing degrees of fit over various styles and performers

7. Training Data 13 complete Mozart piano sonatas Performed by Roland Batik On computer monitored grand piano MIDI format Includes hammer speed, impact times, pedal movements measured & xform’ed Written score coded into computer format Timing, dynamics, & articulation computed 106,000 total notes Melody restriction limits us to 41,000 notes

8. Target Classes Objective: find note-level rules Limit predictions to categorical decisions Timing Dimension: note N is considered lengthened If the note is lengthened relative to the instantaneous tempo over the previous note If lengthened relative to local tempo over the last 20 notes Analogous to this is a note shortened

9. Target Classes Dynamics: louder if Louder than previous note And louder than average level of piece Analogous to this is softer Articulation Staccato if played duration ratio (PDR) is less than 0.8 Legato if greater than 1.0 Portato otherwise, but study only concerned with staccato and legato Pedaling not taken into account for articulation Notes do not necessarily have to fall into one of these classes

10. Learning Partial Rule-based Models No expectation to cover and describe all instances Describe parts and define in meaningful terms PLCG algorithm developed with these ideas in mind Goal to come up with rules that covered lots of cases with good accuracy

11. Learning Partial Rule-based Models General Steps Separation into subsets Learning partial rules within subsets Merge all rules Clustering of rules One generalization per cluster Optimize trade-offs (coverage vs. accuracy) Result: 383 specialized rules narrowed to 17 general rules

12. Experimental Results Timing: Lengthening Notes "Lengthen the middle note in a “cummulative” 3- note rhythm situation (ie, given 2 notes of equal duration followed by a longer note, lengthen the note that precedes the final, longer one).” Most important one as it has highest prediction value “Lengthen a note if it is followed by substantially longer note (ie the ratio between its duration and the duration of the next note is < 1:3)” “Lengthen a note if it preceds an upward melodic leap of more than a perfect forth, if it is in a metrically weak position, and if it is preceded by (at most) stepwise motion” 2 cases above have atleast 70% prediction rate

13. Experimental Results Timing: Lengthening Notes “Lengthen a note if it preceds an upward melodic leap of more than a perfect forth, if it is in a metrically weak position, and if it is preceded by (at most) stepwise motion” More of a “tendancy” than a rule Interesting Note: previously observed But not over such a large data set

14. Experimental Results Timing: Shortening Notes Difficult, but understandable No strong rules, but a few tendencies “Shorten a note in a sequence PN-N-NN if it is longer than its predecessor and longer than its successor.” “Shorten a note in fast pieces in 3/8 time if the duratio ratio between previous note and current note is larger than 2:1, the current note is at most a sixteenth, and is again followed by a longer note.” Example of a specialized rule Correlation with Gabrielsson 1987

15. Experimental Results Dynamics: Stressing Notes Clear rules emerge, low coverage Interesting note: relating stress to melodic contour Upward melodic movement Observation by previous research as well “Stress a note by playing it louder if it is preceded by an upward melodic leap larger than a perfect fourth.”

16. Experimental Results Dynamics: Stressing Notes “Stress a note by playing it louder if it forms the apex of an up-down melodic contour and is preceded by an (upward) leap larger than a minor third.” “Stress a note by playing it louder if it at least twice as long as its predecessor, is reached by upward motion, and is in a quite strong metrical position.”

17. Experimental Results Dynamics: Attenuating Notes Difficult to predict “Attenuate a note by playing it softer if it is less than 1/5 the duration of its predecessor.” “Attenuate a note by playing it softer if it is preceded by a downward leap larger than a major third, is metrically weak, and is preceded by a note at least 1/3 of a beat long.” “Attenuate a note by playing it softer if it is preceded by a downward leap larger than a perfect fifth and is metrically weak.” Observation: linking metrically weak notes reached by downward leaps

18. Experimental Results Articulation Staccato Most easily predictable, 4 strong rules “Play a note staccato if the note is marked with a staccato dot in the score.” “Play a note staccato if it is followed by a note of the same pitch (ie the interval between the note and its successor is a unison).” Observations: Combine for +90% accuracy & 6,000 cases Previously observed in KTH Rules (Friberg 1995) Physical reasons and explanations

19. Experimental Results Articulation Staccato “Insert a micropause after a note if it precedes an upward leap larger than a perfect fourth and is metrically weak.” “Insert a micropuase after a note of it is reached by downward motion and is followed by a note more than twice as long (ie the ratio between its duration and duration of the next note is < 0.4).” Observations: Correlation to lengthening rules Supported by “Cumulative Rhythm” (Nramour 1977) Articulation Staccato  30% of expression observed

20. Experimental Results Articulation Legato Most difficult to predict A LOT fewer instances vs. staccato No markings on score Low prediction rate (53.7%) “Play a note legato if it is not marked staccato in the score, if it forms the apex of an up-down melodic contour, if it is quite short (<1/3 of a beat), and is metrically quite strong.” Observations: Melodic peak  legato?

21. Quantitative Validation: Generality I Different Performer (Philippe Entremont) Same pieces No significant degradation in coverage and accuracy Exception of “softer” Higher coverage in “lengthen” “louder” “staccato”

22. Quantitative Validation: Generality II Testing on Different Styles & Artists 2 Chopin pieces 22 skilled pianist from Univ. of Music in Vienna Surprising Results “softer” and “legato”  unpredictable “louder”  high % of positive examples, but high level of false predictions too “lengthen,” “shorten,” & “staccato”  extremely good Need more diversity of pieces

23. Conclusion Small Step Basic & simple rules Robust model of local expression principles Observations from other researchers Autonomous discovery Large data sets Possible foundation

24. Further Research Further evaluation of rules different performers Different types of music Extension to other dimensions (e.g. Harmony) Going beyond note level (e.g. phrase structure) Comprehensive multi-level model