1. ACIGR ACIRG ACGIR ACGRI ACRGI ACRIG AICGR AICRG AIGCR AIGRC AIRGC AIRCG AGICR AGIRC AGCIR AGCRI AGRCI AGRIC ARIGC ARICG ARGIC ARGCI ARCGI ARCIG CAIGR CAIRG CAGIR CAGRI CARGI CARIG CIAGR CIARG CIGAR CIGRA CIRGA CIRAG CGIAR CGIRA CGCIA CGARI CGRAI CGRIA CRIGA CRIAG CRGIA CRGAI CRAGI CRAIG GCIAR GCIRA GCAIR GCARI GCRAI GCRIA GICAR GICRA GIACR GIARC GIRAC GIRCA GAICR GAIRC GACIR GACRI GARCI GARIC GRIAC GRICA GRGIA GRACI GRCAI GRCIA ICAGR ICARG ICGAR ICGRA ICRGA ICRAG IACGR IACRG IAGCR IAGRC IARGC IARCG IGACR IGARC IGCAR IGCRA IGRCA IGRAC IRAGC IRACG IRGAC IRGCA IRCGA IRCAG RCIGA RCIAG RCGIA RCGAI RCAGI RCAIG RICGA RICAG RIGCA RIGAC RIAGC RIACG RGICA RGIAC RGCIA RGCAI RGACI RGAIC RAIGC RAICG RAGIC RAGCI RACGI RACIG

2. A Quantitative Measure of Melodic Structure: Computational Infrastructure and Cognitive Implications Craig Graci, Cognitive Science Program State University of New York at Oswego, USA

3. What? I will talk about a metric which is intended to assess the degree to which structural interpretations of melody are plausible with respect to tonal theory.

4. Why? The problem of modeling melodic structure in tonal music has relevance to the study of listening (Berger, 2004), performance (Clarke, 2005) and composition (Marsella & Schmidt, 1999). Consequently, the problem of measuring the degree to which a model actually captures melodic structure in tonal music should be of interest.

5. Outline Introduction The Metric: Conception and Evaluation Computational Framework for the Metric Cognitive Relevance of the Metric Conclusion

6. Prelude to a Talk Generative Theory of Tonal Music ✗ Grouping Well-Formedness Rules ✗ Grouping Preference Rules ✗ Gestalt Principles of Organization ✗ Knowledge Representation ✗ Structural Generality ✗ Lisp ✗ Java ✗ Correlation ✗ Analysis of Covariance ✗ Microworld ✗ Cognitive Artifact

7. Introduction

8. The Grouping Problem A grouping structure for a melody is what results from recursively partitioning the sequence of notes which constitute the melody into subsequences of notes. The grouping problem for a tonal melody is to determine a psychologically plausible grouping structure for a given melody.

9. Two Very Different Grouping Structures

10. Two Rather Similar Grouping Structures

12. GTTM Chapter 3 Perception - The process of finding meaningful patterns in sensory information. Gestalt Principles - Ideas (e.g., proximetry, similarity, “good form”) pertaining to how things are perceptually grouped. Grouping Preference Rules - With respect to grouping in tonal music, the GTTM GPRs are a manifestation of various Gestalt principles.

13. Sample GTTM Grouping Preference Rule Applications Proximity Similarity “Good Form”

14. Preference Rule Conflict Similarity Symmetry

15. GTTM GPR Summary GPR 1 Singleton “Avoidance” GPR 2 Proximity GPR 3 Similarity GPR 4 IntensificationGPR 5 SymmetryGPR 6 ParallelismGPR 7 Time-Span and Prolongational Stability

16. Gamma Gamma is a metric which computes the degree to which a structural interpretation of a melody is consistent with the Gestalt principles of perceptual organization, as manifested in the GTTM GPRs.

17. The Metric: Conception and Evaluation

18. Definition of Gamma γ i is the GPR i factor - a function mapping a structural interpretation of the melody onto a real number between 0 and 1. ω i are weights - real numbers which sum to 1.0 γ = ω 1 γ 1 + ω 2 γ 2 + ω 3 γ 3 + ω 5 γ 5 + ω 6 γ 6 where

19. Two Very Different Grouping Structures γ = 0.349 γ = 0.622

20. Two Rather Similar Grouping Structures γ = 0.592 γ = 0.560

21. γ 1 computation for a Little Tune Interpretation EDECDCD2EDEC EDECDC EDEDC2 γ 1 = (15/15) 5

22. Claims NOT Made About Gamma Gamma is really good at doing what it is intended to do (which is to measure the quality of grouping structures for tonal melody). There is such a thing as a fixed “one size fits all” metric for judging the quality of a grouping structure for all tonal melodies.

23. Claims Made About Gamma Gamma is a useful tool for investigating phenomena surrounding the grouping problem in tonal melody. Gamma is useful as an analytical tool for helping to determine sound grouping structures in tonal melody, and helping to learn about structural interpretation.

24. Example Gamma Computation Little Tune (Kavalevsky) γ = ω 1 γ 1 + ω 2 γ 2 + ω 3 γ 3 + ω 5 γ 5 + ω 6 γ 6 = 1.0*0.1 + 0.22*0.42 + 0.22*0.5 + 0.2*1.0 + 0.34*0.75 = 0.667

25. γ 2 computation for a Little Tune Interpretation EDECDCD2EDEC EDECDC EDEDC2 γ 2 = 5.41/13 = 0.416

26. γ 3 computation for a Little Tune Interpretation EDECDCD2EDEC EDECDC EDEDC2 γ 3 = 6.93/14 = 0.495

27. γ 5 computation for a Little Tune Interpretation EDECDCD2EDEC EDECDC EDEDC2 γ 5 = 7.0/7 = 1.0

28. γ 6 computation for a Little Tune Interpretation EDECDCD2EDEC EDECDC EDEDC2 γ 3 = (12.0 / 32.0) * 2.0 = 0.75

29. Computational Framework for the Metric

30. Clay? Clay is a simple symbolic language which can be adapted to manipulate different sorts of virtual objects. Clay has been adapted to manipulate rectangles, coins and dice, number sequences, and notes to obtain Mondrian, Chance, Number Theory, and Music “Worlds”.

31. Characteristics of Clay Clay is executable Clay is procedural... Clay is “structurally general” with respect to grouping As a music knowledge representation Clay possesses a number of significant properties:

32. Structural Generality According to Wiggins and Smaill (2000), structural generality “measures the amount of information about musical structure which can be encoded explicitly.” Clay facilitates study of grouping structure in tonal melody by virtue of its ability to explicitly encode melodic structure.

33. Clay and the Note Clay, as a music knowledge representation language, features a note. The note has lots of properties, including a scale, pitch (degree within the scale), duration (with respect to one beat), amplitude, and timbre. Melodies are modeled by playing, resting, and manipulating the state of the note.

34. The Note

35. What can you do with the note? Assuming that it is contextualized (has a scale associated with it), and that it is relatively well defined (has a pitch, duration, timbre and volume associated with it), you can: play/rest it ■ raise/lower its pitch one scale degree ■ stretch/shrink it (expand/reduce its duration) ■ change its volume ■ change its timbre ■ change its scale...

36. Some “Lower Level” Clay Primitives P - play the note R - rest the note X2 / X3 / X5 / X7 - expand the duration S2 / S3 / S5 / S7 - shrink the duration RP / LP - raise/lower the pitch a scale degree

37. Lower Level Clay Interaction Examples ? P P P X3 P S3 ? P LP LP P RP P RP P C C C C3 C \ A / B / C

38. Some “Higher Level” Clay Primitives PL - play the note for twice its duration PS - play the note for half its duration PD - play the note for 1.5 times its duration RP2 / RP3 / RP4... - raise the pitch of the note the number of scale degrees specified LP2 / LP3 / LP4... - lower the pitch of the note the number of scale degrees specified

39. Higher Level Clay Interaction Examples ? RP2 P LP P RP P LP2 P ? RP P LP P RP PL LP / E \ D \ E \ C / D \ C / D2

40. Clay Programming Example ? G1 = RP2 P LP P RP P LP2 P ? G2 = RP P LP P RP PL LP ? PH1 = G1 G2 ? PH1 / E \ D / E \ C / D \ C / D2

41. Clay Programming and Structural Interpretation ? G1 = RP2 P LP P RP P LP2 P ? G2 = RP P LP P RP PL LP ? PH1 = G1 G2 As a rule, a nonprimitive Clay command corresponds to a group.

42. MxM: Music Exploration Machine MxM is the host computational environment for Clay Lurking within MxM, right along side Clay, are MetaClay commands for displaying, sketching, scoring, and analyzing Clay commands.

43. “Little Tune” in Clay LT = P1 P2P1 = PH1 PH2P2 = PH1 PH3PH1 = G1 G2PH2 = G1 G3PH3 = G4 G5G1 = RP2 P LP P RP P LP2 PG2 = RP P LP P RP PL LPG3 = RP PL PL LPG4 = RP2 P LP P RP P LP P LPG5 = PL PL

44. Text / Tree

45. The Score

46. Gamma as a MetaClay Command

47. Gamma X: Gamma with Explanation

48. Uses of Gamma? Gamma might be used as a fitness metric in a genetic program for determining grouping structure in tonal melody. Gamma might play a role in an “educational microworld” (Papert, 1980) designed to engage students in melodic analysis and composition in order to learn about the nature of melody. Gamma might play a role in a “cognitive artifact” (Norman, 1993) designed to enhance ability to perform structural analysis of melodies.

49. Cognitive Relevance of the Metric

50. Two Small Studies Study 1: Correlational study comparing Gamma and ratings of grouping structure. Study 2: Quasi-experiment investigating the role that computational modeling, informed by Gamma, may play in developing structural grouping knowledge and ability.

51. The Correlational Study This study was designed to empirically investigate the validity of Gamma as a measure of grouping structure.

52. Method Three melodies. Twenty-six structural interpretations of each melody. Ratings from five musical people. Gamma values. The correlation between the average ratings and the values was calculated.

53. The Three Melodies German Folk Song (GFS) Dona Nobis Pacem (DNP) Ecossaise (Beethoven) (ECOS)

54. The Musical People Recording engineer / horn player / Music Department faculty member HCI Graduate Student with MIR experience School teacher / linguist who did chorus and band throughout high school Network administrator who did chorus and band throughout high school Psycholinguistics professor / banjo picker

55. The “Rating” Procedure Introduction / Instruction (30 min) Melody 1: Listen/Study (4 min) followed by Evaluations (26 min) Melody 2: Listen/Study (4 min) followed by Evaluations (26 min) Melody 3: Listen/Study (4 min) followed by Evaluations (26 min)

56. General Instructions You will be presented with 27 grouping structures for each of 3 different melodies. Please evaluate each of the grouping structures by placing an ✗ in the bubble below the melodic interpretation which best reflects your judgement of the quality of the structural interpretation, where...

57. Extremely Good means “the interpretation is arguably a candidate for what an experienced listener would deem to be among the best possible interpretations with respect to expressive performance, or learnability”. Extremely Bad means “the interpretation appears to be random, incoherent, or pathologically incorrect”. The meanings of the remaining five categories are consistent with the category names and the meanings of the “end-point categories.”

58. Specific Instructions I am particularly interested in the quality of the grouping structure at all levels of the interpretation, not just the phrase level. Consequently, please consider each of the following levels of interpretation in determining your judgement of melodic structure, giving roughly equal weight to each level:

59. Phrase level structure Segmentation of the melody into phrases. Lower level structure Structure from the phrase level to the musical surface. Higher level structure Structure above the phrase level.

60. “Marginally Bad” Example

61. “Marginally Good” Example Little Tune (Kabalevsky)

62. “Extremely Good” Example Little Tune (Kabalevsky)

63. “Extremely Bad” Example Little Tune (Kabalevsky)

64. “OK” Example Little Tune (Kabalevsky)

68. Example (GFS)

69. Interrater Reliability For DPN, Cronbach’s Alpha = 0.81 for the 5 raters For GFS, Cronbach’s Alpha = 0.90 for the 5 raters For ECOS, Cronbach’s Alpha = 0.94 for the 5 raters

70. Data for Gamma and Humans

71. Correlation Values For the DPN melody, r(24) = 0.816 For the GFS melody, r(24) = 0.832 For the ECOS melody, r(24) = 0.693 In each case, correlation is significant at the 0.01 level.

72. The Quasi-experiment The study was designed to investigate the role that computational modeling, informed by Gamma, may play in developing structural grouping knowledge and ability.

73. Participants Computational Modeling Group: 26 undergraduate students, each enrolled in a cognitive science course. Control Group: 17 undergraduate students, each enrolled in a semiotics course.

74. Procedure Background survey - computing/music Pretest - structure 3 melodies Lecture - Gestalt principles / melodic grouping structure Training (experimental group only) - Computer modeling with Gamma in mind Posttest - structure 3 melodies

75. The Background Survey Ten questions about computing. Ten questions about math. On the basis of the answers, subects are classified as “high/low computing” and “high/low music”.

76. A Note on the Participants Some of the students (more than half) have no computer programming skills. Some of the students (roughly a third) cannot read music. Consequently, computer programming skills and music reading skills could not be presumed.

77. A Note on the Pre/Post Tests Sloboda (2005) suggests the need for more ‘indirect’ measures to probe structural awareness in ordinary untrained listeners. The pretest/posttest methodology in this study appears to fall squarely into this category of measure.

78. Pre/Post Test Questions Subjects listen to a the melody. Subjects provide phrase level grouping by boxing 7 to 12 note sequences, while listening to the melody twice more. Subjects provide lower level grouping by boxing 1 to 6 note sequences, while listening to the melody twice more.

79. The First Pretest Melody

80. A Participant’s Lower Level Grouping

81. The Participant’s Phrase Level Grouping

82. STOR File for the Participant’s Grouping Structure / E E | / F0.5 \ E0.5 \ D0.5 \ C0.5 / D | / G G2 | || \ E E | / F0.5 \ E0.5 \ D0.5 \ C0.5 \ B | / D \ G2 | || / G G | / A0.5 \ G0.5 \ F0.5 \ E0.5 / A | A A2 | \ G \ E | || / G0.5 \ F0.5 \ E0.5 \ D0.5 \ C2 | || |||

83. Reverse Compilation STOR ➞ Clay

84. The Structured Score

85. The Training Training for the computer modeling group consisted of a Clay tutorial phase and a Clay modeling phase. The Clay tutorial phase consisted of 2 hours of prescribed Clay interaction/programming in a lab setting. The Clay modeling phase consisted of modeling four melodies in Clay as the main part of a take home exam. The first melody was “practice”. The others were “real”.

86. Training Melodies The practice melody consisted of four bars of Mozart. The real melodies were Dona Nobis Pacem, the German Folk Song, and Beethoven’s Eccossaise. Students were instructed to try to maximize the Gamma value of their model. The resulting 26 interpretations of each of the three melodies were used as the basis of the correlational study.

87. Posttest Scores

88. Results An ANCOVA was performed to determine the effect of the computational modeling activity on hierarchical structuring of melodies, using performance on the pretest measure of hierarchical structuring knowledge and musical ability as covariates. Participants in the computer modeling group were better able to structure the three posttest melodies than participants in the control group. The result was significant at the 0.1 level.

89. Conclusion

90. Clay and MxM support the claim that computational systems which are sensitive to representational issues in music are viable environments for studying musical phenomena from an empirical perspective (e.g., Honing, 1993). Metrics, such as Gamma, grounded in Gestalt principles, may have a useful role to play, not only in the process of determining sound grouping structure, but also in the process of learning to establish and evaluate grouping structure.