Presentation is loading. Please wait.

Presentation is loading. Please wait.

Functional Data Analysis of Continuous Judgments in Music Cognition.

Similar presentations


Presentation on theme: "Functional Data Analysis of Continuous Judgments in Music Cognition."— Presentation transcript:

1 Functional Data Analysis of Continuous Judgments in Music Cognition

2 Gesture in Musical Performance What role do a musician’s gestures play in a performance? What role do a musician’s gestures play in a performance? Do they convey emotion? Do they convey emotion? Do gestures convey the same things as does the music? Do gestures convey the same things as does the music? Popular study in field of music psychology Popular study in field of music psychology Data on listener’s emotions are collected in real-time Data on listener’s emotions are collected in real-time With a real-time Optotrak slider, which With a real-time Optotrak slider, which Measures location of the slider 10 times per second. Measures location of the slider 10 times per second. Continuous Judgments of Music

3 The Tension & Gesture Experiment Musical performance recorded on video Musical performance recorded on video Stravinsky’s 2 nd Piece for Solo Clarinet Stravinsky’s 2 nd Piece for Solo Clarinet 30 musically-trained participants either 30 musically-trained participants either Watched & Listened to the recording (Natural) Watched & Listened to the recording (Natural) Watched the silent recording (Video Only), or Watched the silent recording (Video Only), or Listened to the recording (Audio Only). Listened to the recording (Audio Only). And reported their continuous “level of tension [emotion]”. And reported their continuous “level of tension [emotion]”.

4 The Data Vectors of length 800 Vectors of length “Audio Only,” 10 “Video Only,” 10 “Audio + Video” 10 “Audio Only,” 10 “Video Only,” 10 “Audio + Video” Scaled to [0,1] interval Scaled to [0,1] interval

5 The Functional Objects 150 order 6 B-splines, using FDA software in Matlab. 150 order 6 B-splines, using FDA software in Matlab. Then smoothed. Then smoothed.

6 Functional Principal Components Analysis 25 s – 65 s: crucial separation 25 s – 65 s: crucial separation Effect of amplification / attenuation: how strong are the changes of emotion? Effect of amplification / attenuation: how strong are the changes of emotion?

7 Emotion(t) = µ(t) + β 0 (t){AudRemoved} + β 1 (t){VidRemoved} + ε(t) Functional Linear Model

8 Emotion(t) = µ(t) + β 0 (t){AudRemoved} + β 1 (t){VidRemoved} + ε(t)

9 Derivatives describe music dynamics: Tension These judgments are really measures of musical emotion. These judgments are really measures of musical emotion. ‘Tension/Resolution’ is rate of change of emotion (velocity). ‘Tension/Resolution’ is rate of change of emotion (velocity). When emotion is rapidly increasing, music has strong tension. When emotion is rapidly increasing, music has strong tension. When emotion is rapidly decreasing, music has strong resolution. When emotion is rapidly decreasing, music has strong resolution.

10 Derivatives describe music dynamics: Force/Release ‘Force/Release’ is rate of change of tension (acceleration). ‘Force/Release’ is rate of change of tension (acceleration). When tension is rapidly increasing, we feel a musical force. When tension is rapidly increasing, we feel a musical force. When tension is rapidly decreasing, we feel a musical release. When tension is rapidly decreasing, we feel a musical release. When both derivatives are near zero, music is inert (the “new age/massage music” effect). When both derivatives are near zero, music is inert (the “new age/massage music” effect).

11 Phase-Plane Plots Can examine dynamics with plot of acceleration vs velocity. Can examine dynamics with plot of acceleration vs velocity. Purely harmonic behavior gives a circle. Purely harmonic behavior gives a circle. The larger the radius, the more musical energy transfer. The larger the radius, the more musical energy transfer.

12

13

14

15 25 – 33 s: AUDIO: high volume, note density, and pitch; end of musical phrase AUDIO: high volume, note density, and pitch; end of musical phrase tension then strong resolution; big energy transfer VIDEO: routine gestures; then dramatic flourish VIDEO: routine gestures; then dramatic flourish low tension and small resolution; moderate pull with flourish; small energy transfer 33 – 65 s: AUDIO: decrease in volume (mezzo forte to pianissimo), note density, and pitch AUDIO: decrease in volume (mezzo forte to pianissimo), note density, and pitch continued resolution; push from new phrase; back to inertness VIDEO: eyebrow and body movements VIDEO: eyebrow and body movements push from new phrase, moderate tension from movements

16 New applications New applications PCA as exploration PCA as exploration Derivatives have physical (and scientific) meaning Derivatives have physical (and scientific) meaning Phase-Plane plots highlight relationships Phase-Plane plots highlight relationships What have we learned?

17 References Data courtesy of Daniel Levitin and Bradley Vines, McGill University Departments of Psychology and Music. Data courtesy of Daniel Levitin and Bradley Vines, McGill University Departments of Psychology and Music. For functional principal components analysis, see Functional Data Analysis (1996), Ch. 6. For functional principal components analysis, see Functional Data Analysis (1996), Ch. 6. For phase-plane plots, see Applied Functional Data Analysis (2002), Ch. 3. For phase-plane plots, see Applied Functional Data Analysis (2002), Ch. 3. For functional linear models, see Functional Data Analysis (1996), Ch. 9 – 11. For functional linear models, see Functional Data Analysis (1996), Ch. 9 – 11.


Download ppt "Functional Data Analysis of Continuous Judgments in Music Cognition."

Similar presentations


Ads by Google