1. Emotion-Based Music Recommendation By Assciation Discovery from Film Music Fang-Fei Kuo1, Meng-Fen Chiang2, Man-Kwan Shan2 and Suh-Yin Lee1 Department of Computer Science and Information Engineering, National Chiao- Tung University Hsinchu, Taiwan {ffkuo, sylee}@csie.nctu.edu.tw Department of Computer Science, National Cheng-Chi University Taipei, Taiwan {g9309, mkshan}@cs.nccu.edu.tw 13th ACM international conference on Multimedia MM '05 Publisher

2. outline Introduction Model Emotion Detection Music Feature Extraction Association Discovery and Recommendation Performance evaluation Conclusions

3. introduction Two major approaches for the personalized music recommendation (1) Content-based filtering (2) collaborative filtering Recommend music based on the emotion

4. Introduction(con) Potential application production of home video, shopping mall to stimulate sales, music therapy etc.

5. Introduction(con) Recommend music based on emotions 1. recommend music by the rules (psychological research) 2. learn the rules by training from music labeled with emotion types Recommendation model to recommend music by association discovery from film music

6. Model

7. Emotion Detection

8. Music Feature Extraction Music elements which affect the emotion include melody, rhythm, tempo, mode, key, harmony, dynamics and tone-color

9. Melody Extracting (1) All-mono (2) Entropy-channel (3) Entropy-part (4) Top-channel Improved the All-mono to get more precise melody sequence instrument and volume

10. Melody Extracting Improved the All-mono Three steps: step1.Remove channels of instruments which are unlikely for melody performing step2.For each measure, select the channel of the largest volume step3.Keep the highest note

11. Chord assignment algorithm 1.We chose 60 common chords as the candidates 2.count score of each candidate Stage1:Determine the chord sampling unit Stage2: step:A step:B step:C Stage3: step:A step:B step:C Melody Extracting

13. Stage1: Determine the chord sampling unit find the prevailing note, which has the longest total performance length half note = four measures quarter note = two measures eighth note = one measure Melody Extracting

14. Stage2: 1.chord of the sampling unit which sounds first (10) 2.each kind of note (1) 3.longest pitch duration>half (2) else(1) 4.no sharps or flats—discard the chord expect the first chord

15. Melody Extracting Stage3: 1. root is descending fifth,descending third or ascending second (2) 2. if I7, add two points to IV chord if II7, add two points to V chord if III7, add two points to VI chord if IV7, add two points to V chord if V7, add two points to I and VI chord, one point to V chord if VI7, add two points to II chord if VII7, add two points to III chord 3. add two points to the chords whose root is the lowest pitch

16. Rhythm is the music feature that describes the timing information of music Extracting the beat sequence base on percussion instrument quarter note = 1000 basic unit is set to sixteenth note long Highest Repeating pattern Rhythm Extracting

17. Tempo calculated Tempo=NB/NS NB is number of beat NS is the length of the rhythmic pattern

18. Association Discovery and Recommendation Mixed Media Graph Music Affinity Graph

19. Mixed Media Graph Represent all the objects, as well as their attributes as nodes in a graph Nearest neighbor links (NN-links) Object-attribute-value links (OAV-links) Similarity function si(*,*) to computing the k nearest neighboprs

20. Mixed Media Graph (con) K=1

21. Mixed Media Graph (con) Correlation discovery by random walk with restarts u A (B) denoted the steady-state probability

22. Music Affinity Graph For each music object vertex, four types of attribute vertices – emotion, chord, rhythm, and temp vertices

23. Music Affinity Graph (con) The edge between chord vertices is constructed based on the k-nearest neighboring The edge between emotion (rhythm) vertices is constructed only the same emotion (rhythm) vertices K=1

24. Music Affinity Graph (con) steady-state probability Not necessary to say that the music feature value with high affinity is highly corrlative to the query emotions Complement affinity graph G’ Final affinity equal G-G’

25. Review

26. Performance evaluation K=7 Score i = |E i ∩E q | / (√|E i |x|E q |)

27. Conclusions Recommend music based on emotion Construct the recommend model from film music Experimental result shows that the top-one result’s average score achieves 85%

28. End