1 The Representation, Indexing and Retrieval of Music Data at NTHU Arbee L.P. Chen National Tsing Hua University Taiwan, R.O.C.

2 Outline Content-based media data retrieval Music data retrieval Features of music data Feature indexing and matching Prototypes Reference

3 Content-based Media Data Retrieval Representation of media contents features Feature extraction from media data Feature indexing Query interface

4 Content-based Media Data Retrieval Matching query features against the feature index approximate/partial matching similarity measure precision: how many of the answers are in fact correct recall: how many of the correct answers are in fact retrieved relevance feedback

5 Music Data Retrieval: System Architecture

6 Features of Music Data

7 Static music information The intrinsic music characteristics of music objects Key, beat, and tempo E.g., the Beethoven Symphony No. 5, Op. 67, C minor, 4/4, Allegro con brio Acoustical features Loudness, pitch, duration, bandwidth and brightness Can be computed and represented as numerical values

8 Features of Music Data Thematic features Themes, melodies, rhythms, and chords Can be derived from the staff information of a music object Melody The melody of a song is the sequence of the pitches of all notes in the songs E.g., the melody of the theme of the Beethoven ’ s Symphony No.5 is “ sol – sol – sol – mi – fa – fa – fa - re ”

9 Features of Music Data Rhythm The rhythm of a song is the sequence of the durations of all notes in the songs E.g., the rhythm of the theme of the Beethoven ’ s Symphony No.5 is “ 1/2-1/2-1/2-2-1/2-1/2-1/2-4 ” Chord A chord consists of three (root, third, and fifth) or more notes which sound together in harmony

10 Features of Music Data Coding scheme: a music object  a sequence of music segments music segment = (segment type, segment duration, segment pitch) four segment types: ┌┐ (type A), └┘ (type B), ┌┘ (type C), and └┐ (type D)

11 Features of Music Data For example, the sequence of music segments: (B,3,-3) (A,1,+1) (D,3,-3) (B,1,-2) (C,1,+2) (C,1,+2) (C,1,+1)

12 music segment = (type, duration, pitch)

13 Features of Music Data Repeating Pattern A sequence of notes appearing more than once in the music object Efficient content-based retrieval Semantics-rich representation Extracting repeating patterns Tree-based approach Matrix-based approach

14 Features of Music Data Experiment 1

15 Features of Music Data Dissimilarity of melody strings

16 Features of Music Data Dissimilarity of repeating patterns

17 Features of Music Data Experiment 2

18 Features of Music Data Validity of classes

19 Finding Repeating Patterns: Tree-based Approach Construct an RP-tree for RP ’ s with lengths 2 n, n  0, 1,... S = “ ABCDEFGHABCDEFGHIJABC ”

20 Finding Repeating Patterns: Tree-based Approach Length 1 {A, 3, (1, 9, 19)} {B, 3, (2, 10, 20)} {C, 3, (3, 11, 21)} {D, 2, (4, 12)} {E, 2, (5, 13)} {F, 2, (6, 14)} {G, 2, (7, 15)} {H, 2, (8, 16)}

21 Finding Repeating Patterns: Tree-based Approach Length 2 {AB, 3, (1, 9, 19)} = {A, 3, (1, 9, 19)}  0 {B, 3, (2, 10, 20)} {BC, 3, (2, 10, 20)} = {B, 3, (2, 10, 20)}  0 {C, 3, (3, 11, 21)} {CD, 2, (3, 11)} = {C, 3, (3, 11, 21)}  0 {D, 2, (4, 12)} …

22 Finding Repeating Patterns: Tree-based Approach Length 4 {ABCD, 2, (1, 9)} = {AB, 3, (1, 9, 19)}  0 {CD, 2, (3, 11)} {BCDE, 2, (2, 10)} = {BC, 2, (2, 10, 20)}  0 {DE, 2, (4, 12)} … Length 8 {ABCDEFGH, 2, (1, 9)} = {ABCD, 2, (1, 9)}  0 {EFGH, 2, (5, 13)}

23 Finding Repeating Patterns: Tree-based Approach

24 Finding Repeating Patterns: Tree-based Approach Prune trivial patterns of length 2 n, n = 0, 1, … Let X be an RP of S, Y a substring of X, and Z a substring of Y If freq(X) = freq(Z), Y is trivial

25 Finding Repeating Patterns: Tree-based Approach Length 1 {ABCDEFGH, 2, (1, 9)} {ABCD, 2, (1, 9)} {BCDE, 2, (2, 10)} {CDEF, 2, (3, 11)} {DEFG, 2, (4, 12)} {EFGH, 2, (5, 13)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)} {CD, 2, (3, 11)} {DE, 2, (4, 12)} {EF, 2, (5, 13)} {FG, 2, (6, 14)} {GH, 2, (7, 15)}

26 Finding Repeating Patterns: Tree-based Approach Length 2 {ABCDEFGH, 2, (1, 9)} {ABCD, 2, (1, 9)} {BCDE, 2, (2, 10)} {CDEF, 2, (3, 11)} {DEFG, 2, (4, 12)} {EFGH, 2, (5, 13)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)}

27 Finding Repeating Patterns: Tree-based Approach Length 4 {ABCDEFGH, 2, (1, 9)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)}

28 Finding Repeating Patterns: Tree-based Approach Generate all patterns of lengths  2 n, n  0, 1,... {ABCDEFGH, 2, (1, 9)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)} {ABC, 3, (1, 9, 19)} order-1 string-join AB  1 BC = ABC

29 Finding Repeating Patterns: Tree-based Approach Prune all trivial patterns {ABCDEFGH, 2, (1, 9)} {ABC, 3, (1, 9, 19)}

30 Feature Indexing and Matching 1D-List PAT-Tree L-Tree Augmented Suffix Tree Grid-Twin Suffix Tree

31 Feature Indexing and Matching: 1D-List There are two music objects M1 and M2 M1: ” sol-mi-mi-fa-re-re-do-re-mi-fa-sol-sol- sol ” M2: ” do-mi-sol-sol-re-mi-fa-fa-do-re-re-mi ” The melody string of the music query Q: ” do-re-mi ” Problem: to find whether M1 and M2 contain the melody string Q

32

33

34 Feature Indexing and Matching: PAT-Tree Example, songs in chord strings Song1 : Am F2 Dm Am Song2 : C C F C Song3 : G E1 C D Song4 : E1 G Am Bm

35 Feature Indexing and Matching: PAT-Tree

36 Prototype 1

37 Prototype 1

38 Prototype 2

39 Prototype 2

40 Prototype 2

41 References ( Chen, A.L.P., M. Chang, J. Chen, J.L. Hsu, C.H. Hsu, and S.Y.S. Hua, “ Query by Music Segments:An Efficient Approach for Song Retrieval, ” in Proc. of IEEE Intl. Conference on Multimedia and Expo, Chen, J.C.C. and A.L.P. Chen, “ Query by Rhythm:An Approach for Song Retrieval in Music Database, ” in Proc. of IEEE Intl. Workshop on Research Issues in Data Engineering, Chou, T.C., A.L.P. Chen, and C.C. Liu, “ Music Databases: Indexing Techniques and Implementation, ” in Proc. of IEEE Intl. Workshop on Multimedia Data Base Management System, Hsu, J.L., C.C. Liu, and A.L.P. Chen, “ Efficient Repeating Pattern Finding in Music Databases, ” in Proc. of ACM Intl. Conference on Information and Knowledge Management, 1998.

42 References ( Lee, W and A.L.P. Chen, “ Efficient Multi-Feature Index Structure for Music Data Retrieval, ” in Proc. of SPIE Conference on Storage and Retrieval for Image and Video Databases, Liu, C.C., J.L. Hsu, and A.L.P. Chen, “ An Approximate String Matching Algorithm for Content-Based Music Data Retrieval, ” in Proc. of IEEE Intl.Conference on Multimedia Computing and Systems, Liu, C.C., J.L. Hsu, and A.L.P. Chen, “ Efficient Theme and Non- Trivial Repeating Pattern Discovering in Music Databases, ” in Proc. of IEEE Intl. Conference on Data Engineering, 1999.