1. The Effectiveness Study of Music Information Retrieval Arbee L.P. Chen National Tsing Hua University 2002 ACM International CIKM Conference

2. Outline Motivation The Ultima Project –The 1D-List approach –The APS approach The Effectiveness Study –Estimating all relevant references Conclusion

3. Motivation Various approaches were proposed to provide efficient and effective content-based retrieval of music objects –Music representations pitch, rhythm, contour, chord –Index structures tree-based index, list-based index, n-gram index –Query processing methods exact match, partial match, approximate match

4. Motivation (Cont.) A platform is needed for the evaluation of various music information retrieval methods –Efficiency response time –Effectiveness recall-precision curve The Ultima project builds such a platform –Same data set and query set –Also serves as a testbed whenever new approaches are proposed

5. The Ultima Project (Cont.) Architecture Mediator Query processing module Report module Summarization module Query generation module Data store

6. The Ultima Project (Cont.) Two approaches have been compared –1D-List –APS ApproachRepresentationIndex structure 1D-Listmelody stringlist-based APSsequence of music segmentssuffix tree-based

7. The 1D-List Approach The 1D-List approach –Music objects are coded as melody strings “so-mi-mi-fa-re-re-do-re-mi-fa-so-so-so” –Melody strings are organized as linked lists –Both exact and approximate matching can be handled Exact link, insertion link, dropout link, transposition link

8. The 1D-List Approach (Cont.)

9. The APS Approach The APS approach –Music objects are coded as sequences of music segments four segment types to model the music contour pitch and duration are considered –Index structures one-dimensional and two-dimensional augmented suffix tree –Both exact and approximate matching can be handled

10. The APS Approach (Cont.) Representation type A type B type C type D

11. The APS Approach (Cont.) The suffix tree of the string S=“ABCAB” (a)An example of suffix tree (b)A 1-D augmented suffix tree 1 A $ A B $ B 2 A B $ 3

12. The APS Approach (Cont.) Similarity measure –Given a query sequence Q = (i 1, j 1, k 1 ) (i 2, j 2, k 2 )... (i n, j n, k n ), and a candidate sequence from the database C = (i 1, x 1, y 1 ) (i 2, x 2, y 2 )... (i n, x n, y n ).

13. The Effectiveness Study Traditional measures of effectiveness are precision and recall However, the number of relevant references are usually unknown –it is unrealistic for the user to make relevant judgments to all music objects in the database

14. The Effectiveness Study (Cont.) How to estimate the number of relevant references NR? –AS x is the set of relevant objects from the top x ranked results –RS x is the set of the top x ranked results retrieved by an approach –Assumption 1:,, the number of the retrieved results is a function of the number of retrieved relevant objects Assumption 2:, where B is a positive integer Based on the two assumptions, NR can be derives as follows:

15. The Effectiveness Study (Cont.) rankrelevancerecallprecision 1Y0.11 2Y0.21 3Y0.31 4 0.75 5Y0.40.80 60.40.67 7Y0.50.71 80.50.63 9Y0.60.67 10Y0.70.70 110.70.64 120.70.58 13Y0.80.62 140.80.57 150.80.53 16Y0.90.56 170.90.53 180.90.50 19Y10.53 2010.50 Rankrelevancerecallprecision 1Y0.11 2Y0.21 3Y0.31 4 0.75 5Y0.40.80 60.40.67 7Y0.50.71 80.50.63 |RS x | = 8, |AS x | = 5, |DB| = 20

16. The Effectiveness Study (Cont.) Method Factor APS 1D-List 1-D AST (duration) 1-D AST (pitch) 2-D AST Number of music objects for generating queries 10 Is the query sample a refrain or an incipit? refrain/incipit Length of query sample, denoted L 6/10 (segment)8/12 (note) Number of query samples per music object 44 Threshold setting of approximation for a query sample th_d = 0, 0.5, 1.0 th_p = 0, 0.5, 1.0 K=0, 4, 7 (for L=8) K=0, 6, 11 (for L=12) Total number of posing queries120 Experiment setup

17. The Effectiveness Study (Cont.) Experiment results

18. The Effectiveness Study (Cont.) Experiment results

19. The Effectiveness Study (Cont.) Experiment results –1D-list achieves a high precision in the limited range of recall, while a moderate precision for the APS family can be obtained –Comparing the APS family, the precision in a descending order is: 1D-AST (pitch), 2D-AST, and 1D-AST (duration) –In average, the effectiveness of “incipit” queries is better than “refrain” queries

20. Conclusion The Ultima project builds a platform for evaluating the performance of various approaches of music information retrieval A new measure for estimating the number of relevant references is proposed Future work –Design and implement the summarization module as well as the query generation module –Extend the project for evaluating polyphonic music retrieval methods