1. 1 Robust Temporal and Spectral Modeling for Query By Melody Shai Shalev, Hebrew University Yoram Singer, Hebrew University Nir Friedman, Hebrew University Shlomo Dubnov, Ben-Gurion University

2. 2 Prelude

3. 3 Problem Setting Database of real recordings Query: a melody Find: performances of the queried melody

4. 4 Challenge Find performances of the queried melody independent of: –Tempo –Performing instrument –Dynamics –Expression –Accompaniment

5. 5 Related Work A. Ghias, et al. “Query by humming” A. S. Durey and M. A. Clements. “Melody spotting using hidden markov models” C. Raphael. “Automatic segmentation of acoustic musical signals using HMMs” B. Doval and X. Rodet. “Fundamental frequency estimation using a new harmonic matching method”

6. 6 Overview of Solution Employ a statistical framework Align a melody to a performance using an explicit tempo modeling Employ a maximum likelihood model for the spectrum of a note given the note’s pitch value Find the best alignment of a melody to a performance using dynamic programming

7. 7 Statistical Framework Query Engine For each recording find: A database of real recordings A melody query Ranked list of According to

8. 8 Melody Modeling Hidden Variable Observed Variable Legend: MelodyTempo Aligned Melody Sound

9. 9 Tempo Modeling Sequence of scaling factors (one per note) Model tempo as a first order Markov model Use log-normal distribution to model conditional probability of tempo

10. 10 Spectral Modeling

11. 11 Spectral Modeling

12. 12 Spectral Modeling (cont.)

13. 13 Spectral Modeling (cont.) Estimate the amplitude at each harmony and global variance of the noise using the maximum likelihood principle Resulting signal-to-noise likelihood function:

14. 14 Finding the best melody-performance alignment Recurse over tempo and end-time of the previous note  Dynamic Programming procedure Complexity: #notes Length of Signal #Possible Tempo values

15. 15 Queries: 50 melodies from opera arias (from Midi files) Database: over 800 performances of opera arias performed by over 50 tenors with full orchestral accompaniment Compared our variable-tempo (VT) model vs. fixed-tempo (FT) and locally-fixed-tempo (LFT) models Compared our Harmonic with Scaled Noise (HSN) spectral model vs. Harmonic with Independent Noise (HIN) model Experimental Results

16. 16 Evaluation Measures Oerr = 0 Cov = 3 - 2 + - + - - - - - Likelihood Value Index of Performance in the ranked list 12345

17. 17 Summary of Results One Error of VT+HSN: 8% Average Precision of VT+HSN: 95% Coverage of VT+HSN: 0.21

18. 18 Results 0.7521.670.350.6922.960.38 FT 0.7517.940.370.6917.330.43 LFT 0.6911.830.460.6510.670.51 VT 5 Sec. 0.7319.080.360.7119.830.38 FT 0.428.150.660.448.100.66 LFT 0.193.020.830.191.750.86 VT 15 Sec. 0.7922.460.330.7720.690.34 FT 0.485.980.630.465.900.66 LFT 0.100.400.920.080.210.95 VT 25 Sec. OerrCovAvgPOerrCovAvgP HINHSN Spectral Distribution Model

19. 19 Precision-Recall

20. 20 Illustration of Segmentation

21. 21 Future Work More data Other genre of music Alternative spectral distribution models using supervised learning methods. Use alignment results for separating a soloist from the accompaniment