Query by Singing/Humming via Dynamic Programming J.-S. Roger Jang (張智星) jang@mirlab.org http://mirlab.org/jang MIR Lab, CSIE Dept. National Taiwan University

Introduction to Query by Singing/Humming Query by singing/humming (QBSH, 哼唱選歌) Goal: Identify a song by singing or humming Demos http://mirlab.org/demo/miracle (PC) http://www.midomi.com (PC) http://www.soundhound.com (Mobile) Approach to QBSH Pitch tracking: Convert singing/humming into pitch vector Retrieval: Find the distance between the pitch vector and each song in the database Our homework Explore how we can use dynamic programming (DP) to find the distance in the retrieval part of QBSH

Examples of Pitch Vectors and Music Notes MIDI numbers Used in MIDI files AKA semitones Music note vector Integer semitones Example MIDI file of 小星星 Note vector: [60 60 67 67 69 69 67 65 65 64 64 62 62 60] Pitch vector Real-number semitone Example of singing clips 小星星, pitch vector (play) 在那遙遠的地方, pitch vector (play) Pitch rate = 31.25 pitch/second

Our Task: Optimal Alignment How to find the distance between a pitch vector p(i), i=1~m, and a note vector q(j), j=1~n? We need to find the optimal alignment. This can be achieved by DP. Example: The alignment path: (1,1), (2,1), (3,2), (4,2), (5,2), (6,2), (7,3), (8,3), (9,4), (10,4), (11,4), (12,4) Distance:

Three-step Formula of DP for Alignment Three-step DP formula Optimum-value function: D(i,j) is the min distance between p(1:i) and q(1:j) Recurrent equation: Answer: Assumption Anchored at beginning  p(1) is assigned to q(1) No rest in p  No zeros in p No need to do key transposition for p

Walk-through Example q P |p(i)-q(j)| 40 20 30 10 20 21 25 12 14 28 26 19 15 28 26 12 14 19 9 19 14 17 20 1 5 8 6 8 6 1 11 1 6 3 q 30 9 5 18 16 2 4 9 1 9 4 7 10 11 15 2 4 18 16 11 21 11 16 13 20 1 5 8 6 8 6 1 11 1 6 3 21 25 12 14 28 26 21 31 21 26 23 P

Hints and Caveats Useful hints to implementation Caveats Be aware of the recurrent equation when i=1 or j=1. Pad an extra layer with D(i, j)=inf. for simplified code Caveats The optimum path may not be unique, but the minimum distance is. The last element in the pitch vector does not have to be assigned to the last music note.

Example Singing clip of 小星星 MIDI file of 三輪車 Alignment path: Pitch vectors:

Example Singing clip of 小星星 MIDI file of小星星 Alignment path: Pitch vectors:

Retrieval Result Other considerations Key transposition Anchor point Music note duration Min distance!