1. Query by Singing/Humming via Dynamic Programming
J.-S. Roger Jang (張智星)
MIR Lab, CSIE Dept.
National Taiwan University

2. Introduction to Query by Singing/Humming
Query by singing/humming (QBSH, 哼唱選歌)
Goal: Identify a song by singing or humming
Demos
(PC)
(PC)
(Mobile)
Approach to QBSH
Pitch tracking: Convert singing/humming into pitch vector
Retrieval: Find the distance between the pitch vector and each song in the database
Our homework
Explore how we can use dynamic programming (DP) to find the distance in the retrieval part of QBSH

3. Examples of Pitch Vectors and Music Notes
MIDI numbers
Used in MIDI files
AKA semitones
Music note vector
Integer semitones
Example
MIDI file of 小星星
Note vector: [ ]
Pitch vector
Real-number semitone
Example of singing clips
小星星, pitch vector (play)
在那遙遠的地方, pitch vector (play)
Pitch rate = pitch/second

4. Our Task: Optimal Alignment
How to find the distance between a pitch vector p(i), i=1~m, and a note vector q(j), j=1~n?
We need to find the optimal alignment.
This can be achieved by DP.
Example:
The alignment path: (1,1), (2,1), (3,2), (4,2), (5,2), (6,2), (7,3), (8,3), (9,4), (10,4), (11,4), (12,4)
Distance:

5. Three-step Formula of DP for Alignment
Three-step DP formula
Optimum-value function: D(i,j) is the min distance between p(1:i) and q(1:j)
Recurrent equation:
Answer:
Assumption
Anchored at beginning  p(1) is assigned to q(1)
No rest in p  No zeros in p
No need to do key transposition for p

6. Walk-through Example q P |p(i)-q(j)| 40 20 30 10 20 21 25 12 14 28 2619 15 28 26 12 14 19 9 19 14 17 20 1 5 8 6 8 6 1 11 1 6 3 q 30 9 5 18 16 2 4 9 1 9 4 7 10 11 15 2 4 18 16 11 21 11 16 13 20 1 5 8 6 8 6 1 11 1 6 3 21 25 12 14 28 26 21 31 21 26 23 P

7. Hints and Caveats Useful hints to implementation Caveats
Be aware of the recurrent equation when i=1 or j=1.
Pad an extra layer with D(i, j)=inf. for simplified code
Caveats
The optimum path may not be unique, but the minimum distance is.
The last element in the pitch vector does not have to be assigned to the last music note.

8. Example Singing clip of 小星星 MIDI file of 三輪車 Alignment path:
Pitch vectors:

9. Example Singing clip of 小星星 MIDI file of小星星 Alignment path:
Pitch vectors:

10. Retrieval Result Other considerations Key transposition Anchor point
Music note duration
Min distance!