2015/10/241 Query by Tapping 敲擊選歌 J.-S. Roger Jang ( 張智星 ) Multimedia Information Retrieval Lab CS Dept., Tsing Hua Univ., Taiwan

Query by Tapping zGoal: yMusic search based on uses’ tapping (at notes’ onsets) over the microphone/keyboard zCharacteristics yOnly note duration is used for comparison, note pitch is discarded. yA hard task for human to recognize (which is different from query by singing/humming) xTry this…

Query by Tapping zGoal: yMusic search based on uses’ tapping (at notes’ onsets) over the microphone/keyboard zCharacteristics yOnly note duration is used for comparison, note pitch is discarded. yA hard task for human to recognize (which is different from query by singing/humming) xTry this…

Query by Tapping zChallenges: yUsers is unlikely to use the same tempo as the intended song yUsers tend to lose notes instead of gaining ones yWe have about 13,000 songs in the database zMajor approach: yA distance measure based on dynamic programming

Flowchart of Query by Tapping Ryhthm Extraction Microphone Input Off-line processing Note duration extraction About MIDI songs On-line processing DP-based comparison Normalization Query results

Feature Extraction via Microphone yMicrophone input: yAfter frame blocking, energy computation, and thresholding:

Performance Evaluation of Onset Detection zsimSequence.m precision=3/6=0.5 recall=3/5=0.6 f-measure=2pr/(p+r)=0.5455

Similarity Comparison with Songs in Database zA fast method based on IOI ratios yCompute the IOI ratios for both query and db IOI vectors yCompute the Euclidean distance these two ratio vectors

Music Note Alignment t(3) t: test (input) IOI vector r: reference IOI vector r(1) t(1) t(2) r(2) r(3) Normalization Alignment by DP trtrtr

-10- Normalization zNormalization to have (Multiplication of 1000 to guarantee high resolution in fixed-point computation.) z

-11- Dynamic-programming-based Distance i j t(i-2) r(j-1) t: test IOI vector of length m r: reference IOI vector of length n Recurrent relation: r(j-2) t(i-1)t(1) t(2) r(1) r(2)

-12- Experimental Environment z269 test wave files of tapping clips y9 contributors (7 males, 2 females) yWave length: 15 seconds yWave format: PCM, 11025Hz, 8bits, Mono yStart position: Beginning of a song zEnvironment yPentium III 800, 256MB RAM zDatabase y11,744 MIDI files

-13- Test Results Using Clips of 15 Seconds Average response time: 3.42 seconds (29.98 notes) zRecognition rates: yTop-1 (top %): 15% yTop-10 (top 0.085%): 51% yTop-100 (top 0.85%): 80%

-14- Error Analysis zErrors analysis of low-ranked clips ySome users cannot tap consistently through 15 seconds yFeature extraction is not robust enough to handle noisy input. ySome MIDI files are not faithful rendition of the original tunes. yUsers cannot keep up with short consecutive notes.

-15- Recog. Rates w.r.t. Tapping Duration zTop-100 and 1000 curves level off after 10 seconds. zTop-100 curve does not go up monotonically. Top-100 Top-10 Top-1000

-16- Demo zNo. of MIDI files: 12982

-17- Partial List of Songs  All I have to do is dream  You are my sunshine  Beautiful Sunday  Do Re Mi  Feelings  A time for us  Love is blue  Let it be me  My way  Love story  More than I can say  Only you  Rain and tears  Rhythm of the rain  Rose Rose I love you  The sound of silence  Unchained melody  We are the world  Yesterday  I just call to say I love you  Close to you  Mr. Lonely  Ben  Hey Jude  Donna Donna  Sealed with a kiss

-18- Potential Applications zInteractive toys zBeat-tracking training and games zSong retrieval in noisy karaoke bars

-19- Conclusions zOur MIR system is the first one with query- by-tapping capability. zRhythm-based search can be used in conjunction with pitch-contour-based search to achieve a better recognition rate.

-20- Future Work zSearch scope expansion yHow to retrieve MP3 or CD music directly? zScale-up by hierarchical filtering method yHow to deal with database with 100,000 songs? yWhat if the user tap from anywhere in the middle of a song?

-21- 敲擊選歌  目標：  以敲擊的方式，敲出一首歌的節拍，並從歌曲 資料庫中找出這首歌  技術困難點：  敲快、敲慢、多敲、少敲、從頭敲、從中間敲  如何大量歌曲的快速比對  方法： Dynamic programming

-22- 敲擊選歌  展示  /users/jang/demo/tapping/go_tap.m  All I have to do is dream 、 You are my sunshine 、 Puff  平安夜、原來你什麼都不想要、用心良苦、青春舞 曲、榕樹下、雙人枕頭、萍聚、夜來香、月亮代表 我的心、花心、小小羊兒要回家  應用面  玩具