Speaker Associate Professor Ning-Han Liu

What’s MIR  Music information retrieval (MIR) is the interdisciplinary science of retrieving information from music.  MIR is a small but growing field of research with many real-world applications.

Applications of MIR  Recommender systems  Track separation and instrument recognition  Automatic score creation  Automatic categorization  Music generation

Methods used in MIR-Data Source  Music Content Symbolic Data Formats ○ Scores ○ MIDI Digital Audio Formats ○ WAV ○ Mp3  Metadata Song name Singer Performer tags

Methods used in MIR-Feature Representation  Mel-Frequency Cepstral Coefficient (MFCC) is a measure of the timbre of a piece of music.  Chords  Harmonics  Melody  Main pitch  Beats per minute

Methods used in MIR- Statistics and Machine Learning  Music analysis and knowledge representation  Classification, clustering and modeling

Famous Application  iTunes: is a media player computer program, used for playing, downloading, saving, and organizing digital music and video files on desktop or laptop personal computers.

Famous Application  KKBOX

Famous Application  Nike+iPod

Previous research issue I  Query music efficiently Music representation ○ ‘U’,’D’,’S’ ○ Rhythm ○ Chords Index structure ○ Link list ○ Suffix tree Query process ○ Exact matching ○ Approximate matching

Previous research issue I  String Matching Exact String Matching ○ E.g. sol-mi-mi-fa-re-re-do-re-mi-fa-sol-sol-sol ○ Query: do-re-mi ○ To find whether data contain the query melody string Algorithms: KMP, Boyer-Moore Problem ○ Cannot expect users to precisely specify music query ○ Need to retrieve all entire melody string

Previous research issue I  String Matching Approximate String Matching ○ Edit distance is often used as similarity measure ○ E.g. sol-mi-mi-fa-re-re-do-re-mi-fa-sol-sol-sol ○ Query: do-mi matching do-re-mi with edit distance=1

Previous research issue I  String Matching Algorithm Using Index Arbee L.P. Chen (ICMCS 1999)

Previous research issue I  Approximate Matching

Previous research issue I  Query by Music Segments Arbee L.P. Chen (ICME 2000)

Previous research issue I  Query by Music Segments Index: Suffix Tree

Previous research issue II  Music Recommendation Content based filtering (CBF) ○ predicts user preferred data items by matching the representations of the data items relevant to the user Collaborative filtering (CF) ○ uses the correlations between users on the basis of their ratings to predict items for users Mixed method ○ Combine CBF and CF

Previous research issue II  A music recommendation system based on music data grouping and user interests Arbee (ICME 2000)

Previous research issue II

Novel research topic I  Playlist Generation An Intelligent Music Playlist Generator based on the Time Parameter with Artificial Neural Networks (Liu, 2010) N.H. Liu, S.J. Hsieh, C.F. Tsai, "An Intelligent Music Playlist Generator based on the Time Parameter with Artificial Neural Networks," Expert systems with applications. Volume 37, Issue 4, April (2010), Pages (SCI, impact factor=2.9, EI)

 The structure of the mixed artificial neural networks Novel research topic I

Novel research topic II  JoMP: A Mobile Music Player Agent for Joggers based on User Interest and Pace (Liu, 2009) N.H. Liu, H.Y. Kung, "JoMP: A Mobile Music Player Agent for Joggers based on User Interest and Pace," IEEE Transactions on Consumer Electronics, Volume 55, No. 4, Nov, (2009 (SCI, EI)

Novel research topic II  System prototype

Novel research topic III  Intelligent Music Player for Bike Sport Using EEG & GPS Sensors Brainwave earphone (NeuroSky™ Minset) and wearing method. EEG Signal rhythmFrequency α waves8~13 Hz β waves14~30 Hz γ waves> 21 Hz δ wave0.5~3 Hz θ waves4~7 Hz N.H. Liu, H.M. Hsu, H.C. Chu, and S.H. Hsu "Intelligent Music Player for Bike Sport Using Electroencephalogram and Global Positioning System Sensors" Sensor Letters, Volume 11, Number 5, pp (2013) (SCI).

Novel research topic III  The system is based on the Fuzzy Inference System (FIS), combined with EEG and GPS sensors to measure the sport data of cyclists.

Novel research topic IV  Intelligent car audio system in cloud N.H. Liu "Design of an Intelligent Car Radio and Music Player System," accepted by Multimedia Tools and Applications. (2013).(SCI, IF=1.014) DOI: /s z

Novel research topic V  Car audio system with doze prevention

Novel research topic V  Car audio system with doze prevention N.H. Liu, C.Y. Chiang, H.M. Hsu, "Improving Driver Alertness through Music Selection Using a Mobile EEG to Detect Brainwaves" Sensors, 13(7): (2013)(SCI, IF=1.953).

Novel research topic V  Car audio system with doze prevention

Novel research topic VI  Distance estimation methods Methods to calculate a personalized distance measure between different pieces of music based on user preferences. The questions ask the user to rate the similarity between songs selected by the system. N.H. Liu "Comparison of Content-based Music Recommendation Using Different Distance Estimation Methods," Applied Intelligence, Volume 38, issue 2, pp , Mar. (2013).(SCI, IF=1.853)

Novel research topic VI  Weighted squared Euclidean distance function generated from maximum likelihood estimation

Novel research topic VI  Weighted squared Euclidean distance function generated from genetic algorithm Define a fitness function to evaluate the chromosome performance

Novel research topic VI  Distance function generated by genetic programming The fitness function is defined as follows:

Novel research topic VII  Music Selection Interface for Car Audio System In the architecture, users are able to select a genre of music or a playlist from through a 2D interface. Self-organizing map, depending on a personalized distance function and music contents, is utilized to map music tracks to the interface.

Novel research topic VII  Prototype in car N.H.Liu, "Music Selection Interface for Car Audio System Using SOM with Personal Distance Function," EURASIP Journal on Audio, Speech, and Music Processing, 2013:20. (2013) (SCI)

Novel research topic VIII  Query by Singing/Humming When a user cannot remember the title of a song, or its related details, the most direct and convenient method to search for the song is by humming a section of it. The background of the user often influences the genres of the songs being searched. We use the information from a user’s search history, as well as the properties of genres common to users with similar backgrounds, to estimate the genre or style the current user may be interested in based on a probability calculation.

Novel research topic VIII  System flow chart

Novel research topic VIII  The Singing/Humming signals process

Novel research topic VIII  Ranking by User’s Preference

Novel research topic VII  Ranking through Similar Users’ Records naïve Bayesian prediction

Novel research topic VII  User Interface N.H. Liu, "Effective Results Ranking for Mobile Query by Singing/Humming Using a Hybrid Recommendation Mechanism," IEEE Transactions on Multimedia, Aug (2014)(SCI, IF=1.754)