1. Issues in Automatic Musical Genre Classification Cory McKay

2. Introduction to musical genre Practical importance Practical importance Radio stations Radio stations Libraries Libraries Retailers Retailers Theoretical importance Theoretical importance How we construct genre taxonomies How we construct genre taxonomies Mechanisms we use to classify music Mechanisms we use to classify music Mechanisms we use to distinguish between categories Mechanisms we use to distinguish between categories

3. Introduction to musical genre No universally accepted set of categories No universally accepted set of categories Genre descriptions are rarely consistent, comprehensive, clear or objective Genre descriptions are rarely consistent, comprehensive, clear or objective Genre constructed by a complex interaction of Genre constructed by a complex interaction of Marketing strategies Marketing strategies Historical conventions Historical conventions Choices made by music librarians, critics and retailers Choices made by music librarians, critics and retailers Interactions of groups of musicians and composers Interactions of groups of musicians and composers

4. Introduction to musical genre Difficulties with classifying by musical genre Difficulties with classifying by musical genre What categories should be used? What categories should be used? What are the boundaries between categories? What are the boundaries between categories? How are different categories related to each other? How are different categories related to each other? What are characteristics of a particular genre? What are characteristics of a particular genre? What genre(s) do individual pieces belong to? What genre(s) do individual pieces belong to? Genres constantly changing and being created Genres constantly changing and being created Main problems in automatic genre classification Main problems in automatic genre classification Which features should be used? Which features should be used? What taxonomy should be used? What taxonomy should be used?

5. Symbolic vs. audio representation Using a symbolic representation of music rather than an audio representation Using a symbolic representation of music rather than an audio representation Allows one to think of music in terms of musical features rather than signal processing features Allows one to think of music in terms of musical features rather than signal processing features Also allows one to classify scores for which no audio recordings are available Also allows one to classify scores for which no audio recordings are available Future advances in automatic transcription systems will allow use of both types of features Future advances in automatic transcription systems will allow use of both types of features

6. Feature extraction Features: Features: Characteristic pieces of information that can be extracted from music and used to describe or classify it. Characteristic pieces of information that can be extracted from music and used to describe or classify it. Features are very important, as they are the only percepts available to classification systems. Features are very important, as they are the only percepts available to classification systems. Want features that demonstrate differences between categories. Want features that demonstrate differences between categories.

7. Feature extraction Sophisticated theoretical analyses Sophisticated theoretical analyses Too genre-specific Too genre-specific Automatic analysis often an unsolved problem Automatic analysis often an unsolved problem Want features that can be represented as simple numbers that can be fed to classification system. Want features that can be represented as simple numbers that can be fed to classification system. Want features with musicological meaning if possible. Want features with musicological meaning if possible. Want a large catalogue of features so that classifier can choose ones best suited to particular types of classification and sub-classification (hierarchal). Want a large catalogue of features so that classifier can choose ones best suited to particular types of classification and sub-classification (hierarchal).

8. Feature extraction Have devised 160 features based on: Have devised 160 features based on: Instrumentation Instrumentation Texture Texture Rhythm Rhythm Dynamics Dynamics Pitch Statistics Pitch Statistics Melody Melody Chords Chords Scope of these features not limited to genre classification. Scope of these features not limited to genre classification. Could be used for a variety of classification, clustering and analysis tasks. Could be used for a variety of classification, clustering and analysis tasks.

9. Feature extraction Future research: extract non-musical features Future research: extract non-musical features Lyrics Lyrics Clothing Clothing Album art Album art etc. etc. Research of Whitman & Smaragdis (2002) a good start in this direction. Research of Whitman & Smaragdis (2002) a good start in this direction.

10. Automatic classification techniques Three main automatic classification paradigms: Three main automatic classification paradigms: Expert Systems: Use pre-defined rules to process features and arrive at classifications. Expert Systems: Use pre-defined rules to process features and arrive at classifications. Require explicit a priori knowledge of rules Require explicit a priori knowledge of rules Great deal of effort required to change once implemented Great deal of effort required to change once implemented Unsupervised Learning: Cluster the data based on similarities that the systems perceive themselves. No model categories are used. Unsupervised Learning: Cluster the data based on similarities that the systems perceive themselves. No model categories are used. Categories generated “objective” and not likely to correspond to categories used by humans Categories generated “objective” and not likely to correspond to categories used by humans

11. Automatic classification techniques Supervised Learning: Attempt to formulate classification rules by using machine learning techniques to train on model examples. Previously unseen examples are classified into one of the model categories using the patterns learned during training. Supervised Learning: Attempt to formulate classification rules by using machine learning techniques to train on model examples. Previously unseen examples are classified into one of the model categories using the patterns learned during training. Require a pre-defined taxonomy and pre-classified training examples Require a pre-defined taxonomy and pre-classified training examples Supervised learning is the best option for the particular problem of genre classification Supervised learning is the best option for the particular problem of genre classification Several possible implementations: nearest neighbor, neural networks, induction trees, etc. Several possible implementations: nearest neighbor, neural networks, induction trees, etc.

12. Forming genre taxonomies Using hierarchal taxonomy allows the inclusion of both broad and specialist categories Using hierarchal taxonomy allows the inclusion of both broad and specialist categories Could devise rational but artificial categories Could devise rational but artificial categories Not realistic and therefore not useful Not realistic and therefore not useful Experimental approach Experimental approach Music industry categories (e.g. Billboard, Grammies, etc.) Music industry categories (e.g. Billboard, Grammies, etc.) Specialty shows on TV and radio Specialty shows on TV and radio Specialist interviews (DJs, music reporters, etc.) Specialist interviews (DJs, music reporters, etc.) Retailers, including on the Internet Retailers, including on the Internet

13. Forming genre taxonomies Data-mining techniques Data-mining techniques Computers automatically search text resources on the web and attempt to form categories and correlations Computers automatically search text resources on the web and attempt to form categories and correlations Holds a great deal of potential, but is difficult to implement and still untested Holds a great deal of potential, but is difficult to implement and still untested

14. Existing automatic genre classification systems Most experiments to date have been with audio rather than symbolic recordings Most experiments to date have been with audio rather than symbolic recordings Success rates of between 61% to 93% when dealing with between 3 and 10 categories Success rates of between 61% to 93% when dealing with between 3 and 10 categories Only a few studies of symbolic classification Only a few studies of symbolic classification 63% to 84% for between 2 and 3 categories 63% to 84% for between 2 and 3 categories

15. Classification experiment Did initial experiment to test viability of symbolic classification Did initial experiment to test viability of symbolic classification Used 225 MIDI recordings divided into 3 parent genres and 9 sub-genres: Used 225 MIDI recordings divided into 3 parent genres and 9 sub-genres: Classical Classical Baroque, Romantic, Modern Baroque, Romantic, Modern Jazz Jazz Swing, Cool, Funky Swing, Cool, Funky Pop Pop Rap, Country, Punk Rap, Country, Punk Categories were just roughly chosen for test purposes Categories were just roughly chosen for test purposes Could just have easily used other formats, such as Humdrum or GUIDO Could just have easily used other formats, such as Humdrum or GUIDO

16. Classification experiment Performed classification with 8 neural networks and a coordination system. Performed classification with 8 neural networks and a coordination system. Factors increasing difficulty: Factors increasing difficulty: Only 20 recordings per genre were used for training for each run to represent a wide range of musics within each category: increased difficulty. Only 20 recordings per genre were used for training for each run to represent a wide range of musics within each category: increased difficulty. Only 20 features were implemented Only 20 features were implemented

17. Classification experiment Average success rates: Average success rates: 85% for parent genres 85% for parent genres 58% for sub-genres 58% for sub-genres Results were fairly consistent across training runs Results were fairly consistent across training runs These rates comparable to existing audio classification systems using similar numbers of categories and better than existing systems using symbolic data. These rates comparable to existing audio classification systems using similar numbers of categories and better than existing systems using symbolic data. Encouraging Encouraging

18. Classification experiment

19. Future improvements: Future improvements: Use realistic taxonomy Use realistic taxonomy Larger training and sample set Larger training and sample set More features More features More sophisticated classification methodology More sophisticated classification methodology Feature selection sub-system for each level of classification hierarchy Feature selection sub-system for each level of classification hierarchy

20. Software interface A user-friendly interfaced is being developed that will be ported to the classification system. A user-friendly interfaced is being developed that will be ported to the classification system. Easy to use and flexible so that it can be used for a variety of research and applied purposes by people with little technical expertise. Easy to use and flexible so that it can be used for a variety of research and applied purposes by people with little technical expertise. Allows user to: Allows user to: Input and edit arbitrary taxonomies and lists of recordings Input and edit arbitrary taxonomies and lists of recordings Choose which features to extract Choose which features to extract Evaluate the usefulness of particular features in different contexts Evaluate the usefulness of particular features in different contexts Evaluate effectiveness of different classification techniques Evaluate effectiveness of different classification techniques Additional features can be designed and added to the software easily and painlessly by anyone with some basic Java programming skills. Additional features can be designed and added to the software easily and painlessly by anyone with some basic Java programming skills.

23. Conclusions Could use system such as this to study Could use system such as this to study Particular taxonomies Particular taxonomies How well different features perform in different contexts How well different features perform in different contexts Differences between and definitions of particular genres Differences between and definitions of particular genres Could easily adapt system to other types of classification Could easily adapt system to other types of classification Composer / performer Composer / performer Historical / geographical / cultural characteristics Historical / geographical / cultural characteristics Personal preferences Personal preferences Practical applications Practical applications On-line musical databases of any kind On-line musical databases of any kind

24. Questions