Polyphonic Queries A Review of Recent Research by Cory Mckay

Overview Introduction to polyphonic music queries Review of five recent systems Conclusions

Melodic Fragments as Queries Melodic fragments are a useful way of searching electronic music databases Convenient for humans to remember musical phrases Easy to enter musical fragments Monophonic: query by humming Polyphonic: notation software

Search Requirements Want searches to return all occurrences of a given set of notes in a database May also want records that contain similar sets of notes Should be able to deal with incomplete or partially erroneous queries Ideally, want to search music stored in Symbolic formats (MIDI, scores, sketches) Raw audio data

Monophonic Databases Has been some success with special case of monophonic databases Andreas Kornstadt’s Themefinder Roger J. MacNab’s Meledex Searching polyphonic databases is much more difficult

Polyphonic Databases: Problems Notes may begin simultaneously. Makes it impossible to outline an unambiguous sequence of events Multiple voices, with varying roles and relevance to particular queries Hard to deal with both symbolic and raw audio representations: Monophonic: can just transcribe audio Polyphonic: no good transcription system

Current Polyphonic Systems Wide-spread systems rely on meta-data This no good for searches of melodic fragments Currently no widely accepted content- based system A number of papers on topic have recently been published

Wiggins et al. (2002) Designed a new algorithm called SIA(M)ESE For making transposition-invariant queries Matches a query even if there are events in a score being searched that separate musical events in the query Assumes that both queries and database files are in symbolic form and are accurate This limits generality of algorithm No implementation of the algorithm given

Doraisamy & Rüger (2001) Polyphonic queries Partial queries permitted Symbolic queries Symbolic records Pitch and rhythm features

Doraisamy & Rüger (2001) N-grams are produced by converting notes into interval-based representation and grouping intervals into subdivisions of length n using a gliding window Leads to transposition-invariant data N-grams work well with monophonic queries

Doraisamy & Rüger (2001) To deal with potential for simultaneous note onsets in polyphonic music, constructs exhaustive “melodic strings”: Divide piece into overlapping windows of n adjacent onset times Find all possible combinations of onsets within each window

Doraisamy & Rüger (2001) Incorporates rhythmic information as well as intervals into each n-gram window Done by calculating ratio of time differences between adjacent note onsets: Ratio i = (Onset i+2 – Onset i+1 ) / (Onset i+1 – Onset i ) Avoids need to quantize events based on a predetermined beat duration By using onsets only, avoids needing to determine the duration of notes, which can be difficult to detect in raw audio recordings

Doraisamy & Rüger (2001) N-grams converted into text-based representations to allow use of text-based search engines Interval and rhythmic ratio histograms constructed in order to search for patterns in each piece

Doraisamy & Rüger (2001) Tested using a database of 3096 MIDI representations of classical music Studied effects of varying window sizes, bin ranges and query lengths 95% success rate with window sizes of 4 onset times, variable bin ranges and queries involving 50 notes 74% with query lengths of ten notes 65% with queries containing errors

Doraisamy & Rüger (2001) Evaluation: Performs well under ideal conditions Databases or queries containing raw audio not considered Only transposition invariant searches possible Undesirably long query lengths necessary Only classical music records tested Successful in showing the potential utility of n-grams Most recent work uses a variant of this n-gram approach

Doraisamy & Rüger (2002) Monophonic queries Partial queries permitted Symbolic queries Symbolic records Pitch and rhythm features

Doraisamy & Rüger (2002) Focused on monophonic queries because of query by humming N-grams particularly appropriate for error- prone queries resulting from query by humming One or two mistakes only lead to a few incorrect n-grams among a larger number of correct ones

Doraisamy & Rüger (2002) Used same overall design as previous system Includes more sophisticated error models to test effects of query inaccuracies Database expanded to include popular music as well as classical music 80% of the relevant compositions were returned in first 15 hits, on average

Doraisamy & Rüger (2002) Evaluation N-grams are an effective and error-tolerant tool for searching polyphonic music with monophonic queries Improvements still need to be made Queries still symbolic, so true applicability of system to query by humming not tested

Pickens et al. (2002) Polyphonic queries Full-length queries only Audio queries Symbolic records Harmonic features

Pickens et al. (2002) Relies on transcription algorithms to transform audio queries into symbolic form Relies on error tolerance of searches to compensate for transcription errors Two types of polyphonic transcription systems used, including blackboard

Pickens et al. (2002) Transcribed queries and database records analyzed and stored using a harmonic modelling module Characterizes pieces by mapping chords to a probability distribution Breaks into sequences of independent note sets Applies smoothing procedure to sets Markov models created from smoothed sets

Pickens et al. (2002) Performing searches: Scoring function used to compare query models with each model stored in database Dissimilarity scores produced Allows search hits to be ranked

Pickens et al. (2002) Tested with database containing 3150 classical piano pieces Queries consisted of full-length audio recordings On average, searches assigned a rank of between 2 and 6 to the correct database record Moderately successful at matching variations of a piece: on average, 3 of top 5 hits relevant

Pickens et al. (2002) Evaluation Can use polyphonic audio queries Limited by effectiveness of its transcription systems and error tolerance of the query system System only tested on piano music Full-length queries limit usefulness Only one feature used

Song, Bae & Yoon (2002) Monophonic queries Partial queries permitted Audio queries Audio records Melodic features

Song, Bae & Yoon (2002) Intended to be used with query by humming Avoids disadvantages of automated transcription by mapping audio data directly to “mid-level” melody-based feature set description Contrasts with approach of first transcribing audio data and then extracting features from this high- level symbolic representation

Song, Bae & Yoon (2002) “Mid-level” representation produced by processing audio frames using a five- step process

Song, Bae & Yoon (2002) Instead of making definite decision on which notes were present, as a blackboard system would have done, vectors of all possible notes were kept for each audio segment A DP-matching method was used during searches so that potentially error-prone patterns of different lengths could be compared

Song, Bae & Yoon (2002) Tested by attempting to match 176 hummed samples to 92 short extracts (15-20 seconds long) of popular Korean and Western popular songs Resulted in exact matches roughly 43% of the time and a match in the top ten from 69% to 76% of the time (depending on window size) Search time varied from 3 to 14 seconds

Song, Bae & Yoon (2002) Evaluation Performance relatively poor Only monophonic queries possible Only tested using a database of short recordings This was only system using audio data for both queries and database records

Conclusions No truly viable systems produced yet Promising approaches have been proposed Recurring problems: None of systems can deal with both symbolic and audio data None have been tested with both polyphonic and monophonic queries Tend to require long queries to achieve good results Searches do not allow much flexibility (e.g. must be transposition invariant)

Conclusions Difficult to compare systems because they each use different performance evaluation metrics Limited data sets used during testing

Conclusions Possible improvement: use a greater number of feature classes Aside from Doraisamy & Rüger, all systems discussed limited themselves to one feature class Harmonic, melodic, timbral and rhythm- based features could all prove useful Would allow much more flexible searches