2. 1 OMRAS : Online Music Recognition And Searching Donald Byrd School of Music Indiana University 16 April 2002

3. 2 Overview 1. Introduction and Motivation 2. Background 3. OMRAS Research: Audio and Events 4. OMRAS Research: Music Notation 5. Evaluation of IR Systems 6. Conclusions

4. 3 1. Introduction: About OMRAS Funding from JISC (England) and DLI, Phase 2 (U.S.) 1999 - 2002 English team led by Tim Crawford –King’s College London (Music, Computing Science) –Queen Mary, University of London (Electronic Engineering) U.S. team led by Donald Byrd –University of Massachusetts, Amherst (Center for Intelligent Information Retrieval) –Indiana University, Bloomington (School of Music)

5. 4 Introduction and Motivation Music holdings of Library of Congress: over 10M items => content-based searching is very valuable –Includes over 6M pieces of sheet music and tens/hundreds of thousands of scores of operas, symphonies, etc.: all notation Three basic forms (representations) of music –Audio: best for general public Info on >230K CD’s at All Music Guide (www.allmusicguide.com) –MIDI files: often best for many musicians, especially pop, rock, film/TV Hundreds of thousands of MIDI files on the Web –Music Notation: often best for other musicians (even amateurs) and music researchers Smart car radio (research now in progress)

6. 5 2. Background: Basic Representations of Music and Audio Audio (e.g., CD, MP3): like speech Time-stamped Events (e.g., MIDI file): like unformatted text Music Notation: like text with complex formatting

7. 6 Background: General Music comes in three basic representations/forms: (a) Audio (unstructured) (b) Time-stamped events (c) Music notation (highly-structured, symbolic) OMRAS goal: search music in all three forms from query in any form Forms require different approaches Why not reduce all to type b (middle form)? –a -> b is an open research problem –c -> b loses important information “Cross-representation” IR: similar to cross-lingual text IR

8. 7 Why is Music IR Hard? 1. Units of meaning: not clear anything in music is analogous to words (all representations) 2.Polyphony: “parallel” independent voices, something like characters in a play (all representations) 3.Recognizing notes (audio only) 4.Indexing, etc.

9. 8 Independent Voices in Music (Problem 2) J.S. Bach: “St. Anne” Fugue, beginning

10. 9 Independent Voices in Text MARLENE. What I fancy is a rare steak. Gret? ISABELLA. I am of course a member of the / Church of England.* GRET. Potatoes. MARLENE. *I haven’t been to church for years. / I like Christmas carols. ISABELLA. Good works matter more than church attendance. --Caryl Churchill: “Top Girls” (1982), Act 1, Scene 1 M: What I fancy is a rare steak. Gret? I haven’t been... I: I am of course a member of the Church of England. G:Potatoes. Performance (time goes from left to right):

11. 10 3. OMRAS Research: Audio-degraded Music IR Experiment Before (original audio recording) After (audio -> MIDI -> audio) Started with recording of 24 preludes and fugues by Bach Colleagues in London did polyphonic music recognition Audio -> events “an open research problem” Results vary from excellent to just recognizable One of worst-sounding cases is Prelude in G Major from the Well-Tempered Clavier, Book I

12. 11 OMRAS Audio-degraded Music IR Experiment Jeremy Pickens and Tim Crawford used MIDI form of 24 preludes and fugues as queries against “CCARH Plus” database of c. 3000 pieces in MIDI form Outcome for G-major Prelude: the actual piece was ranked 1st! Average outcome: actual piece ranked c. 2nd

13. 12 4. OMRAS Research: Music Notation Conventional Music Notation (CMN) is –Very successful, and useful to many people, but... –Specialized: useless to most –Very complex CMN often best form for musicians (even amateurs) –CMN sometimes essential for music researchers Searching CMN is obviously important... But almost no work on it so far! Why? –Specialized audience, complexity –Lack of test collections Prospects for solving problems are good

14. 13 NightingaleSearch Nightingale ® is high-end commercial music editor NightingaleSearch inherits all normal functionality of Nightingale Searching commands use “Search Pattern” score as query Find next (“editor”) or find in database (“IR”) searching –Find in database is exact- or best-match Options: match pitch, match duration, etc. Does passage-level retrieval

15. 14 *Bach: “St. Anne” Fugue, with Search Pattern

16. 15 NightingaleSearch in Action With BachStAnne, exact-match OK, but... Best-match (threshhold 2) gives much better recall (of passages) with no loss of precision A harder example: user looking in a digital music library for “Twinkle, Twinkle, Little Star”(demo with a tiny personal library)

17. 16 *Mozart: Variations for piano, K. 265, on “Ah, vous dirais-je, Maman”

18. 17 *Suzuki: “Twinkle” Variations

19. 18 5. Evaluation of IR Systems In text IR, standard is TREC (Text REtrieval Conferences) –Sponsored by NIST, other U.S. agencies –Uses Cranfield-model evaluation: Database(s) Information needs against database(s) Relevance judgments for information needs against database(s) –Judgments and information needs by same person In music IR, almost nothing yet –Cranfield method is promising—but need databases, information needs, relevance judgments!

20. 19 6. Conclusions Content-based music IR is decades behind text IR (no surprise) Music retrieval is in mid childhood One big problem is machinery to evaluate research Another is lack of databases Don’t need perfection to be useful: cf. text IR! Technology is ready for limited use