1. Automatic Transcription of Polyphonic Music
Jana Eggink
Supervisor: Guy J. Brown
University of Sheffield
{j.eggink

2. me
Master in systematic musicology with computer science as a second subject, University of Hamburg, Germany, 2001
Interested in perception and computational auditory scene analysis
And, of course, music
Jana Eggink, Sheffield, UK
Automatic Transcription of Music

3. Automatic Transcription of Music
General idea: get a score from an audio file
• Useful for: automatic music indexing and analysis, detection of copyright infringement, ‘query-by-humming’ systems...
Many subtasks: F0 estimation, onset detection, instrument recognition
flute
Jana Eggink, Sheffield, UK
Automatic Transcription of Music

4. Missing Data
Ideas from speech recognition and speaker identification in noisy environments
Missing data: ignore time frequency regions dominated by interfering sound sources
Determination of unreliable of missing features based on F0s of interfering tones
Works well with two simultaneous sounds
But problems already with F0 estimation with 3 or 4 concurrent tones
Jana Eggink, Sheffield, UK
Automatic Transcription of Music

5. Change of Focus…
Instead on using missing data masks based on interfering sound sources,
concentrate on the signal you want to identify
Task: Identify the solo instrument in accompanied sonatas and concertos
Jana Eggink, Sheffield, UK
Automatic Transcription of Music

6. Identify Solo Instrument
• Instrument sounds are harmonic, energy is concentrated in partials which are least likely to be masked by other sounds
• Only most dominant F0 needs to be identified
• Features based only on frequency position and power of lowest 15 partials
• Statistical recogniser (GMMs) trained on monophonic music
flute
clarinet
oboe
violin
cello
audio signal
recog-niser
features
F0 and partials
Jana Eggink, Sheffield, UK
Automatic Transcription of Music

7. Results Instrument Identification
• Solo instrument with accompaniment (piano or orchestra), commercially available CDs, 90 examples, 2-3 min. each
• Instrument 86% correct, as good as other systems that only work with monophonic music
Jana Eggink, Sheffield, UK
Automatic Transcription of Music

8. Find the Melody (assuming the solo instrument is known)
• Extract multiple F0 candidates
• Include additional knowledge about instrument range, tone duration, likely interval transitions to pick correct candidate
F0 strength (~loudness)
F0 likelihood (absolute
frequency | instrument range)
instrument likelihood
(recogniser output)
LOCAL KNOWLEDGE
TEMPORAL KNOWLEDGE
tone length
interval transitions
find most likely
‘path’ through time-
frequency space of
F0 candidates
AUDIO
F0 candidates
silence estimation (only
accompaniment?)
MELODY
Jana Eggink, Sheffield, UK
Automatic Transcription of Music

9. Results
• Improving the number of correct frames from 40% to 54%, greatly reducing the number of spurious tones
• Beginning of Mozart’s Clarinet Concerto, manually annotated F0s (gray) and estimated melody (black)
Melody based on strongest F0
time (frames)
F0 (Hz)
F0 (Hz)
Melody based on knowledge integrating path finding
time (frames)
Jana Eggink, Sheffield, UK
Automatic Transcription of Music

10. Experiences in the Network
Many techniques used for speech recognition are (with modifications) transferable to musical applications, but are often not well known among people who work with music, very good training in Sheffield
Insight into other related areas in the HOARSE meetings,
Which are also great for personal contacts
Jana Eggink, Sheffield, UK
Automatic Transcription of Music