What is automatic music transcription? Transforming an audio signal of a music performance in a symbolic representation (MIDI or score). Aim: This prototype is conceived as a research platform for developing and applying interactive and multimodal techniques to the monotimbral transcription task. Problem decomposition (summary): AUDIO F0 frame by frame estimation Note pitch detection Transcription More accurate problem decomposition (multimodal & interactive) : SIGNAL SCORES Music models Envelope (amplitude) F0 frame by frame Note pitch detection Transcription Tonality Meter Tempo Note onsets Description and Retrieval of Music and Sound Information Descripción y Recuperación de Información Musical y Sonora PROJEC T Operation diagram: New PROJECT Onsets Pulses Notes Onsets Text / Harmony Text / Harmony Rhythm: tempo + meter Rhythm: tempo + meter ANALYSIS (information source) ANALYSIS (information source) INTERACTION with INTERACTION with TRANSCRIPTION based on TRANSCRIPTION based on Spectrogram Onsets Pulses Frames Physical level Musical level off-line melodic and harmonic models Multimodality: it uses three different sources of information to detect notes in a musical audio excerpt: signal, note onsets, and rhythm information. Interactive: Designed to make use of user feedback on onsets, beats, and notes in a left-to- right validation approach: a user interaction validates what remains at the left-hand side, interactions are used to re-compute the rest of the output. Structure overview Signal F0 (in Hz) Piano roll Music score State-of-the-art techniques are far from being accurate, specially in the case of polyphony and multitimbral sounds. So nothing even close to 100% can be expected  User corrections are needed. (off-line) XML file Rhythm Interactions allowed Interface structure: Interaction assistance Menus Play Markers & timing area Tempo and meter area Transcription area: piano roll / score Audio signal area Textual transcription area Chord segmentation area Audio properties Keyboard / staves reference Tonality Rhythm properties Text properties Raw (frame by frame) transcription: Screencast: Based just on harmonic energies in the spectrogram. Smoothed by a frame context. Filtered by a length threshold (in frames). Many short false psoitives and negatives. Spectrogram Pitches in frames Onset-based transcription: Onsets Pitches Spectrogram Onsets impose a segmentation. Only at onsets notes can change. Times are still physical. Transcription is much more accurate. Interaction with onsets affect the transcription Pulse-based transcription: Pulses Notes Spectrogram Beat, tempo and meter are derived from pulses. Transcription is driven by them using a division of beat. Times are now musical. Transcription is score-oriented. A false negative is corrected by the user: This correction solves other FN: Trasncription is recomputed with the new onset Changes are propagated Harmonic analysis (chord segmentation) is provided This work is supported by the Consolider Ingenio 2010 research programme (project MIPRCV, CSD ), the project DRIMS (TIN C02), and the PASCAL2 Network of Excellence (IST ). The authors want to thanks to the people that is involved in this project, specially those who do not appear as Authors of this paper, like Carlos Pérez-Sancho, David Rizo, Javier Sober, José Bernabeu, or Gabriel Meseguer. Acknowledgements: Spectrogram  Frames  Set of pitch candidates  Selection by “salience”  Smoothing in short context  Set of pitches by frame Very short notes can be filtered out by merging or deleting them by parameters controlled by the user. Signal  Rate of change of pitched energy  Threshold  Onsets  Segmentation  Segment transcription  Set of pitches by segment Signal  Energy fluctuations  Pulses  Beats and Tempo  Quantization  Quantized transcription  Notes (pitch and duration) Note durations acquire musical meaning. Required if a music score is aimed as the final output, otherwise only a piano roll can be obtained. Frame-based transcription: Onset-based transcription: Pulse-based transcription: Interactions: Implemented or planned: onsets (add, remove, edit), pulses (modify beat and meter), notes (add, remove, edit), and harmony (chord segmentation). Transcription modes: A Multimodal Music Transcription Prototype First steps in an interactive prototype development Tomás Pérez-García, José M. Iñesta, Pedro J. Ponce de León, Antonio Pertusa Universidad de Alicante, Spain Warning: This is a project in its very early stage, so there are many functionalities still not implemented and it is far from being bug-free. More information: At a video screencast and an on-line demo are available.