1. Optical Music Recognition
Ichiro Fujinaga
McGill University
2003

2. Content Optical Music Recognition Levy Project Gamera
Levy Sheet Music Collection
Digital Workflow Management
Gamera
Guido / NoteAbility

3. Optical Music Recognition (OMR)
Trainable open-source OMR system in development since 1984
Staff recognition and removal
Run-length coding
Projections
Lyric removal / classifier
Stems and notehead removal
Music symbol classifier
Score reconstruction
Demo

4. OMR: Classifier Connected-component analysis Feature extraction, e.g:
Width, height, aspect ratio
Number of holes
Central moments
k-nearest neighbor classifier
Genetic algorithm

5. Overall Architecture for OMR
Image
File
Staff removal
Segmentation
Recognition
K-NN Classifier
Output
Symbol Name
Optimization
Genetic Algorithm
K-nn Classifier
Knowledge Base
Feature Vectors
Best
Weight Vector
Off-line

6. Lester S. Levy Collection

7. Lester S. Levy Collection
North American sheet music (1780–1960)
Digitized 29,000 pieces
including “The Star-Spangle Banner” and “Yankee Doodle”
Database of:
text index records
images of music (8bit gray)
lyrics (first lines of verse and chorus)
color images of cover sheets (32bit)

8. Digital Workflow Management
Reduce the manual intervention for large-scale digitization projects
Creation of data repository (text, image, sound)
Optical Music Recognition (OMR)
Gamera
XML-based metadata
composer, lyricist, arranger, performer, artist, engraver, lithographer, dedicatee, and publisher
cross-references for various forms of names, pseudonyms
authoritative versions of names and subject terms
Music and lyric search engines
Analysis toolkit

9. The problem Suitable OCR for lyrics not found
Commercial OCR systems are often inadequate for non-standard documents
The market for specialized recognition of historical documents is very small
Researchers performing document recognition often “re-invent” the basic image processing wheel

10. The solution
Provide easy to use tools to allow domain experts (people with specialized knowledge of a collection) to create custom recognition applications
Generalize OMR for structured documents

11. Introducing Gamera
Framework for creation of structured document recognition system
Designed for domain experts
Image processing tools (filters, binarizations, etc.)
Document segmentation and analysis
Symbol segmentation and classification
Feature extraction and selection
Classifier selection and combiners
Syntactical and semantic analysis
Generalized Algorithms and Methods for Enhancement and Restoration of Archives

12. Features of Gamera Portability (Unix, Windows, Mac)
Extensibility (Python and C++ plugins)
Easy-to-use (experts and programmers)
Open source
Graphic User Interface
Interactive / Batchable (scripts)

13. Architecture of Gamera
Graphic User Interface (wxWindows)
Scripting Environment (Python)
Plugins (Python)
Automatic Plugin Wrapper (Boost)
Plugins (C++)
GAMERA Core (C++)

14. Example of C++ Plugin // Number of pixels in matrix
#include “gamera.hh”
#ifdef __area_wrap__
#define NARGS 1
#define ARG1_ONEBIT
#endif
using namespace Gamera;
template <class T>
feature_t area(T &m) {
return feature_t(m.nrows() * m.ncols()); }

15. Example of Python Plugin
// This filters a list of CC objects
import gamera
def filter_wide(ccs, max_width):
tmp = []
for x in ccs:
if x.ncols() > max_width:
x.fill_matrix(0)
else:
tmp.append(x)
return tmp

16. Gamera: Interface (screenshot in Linux)

17. Gamera: Interface (screenshot in Linux)

18. Histogram (screenshot in Linux)

19. Thresholding (screenshot in Linux)

20. Thresholding (screenshot in Linux)

21. Staff removal: Lute tablature

23. Classifier: Lute (screenshot in Linux)

24. Staff removal: Neums

25. Classifier: Neums (screenshot in Linux)

26. Greek example

27. GUIDO Music Notation Format H. Hoos, K. Renz, J. Kilian
“A formal language for score-level representation”
Plain text: readable, platform independent
Extensible and flexible
Adequate representation
NoteServer: Web/Windows
GUIDO/XML
NoteAbility (K. Hamel)

28. GUIDO: An example {
[ \beamsOff | \clef<"treble"> \key<"D">
f#*1/8. g*1/16 |
a*1/4. d2*1/8 d*1/4. c#*1/8 |
e1*1/2 _*1/4 f#*1/8. g*1/16 |
c#2*1/4. b1*1/8 a*1/4. g*1/8 |
| e#*1/2 f#*1/4 f#*1/8. g*1/16 |
e1*1/2 _*1/4 f#*1/8 g |
c#2*1/4. b1*1/8 a*1/4. c#*1/8 ], …

29. NoteAbility Demo

30. Conclusions
Gamera allows rapid development of domain-specific document recognition applications
Domain experts can customize and control all aspects of the recognition process
Includes an easy-to-use interactive environment for experimentation
Beta version available on Linux
OS X version in preparation

31. Projections
X-projections
Y-projections
back