NOOJ 0.1 Max Silberztein Université de Franche-Comté 6th INTEX Workshop Sofia, Bulgaria, May 2003

NOOJ v0.1 Rewritten entirely Fully compatible with INTEX 4.3x Corpus processor WEB support Support for any Text Encoding & Format Object-Oriented linguistic engine Dynamic Programming with Published methods

Corpus Processor A Corpus is a set of homogeneous files: same language, same linguistic resources A corpus may include tens of thousand small (i.e. WEB pages) or large files (i.e. Le Monde, 1 year), stored anywhere Different corpora can share certain text files with no extra cost

Text Encoding & Format support Native support means NO FILE CONVERSION -- for TXT files: Windows Default (8 or 16 bits), DOS & ISO (any codepage), Unicode (7 bits, 8 bits, 16 bits, low and big endian) -- for HTML (any encoding) and RTF (ASCII & Unicode) files -- for Microsoft Word files, any version including Apple -- No limit: XML, PDF, LaTeX, Outlook...

WEB support Nooj includes a WEB crawler that can import WEB sites Exploration is performed up to a user- declared depth, or until the WEB site is fully explored (danger!) Indirections are processed during the exploration; they may produce empty text files (i.e. no text unit in the WEB page).

OO linguistic engine the engine can be easily adapted –inheritance means that one can build quickly a new module by inheriting another module’s properties, by default –override means that one needs to provide description only for the methods that perform tasks differently dynamic programming: –NOOJ loads parts of the linguistic engine only when needed –describing extremely specific phenomena or behavior carries no cost for the overall architecture open interface: –Applications access NOOJ from command-line programs (i.e. SHELL), as well as from object & class methods, from user’s programs or from other applications (such as Microsoft Office or Adobe Acrobat).

Français.il namespace Nooj { public class Français: Language // Language is a virtual class; MUST BE OVERRIDEN { // tokenizer public override static bool rightToLeft () { return false; } // true for Arabic public override static bool oneCharPerToken () { return false; } // true for Chinese public override static bool transcription () { return false } // true if text processed != text displayed... // tokens’ properties public override static bool upperCaseLetter (char letter) {... } public override static bool lowerCaseLetter (char letter) {... } public override static bool lowerCassForm (string token) {... }... // dictionary & list lookup & match public override static int compareForms (string wform1,string wform2) {... } public override static int matchForm (string wform,string entry) {... }... // localization... }

Perspectives Text processing is fully operational; Linguistic engine is ½ operational Morphological module by September Dictionaries (new types & tools) by Sept. Grammars developing tools (new types and tools) by end of 2003 => Alpha version by the end of 2003 All 4.3x functionalities by may 2004