1. AutoEval and Missplel: Two Generic Tools for Automatic Evaluation Johnny Bigert, Linus Ericson, Anton Solis Nada, KTH, Stockholm, Sweden Contact: johnny@kth.se www.nada.kth.se/theory/humanlang/tools.html

2. Manual evaluation Time-consuming, tedious, error-prone Computers are good at repetitive tasks, humans are not Unavoidable in some situations

3. Automatic evaluation Cheap, fast, accurate, easily reproducible Incorporated in the development of most NLP system

4. Automatic evaluation AutoEval: simplifies the construction of (NLP system) evaluation Missplel: introduces human-like errors into text

5. AutoEval "I write evaluation code myself in all our NLP projects" "Why would I need AutoEval?"

6. AutoEval Our point exactly Repetition of: Input and output file handling XML parsing and XML output Error handling, malformed input Data storage, management and processing

7. AutoEval Features — avoids repetition: Handles input (XML/structured plain- text) and generates output (XML) Handles data storage and processing...and also: Generic and extendible script language Efficient

8. AutoEval Script language: Simple C-like syntax Powerful Modules and macros in repository files Extendible, add your own functions

9. AutoEval TnT.wt out.xml field(file("datafile"), "\t", "\n", var("word"), var("tag")); inc(cnt("tot")); inc(cnt(lookup("tag"))); outputintcon(out("outfile"), cntmap("global"), "global"); Example of configuration and script language:

10. AutoEval 14119 714 44 149 24... 117 35 The result:

11. Missplel Missplel is a highly configurable tool to introduce human-like spelling errors Language, PoS tag set, character set and keyboard layout independent All you need is a word/tag/lemma dictionary

12. Missplel Performance errors – Damerau: Keyboard mistypes (Damerau, 1964): Insertion, deletion, substitution, transposition of letters wellcvome, wellcme, wellcpme, wellcmoe Result: a new existing/non-existing word word class (PoS tag) change or not

13. Missplel Competence errors – split compounds: May alter the semantics of a sentence Kycklinglever – chicken liver Kyckling lever – chicken is alive Settings of split compound elements: Minimum length? Allowed PoS tag? Found in dictionary? Word class change? etc.

14. Missplel Competence errors – sound errors: Letter level e.g. sound-alike errors Regular expression rules: (.+)ei(.+) @1ie@2 receive recieve

15. Missplel Competence errors – syntax errors: Word/letter level Form new words from PoS tags, missing/doubled words etc. Regular expression rules: vb\.sup\.akt(.*) vb\.inf.* vb.sup.akt@1 vb.sup.akt

16. Missplel Letters NN2 would VM0 be VBI welcome AJ0-NN1 Litters NN2 damerau/wordexist-notagchange would VM0 ok bee NN1 sound/wordexist-tagchange welcmoe ERR damerau/nowordexist-tagchange

17. Missplel TnT.wt ([^\t]+)\t([^\t]+)([^\r\n]*).* output.wte <!-- %1% Word, %2% Tag, %3% Lemma, %4% Rest of line, %5% Error descr --> %1% %2% %5% ok exist noexist -wordch -nowordch -tagch -notagch...

18. Missplel... unknown unknownLemma @ ' mad 30 felstava/conf/ Swedish.cwtl.+\t([^\t]+)\t([^\t]+)\t+([^\t]+) outfile.gz tagfile...

19. Missplel... damerau yes 10.0 confusionfile 1 no yes no...

20. Missplel... split no 99.0 yes 50 6 3 1 10 0 15...

21. Missplel... sound no sound.test 100.0 (.+)\t(.+)\t(.+) yes no yes no...

22. Missplel... introduced no error.rules 100.0 yes no yes no

23. Applications AutoEval has been used to evaluate Parsers PoS taggers PoS majority/ensemble tagging Missplel has been used to evaluate Spell checkers Grammar checkers Robustness of parsers and taggers

24. Licence AutoEval and Missplel are open source under the Gnu General Public Licence Source code available at www.nada.kth.se/theory/ humanlang/tools.html