1 Corpora for the coming decade Adam Kilgarriff Lexical Computing Ltd

Aston May 2009 Kilgarriff: Corpora for the coming decade How to build a corpus  Like this: (demo) 

Aston May 2009 Kilgarriff: Corpora for the coming decade3 How should they be different?  Bigger  Better

Aston May 2009 Kilgarriff: Corpora for the coming decade4 Bigger  Motivation Ample data for rare phenomena Big subcorpora For language modelling  More like Google-scale but without Google disadvantages  See Googleology is Bad Science, CL 2007

Aston May 2009 Kilgarriff: Corpora for the coming decade5 Better  Less noise  Fewer duplicates  Richer markup At word, sentence level At document level (text type, subcorpora) ‏

Aston May 2009 Kilgarriff: Corpora for the coming decade6 Divide and rule  Bigger (+ cleaning + deduplication) ‏ Big Web Corpus (BiWeC) ‏  Currently 5.5b fully processed  Target 20b words  Jan Pomikalek, Pavel Rychly  Better New Model Corpus

Aston May 2009 Kilgarriff: Corpora for the coming decade7 New Model Corpus  model 1.small version: model train 2.design: data model  New Model Corpus 1:100 scale model To replace BNC as design model

Aston May 2009 Kilgarriff: Corpora for the coming decade8 BNC design model  Most often used Eg for other languages  pre-web f(blog)=0  Corpora now bigger, far quicker, far cheaper, different issues  BNC design model past its sell-by Kilgarriff Atkins Rundell, Corpus Lg 2007

Aston May 2009 Kilgarriff: Corpora for the coming decade9 New model  Data  Markup

Aston May 2009 Kilgarriff: Corpora for the coming decade10 Data  From the web  100m words  Small sample size Copyright ??Creative Commons Licence

Aston May 2009 Kilgarriff: Corpora for the coming decade11 Composition  General crawl50  Targeted Fiction 7 Blog 7 Newspaper (RSS feed) 7 Speech10  Film transcripts, chatshow Domain-specific19  Business, medical, law

Aston May 2009 Kilgarriff: Corpora for the coming decade12 Markup  Collaborative We distribute data Anyone applies their tools  Pos-tagger, parser, co-ref resolution, domain classifier, WSD, semantic classifier, time phrases, named entities... We integrate, display in Sketch Engine Research potential from multiple markup

Aston May 2009 Kilgarriff: Corpora for the coming decade13 Recombine the two strands  Apply methods with good accuracy (and fast) to BiWeC  Result will be Bigger Better

Aston May 2009 Kilgarriff: Corpora for the coming decade The Sketch Engine  Full-functionality corpus system  Fast  Web-based  In daily use for lexicography at OUP, Collins, CUP, Macmillan, … Le Robert, Cornelsen, Patakis, INL, …  Many universities, language teaching  Free trial  Demo:

Aston May 2009 Kilgarriff: Corpora for the coming decade What can computers count up to?  By default, 2 billion 32 bits, one for the sign, 2 31 = 2 billion Re-engineering required to go beyond  Most corpus systems: tough limit  Sketch Engine Recently re-engineered for 64-bit integers No longer limited

Aston May 2009 Kilgarriff: Corpora for the coming decade16 NLP by web services?  Big corpora big to hold, hard to access fast  Sketch Engine: corpus specialist  Web API FrameNet TEDDCLOG: Taiwan English Data Driven Cloze (test sentence) Generation  All welcome

Aston May 2009 Kilgarriff: Corpora for the coming decade17 Practicalities  Free trial accounts  Collaborators, innovative users free longer-term accounts Wikinomics, Tapscott and Williams  API Details under 'help' on SkE home page  New Model Corpus Available by end 2009: watch Corpora

Aston May 2009 Kilgarriff: Corpora for the coming decade Thank you Enjoy!