Language Data Resources About Corpora

J. Sinclair: “Language looks rather different when you look at a lot of it at once.“ P. Eisner: “Znáte jej, ten svůj jazyk? Řekl by přec člověk, že mám-li něco milovat, musím to znát. Vy však češtinu neznáte, a říkám-li to, není to ani obžaloba, ani vůbec výtka. Nemůžete ji znát a obsáhnout, to se dokonale nepodařilo ještě nikomu…“

Merriam-Webster OnLine:

Corpus F. Čermák: corpus – a structured, unified (and often also tagged) large collection of language data T.McEnery: Corpus data – the raw fuel of NLP

Corpus linguistics A study of language that includes all processes related to processing, usage and analysis of written or spoken machine-readable corpora. Corpus linguistics is a relatively modern term used to refer to a methodology, which is based on examples of ‘real life’ language use Corpus linguistics is not a language theory.

A. by medium: –printed, electronic text, digitized speech, video B. by design method: –balanced vs. special C. language variables: –monolingual vs. multilingual –original vs. translations –native speaker vs. Learner D. language evolution: –synchronic vs. diachronic E. Plain vs. annotated Corpora classification

Balanced corpora (?)‏ T.McEnery: “Sampling is inescapable.“ Proportions corresponding to the real language usage Is that possible? Criteria for choosing styles, genres, and eventually concrete texts? reception (a few authors, large audience) vs. perception (produkce of a large community of language users)‏ N. Chomsky: “Any natural corpus will be skewed. Some sentences won’t occur because they are obvious, others because they are false, still others because they are impolite. The corpus, if natural, will be so wildly skewed that the description would be no more than a mere list.“

Corpus size Brown Corpus – 1 MW (1964)‏ British Natural Corpus – 100 MW (1994)‏ – Cosmas – 1.6 GW (2004)‏ –

Exercise Could you estimate the amount of Czech texts (measured in running words) available on the Internet?

Antecedents of corpora Excerption tickets –For Czech systematically from 1911 Electronic corpus of Czech tests –1970s –around 500kW

Corpus annotation K.Pala: “Annotating consist of adding selected linguistic information to an existing corpus of written or spoken language. Typically, this is done by some kind of coding being attached (semi)automatically or manually to the electronic representation of the text.“ Raw texts: difficult to exploit solution: gradual „information adding“ (more exactly: adding information in an explicit, machine tractable form), Annotation  ease of exploitation + reusability

Criticism of corpus annotation Corpus annotations produce impure corpora –forced interpretations Consistency vs. Accuracy

Czech National Corpus ÚČNK (Institute of Czech National Corpus) founded in 1994 diachronous section th century - DIAKORP synchronous section – from around 1900 –written language – 100MW v SYN 2000 –spoken language – Prague spoken corpus (PMK), Brno spoken corpus (BMK)‏ –dialects

Czech National Corpus

SYN2000

Preprocessing Collect textual material –electronic form –scanning+OCR –trend: WWW as a corpus Conversion and cleaning –Unified format (problém: loosing some information)‏ –Unified encoding (problem: encoding detection)‏ Document classification Document segmentation –segmentation on sentence boundaries (problem: tables, direct speech…)‏ –Tokenization on word boundaries (problem: what is a word?)

(Morphological) Tagging (1) Morphological analysis –For each word form, list all possible lemma+tag pairs (or list of sequences of such pairs, if tokenization is not straightforward)‏ (2) Disambiguation –choose one lemma+tag pair

Parallel corpora texts and their translations into another language (or into more languages)‏ added value - alignment –explicit pairing of corresponding chunks of text –ideally diagonal –often just sentence-level alignment –automatized alignment? anchor points, word-pairs, …

MULTEXT-EAST Multilingual Text Tools and Corpora for Central and Eastern European Languages Lexical resources –Entry: word form + lemma + MSD –MSD – morphosyntactic descriptions (Ncms – Noun common masculine singular)‏ Annotated multilingual corpus –Translations of George Orwell's "1984", about 100kW –Bulgarian, Czech, Estonian, Hungarian, Romanian, and Slovene, as well as for English (hub language)‏ –(and recently also Croatian, Lithuanian, Resian, Romanian, Russian, Slovene)‏ –Hand-validated sentence alignment version 3 released in 2004 (publically available)‏ TEI P4 XML

Prague Czech-English Dependency Treebank Czech translation of 21,600 English sentences from the Wall Street Journal part of Penn Treebank 3 corpusPenn Treebank 3 Czech-English corpus of plain text from Reader's Digest consisting of 53,000 parallel sentences automatically morphologically annotated and parsed into two levels (analytical and tectogrammatical) of dependency structures Available via LDC PCEDT

E. Brill: “More data is more important than better algorithms“ E. Charniak: “Future is in statistics.“