New Slovene corpora within the »Communication in Slovene« project Nataša Logar BergincSimon Krek University of LjubljanaAmebis, Kamnik Faculty of Social SciencesJozef Stefan Institut

“Communication in Slovene” Web site: Leading partner: Amebis, d. o. o., Kamnik Duration: June December 2013 Total value: 3,2 million Euro Project consortium: Amebis, d. o. o., Kamnik Jozef Stefan Institute University of Ljubljana Scientific Research Centre of the Slovenian Academy of Sciences and ArtsScientific Research Centre of the Slovenian Academy of Sciences and Arts Trojina, Institute for Applied Slovene Studies

Language data Three corpora of Slovene:  a billion word written corpus  GigaFIDA  100 million word balanced subcorpus  KRES  a million word corpus of spoken Slovene  GOS

Other activities NLP tools & resources –statistical tagger and parser –training corpus ( words) –lexicon ( lemmas) Language learning –integration of resources & tools in Slovene language teaching –pedagogical corpus interface –pedagogical corpus-based grammar Language description –lexical database (NLP & lexicography) –manual of style

Goals

GigaFIDA a billion word written corpus linguistic annotation –lemmatized –morpho-syntactically annotated –partly syntactically annotated format –XML TEI P5 format purpose –data for the new Slovene lexical database, pedagogical grammar and manual of style –freely available on the web

A bit of FIDA history FIDA corpus – –100 million words –available for project partners (academic & industrial) FidaPLUS corpus – –620 million words –publicly available in the web concordancer –available for partners as a data set –text type: fiction 3,5%, non-fiction 96,5% (90% newspapers and magazines)

KRES a 100 million word written subcorpus criteria –balanced (text types, production-reception etc.) –text quality (processing & annotation) –copyright issues: 10 % purpose –downloadable as a data set –freely available for research (BNC style) –Creative Commons (Authorship, Non-Commercial)

New taxonomy KRESGigaFIDA Print8050 <> 90 Books3515 <> 35 Fiction1720 <> 50 Non-fiction1830 <> 60 Periodicals4020 <> 40 Newspapers2030 <> 70 Magazines2030 <> 70 Other55 <> 10 Internet2010 <> 50 News sites830 <> 70 Corp. & govern. sites 1230 <> 70

GOS a million word corpus of spoken Slovene −120 hours of speech criteria −demographic −speech type/situation −additional (language learning, 15%) transcription –pronunciation-based –standardized

Demographic criteria –sex: 50% M –age: <34: 40% –education: primary/secondary school: 70% –region: SW: 35%, Ljubljana r.: 25%, NE: 25%, Maribor r.: 15%

Speech type/situation criteria –public/non-public discourse: 60% : 40% –media: face to face c.: 50% telephone: 10% radio: 20% TV: 20%

Tools for linguistic annotation Tokenization & segmentation –new more trasparent rules Lemmatizer & tagger –rule-based (Amebis) –statistical (JSI) –metatagger (JSI) Parser –statistical (based on MSTParser) Online services (beta) –tagger: –parser:

March 2011 Three publicly and freely available annotated corpora of modern Slovene, all texts copyright (+ gathering of new texts still in progress) New user-friendly interface (see Iztok Kosem presentation) Freely available tools for linguistic annotation of Slovene (tagger, parser) … and not much further down the road: new, up-to-date language descriptions and manuals See: