Presentation is loading. Please wait.

Presentation is loading. Please wait.

MULTEXT-East Version 4: multilingual morphosyntactic specifications for lots of languages Tomaž Erjavec Department of Knowledge Technologies.

Published byEmerson Aubrey Modified over 10 years ago

Similar presentations

Presentation on theme: "MULTEXT-East Version 4: multilingual morphosyntactic specifications for lots of languages Tomaž Erjavec Department of Knowledge Technologies."— Presentation transcript:

1 MULTEXT-East Version 4: multilingual morphosyntactic specifications for lots of languages Tomaž Erjavec http://nl.ijs.si/et/ Department of Knowledge Technologies Jožef Stefan Institute Ljubljana Slovenia http://nl.ijs.si/et/ Dublin April 3 rd, 2009

2 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Overview of the talk 1. Part-of-speech tagging, tagsets and interoperability 2. MULTEXT(-East) morphosyntactic specifications 3. Languages, formats, transformations 4. An application: JOS resources for Slovene 5. Conclusions

3 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Part-of-speech tagging The task of assigning the correct PoS tag to each word in a running text, e.g. The task of assigning the correct PoS tag to each word in a running text, e.g. Under/IN the/DT proposal/NN,/, Delmed/NNP would/MD issue/VB about/IN 123.5/CD million/CD additional/JJ Delmed/NNP common/JJ shares/NNS to/TO Fresenius/NNP … Important HLT infrastructure Very useful annotations for linguists Some applications: pre-processing step for further analyses: lemmas, syntactic structure, etc. text indexing, e.g. nouns are more useful than verbs

4 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Methods of PoS tagging PoS tagging: determine ambiguity class or word (saw NN | VBD) disambiguate to correct tag in (local) context (I saw/VBD a saw/NN ) Tagger training: manually annotated corpus: source of probabilities for tags given a (local) context + manually annotated corpus: source of probabilities for tags given a (local) context + (lexicon: gives possible tags for each word-form) (lexicon: gives possible tags for each word-form) Popular taggers: Popular taggers: TnT (HMM tagger), TreeTagger (decision trees), TBL (transformation based tagging) TnT (HMM tagger), TreeTagger (decision trees), TBL (transformation based tagging) Tagging usefulness as well as accuracy crucially depends on the tagset Tagging usefulness as well as accuracy crucially depends on the tagset

5 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 English tagsets Tagging first developed for English (Brown, CLAWS, PTB tagsets) Tagging first developed for English (Brown, CLAWS, PTB tagsets) English inflectionally very poor language small tagsets ~ 50 different tags English inflectionally very poor language small tagsets ~ 50 different tags Tags are typically synthetic, i.e. the tag does not transparently map to features e.g. : Tags are typically synthetic, i.e. the tag does not transparently map to features e.g. : to/TO (PoS?) Delmed/NNP (number?) shares/NNS (number?)

6 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Tagsets for other languages will often have many more morphosyntactic features associated with a word, so tagsets will be larger will often have many more morphosyntactic features associated with a word, so tagsets will be larger e.g. Slovene nouns: e.g. Slovene nouns: type: common, proper type: common, proper gender: masculine, feminine, neuter gender: masculine, feminine, neuter number: singular, dual, plural number: singular, dual, plural case: nom., gen., dat., acc., loc., ins. case: nom., gen., dat., acc., loc., ins. (animacy: yes, no) (animacy: yes, no) = 104 PoS tags just for Nouns = 104 PoS tags just for Nouns Russian, Czech, Slovene ~ 1000-2000 word level syntactict tags Russian, Czech, Slovene ~ 1000-2000 word level syntactict tags

7 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 PoS tags vs. MSDs PoS tags: PoS tags: used in corpora for corpus annotations / tagging used in corpora for corpus annotations / tagging typically synthetic typically synthetic Morphosyntactic Descriptions (MSDs): Morphosyntactic Descriptions (MSDs): used in inflectional lexica for lexical annotations / morphological analysis used in inflectional lexica for lexical annotations / morphological analysis typically analytic typically analytic Relation of PoS tagsets to MSD tagsets/features Relation of PoS tagsets to MSD tagsets/features in general: |PoS| < |MSD| in general: |PoS| < |MSD| but in most MULTEXT-East languages: [PoS] [MSD] but in most MULTEXT-East languages: [PoS] [MSD]

8 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Developing a multilingual morphosyntactic framework Interoperability: Tagsets developed for various languages (or even for the same language) have no connection with each other and are often poorly documented Interoperability: Tagsets developed for various languages (or even for the same language) have no connection with each other and are often poorly documented Best practice: Languages that do not yet have a tagset could benefit from an operational framework in which to model it Best practice: Languages that do not yet have a tagset could benefit from an operational framework in which to model it

9 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 so, wouldnt it be nice to have: an open, standardised, documented, flexible model for MSD/PoS tagset design, an open, standardised, documented, flexible model for MSD/PoS tagset design, that would be instantiated for lots of languages, that would be instantiated for lots of languages, and could be simply applied to any language? and could be simply applied to any language?

10 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 EU standardisation efforts EAGLES: Expert Advisory Group for Language Engineering Standards (1993-1996) EAGLES: Expert Advisory Group for Language Engineering Standards (1993-1996) MULTEXT: Multilingual Text Tools and Corpora (1995) MULTEXT: Multilingual Text Tools and Corpora (1995) MULTEXT-East: MULTEXT for Central and Eastern European Languages: MULTEXT-East: MULTEXT for Central and Eastern European Languages: Version 1: TELRI edition (1998) Version 1: TELRI edition (1998) Version 2: Concede edition (2002) Version 2: Concede edition (2002) Version 3: TEI edition (2004) Version 3: TEI edition (2004) Version 4: MondiLex edition (2009?) Version 4: MondiLex edition (2009?)...... ISO / TC 37 / LMF / isoCat (2008) ISO / TC 37 / LMF / isoCat (2008)

11 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 MULTEXT-East morphosyntactic resources Basic Language Resource Kit: Basic Language Resource Kit: 1. specifications: define features and MSDs 2. lexica (~15,000 lemmas): triplets: word-form / lemma / MSD 3. parallel corpus: MSD and lemma annotated Freely available for research http://nl.ijs.si/ME/ Freely available for research http://nl.ijs.si/ME/ http://nl.ijs.si/ME/

12 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 1984: aligned and annotated

13 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 MULTEXT-East languages

14 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 The MULTEX(-East) morphosyntactic specifications They specify that e.g.Ncmsn They specify that e.g.Ncmsn corresponds to the feature-structure [Noun, Type=common, Gender=masculine, Number=singular, Case=nominative] corresponds to the feature-structure [Noun, Type=common, Gender=masculine, Number=singular, Case=nominative] is a valid MSD for Slovene is a valid MSD for Slovene Specifications consist of Specifications consist of Front matter Front matter Common part - common definitions for all languages (features) Common part - common definitions for all languages (features) Language particular parts - particulars for each language (MSD set) Language particular parts - particulars for each language (MSD set)

15 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 V4 specs draft in HTML

16 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Specifications in Version 4 Encoded in XML / teiLite (in Version 3: LaTeX) Encoded in XML / teiLite (in Version 3: LaTeX) TEI = Text Encoding Initiative Guidelines P4 TEI = Text Encoding Initiative Guidelines P4 Still in book-like in form, to make authoring easier Still in book-like in form, to make authoring easier XSLT into other formats: XSLT into other formats: HTML HTML tabular mapping formats (e.g. MSD to features) tabular mapping formats (e.g. MSD to features) XML/TEI feature library XML/TEI feature library (OWL) (OWL)

17 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 The common specifications Define categories (parts-of-speech) Define categories (parts-of-speech) For each category define features, i.e. attributes and their values For each category define features, i.e. attributes and their values For each attribute-value specify for which languages it is appropriate For each attribute-value specify for which languages it is appropriate Give positional mapping to MSDs: Give positional mapping to MSDs: each attribute assigned a position each attribute assigned a position each attribute-value assigned a one- character code each attribute-value assigned a one- character code

18 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Common table (HTML)

19 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Common table (source XML/teiLite)

20 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Language particular sections Recap the feature definitions for the language Recap the feature definitions for the language Add combinations, i.e. feature-coocurrence restrictions Add combinations, i.e. feature-coocurrence restrictions Add lexicon, i.e. list of all valid MSDs for language Add lexicon, i.e. list of all valid MSDs for language Possibly localise the features and codes Possibly localise the features and codes Possibly give notes and examples Possibly give notes and examples

21 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Combinations

22 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Lexicon

23 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Jezikoslovno označevanje slovenščine http://nl.ijs.si/jos http://nl.ijs.si/jos

24 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 JOS as a bridge to MULTEXT-East Version 4 FidaPLUS corpus JOS corpora MTE V3 slv specifications JOS (slv) specifications MTE V4 (slv) specifications MTE V4 specifications

25 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009

26 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 JOS specifications XML/teiLite + XSLT transforms XML/teiLite + XSLT transforms Allow reordering of attribute positions (Vm-----d Vmd) Allow reordering of attribute positions (Vm-----d Vmd) i18n / slv+eng: i18n / slv+eng: translation: specifications translation: specifications localisation: attributes, values, codes localisation: attributes, values, codes localisation: TEI element names localisation: TEI element names

27 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009

28 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009

29 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 MSD conversion tables Tabular UTF-8 files Tabular UTF-8 files MSD-slv to -eng MSD-slv to -eng MSD to features MSD to features Collating sequence Collating sequencee.g. 01N0101010100 Somei Ncmsn 01N0101010200 Somer Ncmsg 01N0101010300 Somed Ncmsd Ncmsn Noun Type=common Gender=masculine Number=singular Case=nominative Animacy=0 Ncmsg Noun Type=common Gender=masculine Number=singular Case=genitive Animacy=0 Ncmsd Noun Type=common Gender=masculine Number=singular Case=dative Animacy=0

30 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Adding a new language XSLT scripts: XSLT scripts: mtems-split.xsl: make a template for the language particular section of a new language mtems-split.xsl: make a template for the language particular section of a new language mtems-merge: merge a new language particular section to the common tables mtems-merge: merge a new language particular section to the common tables Maybe shortly to be tested on new Slavic languages in the scope of MondiLex Maybe shortly to be tested on new Slavic languages in the scope of MondiLex

31 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Critiques Its just an exercise in encoding anyway Its just an exercise in encoding anyway Same is different, different is same Same is different, different is same The Procrustean bed of standards The Procrustean bed of standards Policy change: from unification to harmonisation (hippy school) Policy change: from unification to harmonisation (hippy school)

32 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Conclusions Presented work-in-progress on standardisation of multilingual morphosyntactic specifications Presented work-in-progress on standardisation of multilingual morphosyntactic specifications Specifications are a de-facto standard for several languages (Romanian, Slovene, Croatian) Specifications are a de-facto standard for several languages (Romanian, Slovene, Croatian) Could serve as hub encoding for multilingual applications, e.g. MT Could serve as hub encoding for multilingual applications, e.g. MT and as an framework for new languages and as an framework for new languages

33 Erjavec: MULTEXT-East Version 4 Dublin, 4.4.2009 Further work Finishing MTE V4! Finishing MTE V4! Distribution: LDC, ELDA Distribution: LDC, ELDA Relation to ISO-TC37 standards: Relation to ISO-TC37 standards: LMF, isoCAT LMF, isoCAT Connecting to GOLD ontology Connecting to GOLD ontology Adding new languages: Adding new languages: Slavic completion Slavic completion Western European: MULTEXT Western European: MULTEXT Japanese: chasen tagset, jpWaC(-L2) Japanese: chasen tagset, jpWaC(-L2) Irish? Irish?

Download ppt "MULTEXT-East Version 4: multilingual morphosyntactic specifications for lots of languages Tomaž Erjavec Department of Knowledge Technologies."

Similar presentations

Three Basic Problems Compute the probability of a text: P m (W 1,N ) Compute maximum probability tag sequence: arg max T 1,N P m (T 1,N | W 1,N ) Compute.

Three Basic Problems Compute the probability of a text: P m (W 1,N ) Compute maximum probability tag sequence: arg max T 1,N P m (T 1,N | W 1,N ) Compute.
OLIF V2 Gr. Thurmair April 2000. 2 OLIF April 2000 OLIF: Overview Rationale Principles Entries Descriptions Header Examples Status.

OLIF V2 Gr. Thurmair April OLIF April 2000 OLIF: Overview Rationale Principles Entries Descriptions Header Examples Status.
School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING A comparative study of the tagging of adverbs in modern English corpora.

School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING A comparative study of the tagging of adverbs in modern English corpora.
School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING PoS-Tagging theory and terminology COMP3310 Natural Language Processing.

School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING PoS-Tagging theory and terminology COMP3310 Natural Language Processing.
Machine Learning Approaches to the Analysis of Large Corpora : A Survey Xunlei Rose Hu and Eric Atwell University of Leeds.

Machine Learning Approaches to the Analysis of Large Corpora : A Survey Xunlei Rose Hu and Eric Atwell University of Leeds.
Disambiguation of homographic adjective and adverb forms in Croatian texts Danijela Merkler*, Daša Berović*, Željko Agić** * Department of Linguistics.

Disambiguation of homographic adjective and adverb forms in Croatian texts Danijela Merkler*, Daša Berović*, Željko Agić** * Department of Linguistics.
University of Sheffield NLP Module 4: Machine Learning.

University of Sheffield NLP Module 4: Machine Learning.
Corpus Processing and NLP

Corpus Processing and NLP
Three Basic Problems 1.Compute the probability of a text (observation) language modeling – evaluate alternative texts and models P m (W 1,N ) 2.Compute.

Three Basic Problems 1.Compute the probability of a text (observation) language modeling – evaluate alternative texts and models P m (W 1,N ) 2.Compute.
MULTEXT-East Version 4: Multilingual Morphosyntactic Specifications, Lexicons and Corpora Tomaž Erjavec Department of Knowledge Technologies Jožef.

MULTEXT-East Version 4: Multilingual Morphosyntactic Specifications, Lexicons and Corpora Tomaž Erjavec Department of Knowledge Technologies Jožef.
Jing-Shin Chang National Chi Nan University, IJCNLP-2013, Nagoya 2013/10/15 ACLCLP – Activities (2011-2013) & Text Corpora.

Jing-Shin Chang National Chi Nan University, IJCNLP-2013, Nagoya 2013/10/15 ACLCLP – Activities ( ) & Text Corpora.
Totale Multilingual Tokenisation, Tagging and Lemmatisation Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Ljubljana, Slovenia JRC.

Totale Multilingual Tokenisation, Tagging and Lemmatisation Tomaž Erjavec Dept. of Knowledge Technologies, Jožef Stefan Institute Ljubljana, Slovenia JRC.
Part-of-speech tagging. Parts of Speech Perhaps starting with Aristotle in the West (384–322 BCE) the idea of having parts of speech lexical categories,

Part-of-speech tagging. Parts of Speech Perhaps starting with Aristotle in the West (384–322 BCE) the idea of having parts of speech lexical categories,
Annotating language data Tomaž Erjavec Institut für Informationsverarbeitung Geisteswissenschaftliche Fakultät Karl-Franzens-Universität Graz Tomaž Erjavec.

Annotating language data Tomaž Erjavec Institut für Informationsverarbeitung Geisteswissenschaftliche Fakultät Karl-Franzens-Universität Graz Tomaž Erjavec.
The University of Wisconsin-Madison Universal Morphological Analysis using Structured Nearest Neighbor Prediction Young-Bum Kim, João V. Graça, and Benjamin.

The University of Wisconsin-Madison Universal Morphological Analysis using Structured Nearest Neighbor Prediction Young-Bum Kim, João V. Graça, and Benjamin.
The MULTEXT-East multilingual language resources Tomaž Erjavec Department of Knowledge Technologies Jožef Stefan Institute, Ljubljana

The MULTEXT-East multilingual language resources Tomaž Erjavec Department of Knowledge Technologies Jožef Stefan Institute, Ljubljana
Part II. Statistical NLP Advanced Artificial Intelligence Part of Speech Tagging Wolfram Burgard, Luc De Raedt, Bernhard Nebel, Lars Schmidt-Thieme Most.

Part II. Statistical NLP Advanced Artificial Intelligence Part of Speech Tagging Wolfram Burgard, Luc De Raedt, Bernhard Nebel, Lars Schmidt-Thieme Most.
New Slovene corpora within the »Communication in Slovene« project Nataša Logar BergincSimon Krek University of LjubljanaAmebis, Kamnik Faculty of Social.

New Slovene corpora within the »Communication in Slovene« project Nataša Logar BergincSimon Krek University of LjubljanaAmebis, Kamnik Faculty of Social.
Ch 10 Part-of-Speech Tagging Edited from: L. Venkata Subramaniam February 28, 2002.

Ch 10 Part-of-Speech Tagging Edited from: L. Venkata Subramaniam February 28, 2002.
Resources Primary resources – Lexicons, structured vocabularies – Grammars (in widest sense) – Corpora – Treebanks Secondary resources – Designed for a.

Resources Primary resources – Lexicons, structured vocabularies – Grammars (in widest sense) – Corpora – Treebanks Secondary resources – Designed for a.

Similar presentations

Ads by Google