1. J. Turmo, 2006 Adaptive Information Extraction Summary Information Extraction Systems Multilinguality Introduction Language guessers Machine Translators Translingual architectures Information integration in MIE systems Evaluation Adaptability Information Extraction Systems Multilinguality Introduction Language guessers Machine Translators Translingual architectures Information integration in MIE systems Evaluation Adaptability

2. J. Turmo, 2006 Adaptive Information Extraction Introduction Multilinguality Multilingual IE (MIE) tasks: The textual information contained in the output templates is wanted to be presented in a different language than the input documents Tipically: input documents written in one language output templates written in another one

3. J. Turmo, 2006 Adaptive Information Extraction Introduction Multilinguality Relatively little research in MIE LRE program in Europe ECRAN, FACILE, AVENTINUS, SPARKLE, … tools and components for IE in different languages TIDES program in USA PROTEUS, RIPTIDES, CREST, … fast machine translation and information access

4. J. Turmo, 2006 Adaptive Information Extraction Up to now Multilingual IE evaluation just for NE tasks. Two recent scenarios: CoNLL 2002-2003: Language-independent NE recognition ACE 2007: Arabic input documents English output NE mentions Fei Huang (2005). Multilingual NE Extraction and Translation from text and speech. PhD. Thesis Introduction Multilinguality Open research line

5. J. Turmo, 2006 Adaptive Information Extraction Introduction Multilinguality Basic elements of MIE architectures: language guessers monolingual architectures Classical approches: use of Machine Translation with monolingual IE architectures extension of monolingual architectures to translingual architectures

6. J. Turmo, 2006 Adaptive Information Extraction Introduction Multilinguality Basic elements of MIE architectures: language guessers monolingual architectures Classical approches: use of Machine Translation with monolingual IE architectures extension of monolingual architectures to translingual architectures

7. J. Turmo, 2006 Adaptive Information Extraction Summary Information Extraction Systems Multilinguality Introduction Language guessers Machine translators Translingual architectures Information integration in MIE systems Evaluation Adaptability Information Extraction Systems Multilinguality Introduction Language guessers Machine translators Translingual architectures Information integration in MIE systems Evaluation Adaptability

8. J. Turmo, 2006 Adaptive Information Extraction Language guessers Multilinguality Goal: identify the language of a document Linguistic approach: based on a vocabulary of keywords idea: at least one word from a tipical sentence written in some language should be included in the corresponding vocabulary manually built

9. J. Turmo, 2006 Adaptive Information Extraction Language guessers Multilinguality Stochastic approach: most widely used based on: generate a frequency table of elements per language compare frequencies of elements in the document with those in the table. elements = or special characters or word sequences or char sequences (different approaches)

10. J. Turmo, 2006 Adaptive Information Extraction Language guessers Multilinguality Stochastic approach: Pros: good results (over 95% accuracy) Cons: short texts [Zhdanova,02] copes with this problem

11. J. Turmo, 2006 Adaptive Information Extraction Summary Information Extraction Systems Multilinguality Introduction Language guessers Machine Translators Translingual architectures Information integration in MIE systems Evaluation Adaptability Information Extraction Systems Multilinguality Introduction Language guessers Machine Translators Translingual architectures Information integration in MIE systems Evaluation Adaptability

12. J. Turmo, 2006 Adaptive Information Extraction Machine translators Multilinguality A set of monoligual IE systems Language guesser IE (s 1 ) IE (s 2 ) IE (s k )...... mt (s 1,t) mt (s 2,t) mt (s k,t)...... templates sisi t MIE

13. J. Turmo, 2006 Adaptive Information Extraction Machine translators Multilinguality Just one monoligual IE system Language guesser mt (t’,t)...... templates sisi t IE (t’) MT (s 1,t’) MT (s 2,t’) MT (s k,t’)...... MIE

14. J. Turmo, 2006 Adaptive Information Extraction Summary Information Extraction Systems Multilinguality Introduction Language guessers Machine Translators Translingual architectures Information integration in MIE systems Evaluation Adaptability Information Extraction Systems Multilinguality Introduction Language guessers Machine Translators Translingual architectures Information integration in MIE systems Evaluation Adaptability

15. J. Turmo, 2006 Adaptive Information Extraction Translingual architectures Multilinguality Try to overcome the ineficiency the MIE architectures based on MT Fussion of IE and interlingua MT Idea: when dealing with a particular domain, it is possible to build a language-independent conceptual model of the particular scenario of extraction [Gaizauskas et al. 97]

16. J. Turmo, 2006 Adaptive Information Extraction Translingual architectures Multilinguality For each source language requires: Use of different lexical preprocessors Use of different syntactico-semantic parsing Use of different sets of IE patterns (if the MIE system is based on pattern matching) Possible use of language-independent processors (e.g., NERC)

17. J. Turmo, 2006 Adaptive Information Extraction Translingual architectures Multilinguality Use of language-independent ontology The internal representation of the extracted information is language independent Use of soft techniques for NL generation The output templates are generated using the lexicon of the target language lexical choice problem!

18. J. Turmo, 2006 Adaptive Information Extraction Translingual architectures Multilinguality M-LASIE system [Gaizauskas et. al 97] Ad-hoc representation of the domain model Lexicons mapped to concepts Add a new source language, involves Add new lexicon + mappings Add new tagger and parser …

19. J. Turmo, 2006 Adaptive Information Extraction Translingual architectures Multilinguality M-TURBIO system [Turmo et. al 99] EuroWordNet (EWN) Sets of IE-patterns for each source language Mappings from IE-patterns to ILIs in EWN Add a new source language, involves Add new IE-patterns Add new tagger and parser …

20. J. Turmo, 2006 Adaptive Information Extraction Summary Information Extraction Systems Multilinguality Introduction Language guessers Machine Translators Translingual architectures Information integration in MIE systems Evaluation Adaptability Information Extraction Systems Multilinguality Introduction Language guessers Machine Translators Translingual architectures Information integration in MIE systems Evaluation Adaptability

21. J. Turmo, 2006 Adaptive Information Extraction Information Integration in MIEs Multilinguality The most general architecture Input documents in different source languages not aligned Output templates in different target languages Possible approaches: MIE system + II system MIE/II system

22. J. Turmo, 2006 Adaptive Information Extraction Information Integration in MIEs Multilinguality Pros: Versatil An instance can occur just in one document written in a specific language. Can be easier to extract an instance expressed in one language than another better processors or resources Cons: Problems inherent to II inconsistent values, similar values, generalizations, …

23. J. Turmo, 2006 Adaptive Information Extraction Summary Information Extraction Systems Multilinguality Evaluation Introduction Metrics Data sets Adaptability Information Extraction Systems Multilinguality Evaluation Introduction Metrics Data sets Adaptability

24. J. Turmo, 2006 Adaptive Information Extraction Introduction Evaluation The evaluation of the performance of an IE system depends on different factors: The IE task: domain, language, document style, … The user needs : software use, human use, just some clues about the relevant facts, the context in which they occur, … What does correctly extracted means? What are the right metrics? What are the best data sets?

25. J. Turmo, 2006 Adaptive Information Extraction Introduction Evaluation The president of ALP in Spain will leave his job tomorrow night NP NP The president of ALP in Spain will leave his job tomorrow night NP Exact extraction ? The president of ALP in Spain will leave his job tomorrow night NP The president of ALP in Spain will leave his job tomorrow night NP Exact extraction ?

26. J. Turmo, 2006 Adaptive Information Extraction Summary Information Extraction Systems Multilinguality Evaluation Introduction Metrics Data sets Adaptability Information Extraction Systems Multilinguality Evaluation Introduction Metrics Data sets Adaptability

27. J. Turmo, 2006 Adaptive Information Extraction Metrics Evaluation Different evaluation frameworks with different points of view of what is correctly extracted: MUC: correct = partial extraction (-MUC5) correct = exact extraction (MUC6, MUC7) Recall, Precision and F (c.f., Historical Framework) PASCAL: correct = exact extraction Same metrics as in MUC6 ACE: correct = partial extraction (more sophisticated than MUC)

28. J. Turmo, 2006 Adaptive Information Extraction Metrics Evaluation ACE metric Idea: How well match the information extracted by a system with that of the reference model? Given a system output, s, and a reference model, m, find the global optimum of function Value(s,m) that maximizes the matchings between instances in s and instances in m

29. J. Turmo, 2006 Adaptive Information Extraction Metrics Evaluation ACE metric Value(s,m) = Value(sys_token i ) / Value(ref_token j ) Σ i Σ j token = instance extracted = [attributes, args or mentions] Value(token) = Element_value(token) * Argument_value(token) Penalties: unmapped attributes, unmapped arguments, wrong mappings Parameters: weights for penalties

30. J. Turmo, 2006 Adaptive Information Extraction Metrics Evaluation ACE metric Software for ACE evaluation and more information on ACE evaluation available in http://www.nist.gov/speech/tests/ace

31. J. Turmo, 2006 Adaptive Information Extraction Summary Information Extraction Systems Multilinguality Evaluation Introduction Metrics Data sets Adaptability Information Extraction Systems Multilinguality Evaluation Introduction Metrics Data sets Adaptability

32. J. Turmo, 2006 Adaptive Information Extraction Data sets Evaluation Ad-hoc State of the art (e.g., from MUC, ACE, PASCAL) Each one appropriated to evaluate different IE tasks, depending on different factors Availability ? Suitability ?

33. J. Turmo, 2006 Adaptive Information Extraction Data sets: MUC Evaluation Sources: free text written text (Newswire) MUC-6 and MUC7 data sets Suitable tasks: NE subtasks Element Extraction tasks (template element –TE) Event Extraction tasks (scenario template -ST) Relation Extraction tasks are quite easy Language: English Available from LDC (Linguistic Data Consortium) http://www.ldc.upenn.edu

34. J. Turmo, 2006 Adaptive Information Extraction Data sets: ACE Evaluation Sources: Free text written text (Newswires, Weblogs, Discussion Forums) Free text oral transcripts (Broadcast News, Telph. conversations) Suitable tasks (up to now): NE subtasks (extended from MUC) Relation Extraction tasks Event Extraction tasks need more annotation efforts Language: English, Arabic, Chinese, Spanish depending on the input source Available from LDC (Linguistic Data Consortium) http://www.ldc.upenn.edu

35. J. Turmo, 2006 Adaptive Information Extraction Data sets: PASCAL Evaluation Sources: Semi-structure documents (Seminar announcements, Corporate acquisitions, Legal sentences) Suitable tasks (up to now): Element Extraction tasks Language: English, Italian Available from http://nlp.shef.ac.uk/dot.kom/resources.html Similar sources in repository RISE http://www.isi.edu/info-agents/RISE/index.html