Error Analysis of Rule-based Machine Translation Outputs A Case Study on English – Persian MT System تجزیه و تحلیل خطا از حکومت مبتنی بر خروجی دستگاه ترجمه؟ مطالعه موردی در زبان انگلیسی - فارسی سیستم های MT Zahra Pourniksefat Islamic Azad University – Science & Research Branch

Agenda Introduction Machine Translation Overview Evaluation of MT systems Methods & Materials Error Categories & Description Results & Discussion

Machine Translation Overview computerssoftware Definition : The term Machine Translation (MT) is used for translating text or speech from one natural language to another by using computers and software. faster than human translators Systran: MT is much faster than human translators because it is much cheaper and has a better memory than human translators. actualactof translation innovationdiscovery Shahba 2002 believed that Its better to spend our time on the actual act of translation rather than typing the English text or scanning it for the MT to translate. Efforts in MT are by themselves valuable as they at least satisfy one of the needs of human beings: need for innovation and discovery lessaccurate MT is more economic on time and money, but it is less accurate than human translators (Frederking, 2004).

Why MT matters? According to Hatim and Munday its an important topic - socially, politically, commercially, scientifically, and intellectually or philosophically (2004) socialpolitical The social or political importance of MT arises from the socio- political importance of translation in communities where more than one language is generally spoken. So translation is necessary for communication- for ordinary human interaction, and for gathering the information one needs to play a full part in society. commercial The commercial importance of MT is a result of related factors. First, translation itself is commercially important. Second, translation is expensive. Scientifically Scientifically, MT is interesting, because it is an obvious application and testing ground for many ideas in Computer Science, Artificial Intelligence, and Linguistics. Philosophically Philosophically, MT is interesting, because it represents an attempt to automate an activity that can require the full range of human knowledge.

Some Misconceptions about MT MT is a waste of time because you will never make a machine that can translate Shakespeare. This criticism that MT systems cannot translate Shakespeare is a bit like the criticism of industrial robots for not being able to dance.(Hatim and Munday, 2004) First, translating literature requires special literary skills – it is not the kind of thing that the average professional translators normally attempt Second, literary translation is a small proportion of the translation that has to be done. Finally, one may wonder who would ever want to translate Shakespeare by machine – its a job that human translators find challenging and rewarding, and its not a job that MT systems have been designed for.

Approaches to MT Direct Machine Translation Approach The first developed MT systems where a word–for–word translation from the source language to the target language is performed. Transfer Machine Translation Approach 1.The analysis stage that is the direct strategy which takes benefits of a dictionary in source language to demonstrate the source language from linguistic point of view. 2.The transfer stage varies the outputs of the analysis stage to produce structural and linguistic equivalents between the two languages. 3.The generation stage is the third stage in which a target language dictionary is applied to result the target language document on the basis of linguistic information. (Steiner, 1988) Interlingua Machine Translation Approach First the source text meaning is decoded Second the resulted meaning is re-encoded in the target language

Approaches to MT contd. Rule-based Machine Translation Approach linguisticdata semanticmorphological syntactic grammar It operates on the linguistic data on source and target languages fundamentally taken from bilingual dictionaries and the basic semantic, morphological, and syntactic grammar of the individual language (Gelbukh, 2011). Minimally, to get a Persian translation of English sentence one needs: 1.A dictionary that will map each English word to an appropriate Persian word. 2.Rules representing regular English sentence structure 3.Rules representing regular Persian sentence structure 4.And finally, we need rules according to which one can relate these two structures together.

Approaches to MT contd. Statistical Machine Translation Approach corpusdatabase GoogleTranslate This system uses a corpus or database as a translated example for analyzing and decoding source language. In comparison with the machine translation of about three decades ago, Google Translate as an example of more contemporary automated engine for the task of translation has taken a giant leap. However, it is still too imperfect. (Nierenberg, 1998) Hybrid Machine Translation Approach 1.Rules post-processed by statistics in which translation are practiced on the pivot of rule-based engine. Next statistics are applied to correct the output. 2.Statistics guided by rules in which rules have an important role to pre-process date to quite the statistical representation to normalize. This approach is powerful, flexible and under more control when its translating.

Evaluation of MT Systems microtextual macrotextual Human translation assessment (Secară 2005; Williams 2001) has been moving from microtextual, word- or sentence-level error analysis methods toward more macrotextual methods focused on the function, purpose and effect of the text. At the same time, machine translation assessment has mainly been microtextual and focused on the aspects of accuracy and fluency. purposecontext Hovy (2002) discussed the complexity of MT evaluation, and stressed the importance of adjusting evaluation to the purpose and context of the translation.

Evaluation of MT Systems contd. Mary A. Flangan Believed that Machine translation quality can be difficult to quantify for a number of reasons: 1)A text can have several different translations, all of which are correct. 2)Defining the boundaries of errors in MT output is often difficult. Errors sometimes involve only single words, but more often involve phrases, discontinuous expressions, word order or relationships across sentence boundaries. Therefore, simply counting the number of wrong words in the translation is not meaningful. 3)One error can lead to another. For example, if the part of speech of a word is identified incorrectly by the MT software, the entire analysis of the sentence may be affected, creating a chain of errors. 4)The cause of errors in MT output is not always apparent. The evaluator usually does not have access to a trace of the software's tests and actions. Thus it can be difficult to identify what went wrong in the translation of a sentence.

Evaluation of MT Systems contd. Types of Evaluation Automatic Evaluation the Word Error Rate (WER), the Position independent word Error Rate (PER), the BLEU (Papineni et al., 2002) and the NIST (Doddington, 2002) where the MT output is compared to one or more human reference translations. Human Evaluation Due to the complexity of natural language, manual evaluation seems more reliable 1.Three passages were selected and translated by Rule-based MT Systems and compared with one Statistical MT System and Human translator 2.Error categories were derived after the analysis of each text

Methods & Materials Three passages were translated by two different MT systems and also a human translator. From each text type a passage of approximately 400 words was taken from story, user guide and magazine. The rule-based MT – Arya TM – system was designed based on thousands of lexical and grammatical rules. The statistical system, Google Translate by Google Inc., is based on the use of large monolingual and parallel corpora for translation. The unit of analysis was set to a sentence level because its the largest unit which can be easily recognized in MT systems and ST sentence can be clearly corresponded to its TT pairs. Table of Source Text Passages for Analysis Number of WordsNumber of Sentences Short Story The Lottery User Guide Microsoft Access Magazine Academic article

Errors Category Syntactic Word Order Missing Words Punctuation Parts of Speech Conjugation Unknown Words Semantic Incorrect Words Polysemy Idiomatic Expressions For English-to-Persian Rule- based MT systems the following categories were derived Error Categories & Descriptions

Error Categories & Descriptions contd. Description of Error Categories: Syntactic Errors: Those errors that are related to the grammar of the language such as parts of speech or conjugation Word order that means sentence elements ordered incorrectly Example: Commands generally take the form of buttons and lists. (User Guide) Missing words: incorrect elision of some words Example: This requires better data collection and analysis tools for studying outcomes and consistent use of these tools across individual studies. (Magazine) Arya Translation System دستور ها بطور کلی فرم شاگرد می گیرد و فهرست ها. Google Translate دستورات به طور کلی به شکل دکمه ها و لیست. Arya Translation System این مجموعه اطلاعات بهتر نیاز های و ابزار ها تحلیل برای مطالعه می کن حاصل ها و سازگار استفاده می کند. Google Translate این امر مستلزم جمع آوری داده ها بهتر و با استفاده از ابزار تجزیه و تحلیل برای بررسی نتایج و استفاده مداوم از این ابزار در سراسر مطالعات فردی.

Error Categories & Descriptions contd. Unknown words: word not in a dictionary Example: The women, wearing faded house dresses and sweaters, came shortly after their menfolk.( Story) Punctuation: incorrect punctuation Example: The children assembled first, of course. (Story) Arya Translation System زن ها, خسته کننده لباس ها خانه و ژاکت ها محو کردند ، به زودی پس از menfolk شان آمدند Google Translate زنان، پوشیدن لباس و ژاکت پژمرده خانه، در آمد مدت کوتاهی پس از menfolk خود را. Arya Translations بچه ها اول, البته جمع کردند. Google Translations کودکان مونتاژ اول، البته.

Error Categories & Descriptions contd. Parts of speech: errors in identifying pars of speech such as noun or verb Example: If you decrease the width of the ribbon, small button labels disappear. (User Guide) Conjugation: incorrectly formed verb or wrong tense Example: Soon the women, standing by their husbands, began to call to their children, and the children came reluctantly, having to be called four or five times. Arya Translation System اگر شما پهنا نوار کاهشبیابید ، دکمه کوچک ناپدید برچسب می زند Google Translate اگر عرض نوار شما را کاهش دهد، برچسب ها دکمه کوچک ناپدید می شوند Arya Translations بزودی زن ها, حمایت می کن شوهر های شان, به شروع کردن صدا به بچه ها شان, و بچه ها با اکراه آمدند ، می دارد که اشد صدا زده چهار یا پنج دوره. Google Translations به زودی زنان، شوهران خود ایستاده، شروع به تماس به فرزندان خود، و بچه ها به اکراه، به نام چهار یا پنج بار.

Error Categories & Descriptions contd. Semantic Errors: Those errors that are related to the meaning such as incorrect meaning of words or expressions which caused the incorrect meaning of the whole sentence. Incorrect word: completely incorrect meaning Polysemy: incorrect selection of the meaning of the words with more than one meaning Example: The people of the village began to gather in the square, between the post office and the bank, around ten o'clock. Style and idiomatic expression : incorrect translation of multi-word expression Example: They greeted one another and exchanged bits of gossip as they went to join their husbands. Arya Translations آنها سلام همدیگر و ذره غیبت معاوضه کردند آنها رفتند که وصل کنند شوهر های شان. Google Translations آنها استقبال یکدیگر و رد و بدل بیت از شایعات بی اساس را به عنوان آنها را برای پیوستن به شوهر خود رفت. Arya Translations مردم روستا شروع کردند که جمع شوند در مربع, در میان پستخانه و بانک ، حدود دَه ساعت Google Translations مردم روستا در میدان شروع به جمع آوری، بین اداره پست و بانک، ساعت حدود ده.

Results & Discussions RBMT SMT Human Word OrderMissing Words Unknown Words PunctuationParts of SpeechConjugation Story User Guide Magazine Story User Guide Magazine Story User Guide Magazine Syntactic Category Word Order Missing Words Unknown Words Punctuation Parts of Speech Conjugation Table of Syntactic Errors

RBMT SMT Human Incorrect LexiconPolysemyIdiomatic Expression Story User Guide 79 5 Magazine Story 88 9 User Guide 37 3 Magazine Story 002 User Guide Magazine Semantic Category Incorrect LexiconPolysemy Idiomatic Expression Table of Semantic Errors Results & Discussions contd.

simplersentences compound-complexsentences Both systems made the least errors with the simpler sentences and the most ones with the compound- complex sentences, as well as lexically or structurally ambiguous texts. This is because ambiguous source texts with different contents can correspond with more than one representation. conjugationwordorder polysemous wordsidiomatic expressions conjugatingdeterminingthetense For the rule-based system, the most typical errors are in conjugation, word order and also in rendering polysemous words and idiomatic expressions. For the statistical system the most common error is in conjugating and determining the tense. However, it has also some problems in translating words with multiple meaning and idiomatic expression. To see whether machine translation accuracy is affected by text-type three different genres were analyzed thoroughly. And for the different text types, the rule- based system had similar amounts of syntactic and semantic errors in each text.

Future! Evaluating MT quality is necessarily a subjective process because it involves human judgments. Determining the best category for an error in MT output is not easy because we have to place them on how they are realized rather than the cause of errors and many machine translated sentences contained multiple, linked errors. Future work Future work will therefore be focused on the cause of errors and ranking error categories. The error categories presented here is flexible, allowing for the deletion or addition of more categories.