A New Approach for Cross- Language Plagiarism Analysis Rafael Corezola Pereira, Viviane P. Moreira, and Renata Galante Universidade Federal do Rio Grande do Sul CLEF 2010

Outline Introduction Related Work The Proposed Approach Experiments Summary and Future Work

Introduction Plagiarism is one of the most serious forms of academic misconduct It is defined as “the use of another person's written work without acknowledging the source” A study with over 80,000 students in the US and Canada found that many of them have already commited a plagiarism offense –36% of undergraduate students –24% of graduate students Several types –Word-for-word, paraphrasing, text translation, etc.

Introduction Cross-language plagiarism is becoming more commom –Evolution of automatic translation systems –Increasing availability of textual content in many different languages Common scenario –A student downloads a paper, translates it using a automatic translation tool, corrects some translation errors and presents it as his own work It can also involve self-plagiarism –Usually aims at increasing the number of publications

Introduction What is the task? –Detect the plagiarized passages in the suspicious documents and their corresponding text fragments in the source documents even if the documents are written in different languages Known as External plagiarism analysis

Outline Introduction Related Work The Proposed Approach Experiments Summary and Future Work

Related Work Monolingual Plagiarism Analysis –Fingerprints, fuzzy-fingerprints,... Cross-Language Plagiarism Analisys –Statistical bilingual dictionary + bilingual text alignment –Use EuroWordNet to transform words into a language independent representation PAN competition –Enables different methods to be compared against each other –It was held as an evaluation lab in conjunction with CLEF 2010

Outline Introduction Related Work The Proposed Approach Experiments Summary and Future Work

The Proposed Approach Suspicious Documents Original Documents Norm. Susp. Documents Norm. Orig. Documents Language Normalization Suspicious Document for each Index Retrieval Candidate Documents Training Corpus Feature Selection + Classifier Training Classification Model Plagiarism Analysis Preliminary Result Post-Processing Final Result (1) (2) (4) (5) (3)

(1) Language Normalization All documents are converted into a common language English was chosen –More translation resources –One of the easiest languages to translate into Used a language guesser and an automatic translation tool

The Proposed Approach Suspicious Documents Original Documents Norm. Susp. Documents Norm. Orig. Documents Language Normalization Suspicious Document for each Index Retrieval Candidate Documents Training Corpus Feature Selection + Classifier Training Classification Model Plagiarism Analysis Preliminary Result Post-Processing Final Result (1) (2) (4) (5) (3)

(2) Retrieval of Candidate Documents Problem: It is not feasible to perform exhaustive comparisons Solution: Use passages from the suspicious document as a query to be sent to an IR system Note that documents are divided into subdocuments (paragraphs) in order to reduce the amount of text that must be analyzed At the end of this phase, we have a list of at most ten candidate subdocuments for each passage in the suspicious document

The Proposed Approach Suspicious Documents Original Documents Norm. Susp. Documents Norm. Orig. Documents Language Normalization Suspicious Document for each Index Retrieval Candidate Documents Training Corpus Feature Selection + Classifier Training Classification Model Plagiarism Analysis Preliminary Result Post-Processing Final Result (1) (2) (4) (5) (3)

(3) Feature Selection and Classifier Training The goal is to build a classification model that can learn how to distinguish between a plagiarized and a non-plagiarized text passage Annotated synthetic examples used for training J48 classification algorithm Features –The cosine similarity between the suspicious passage and the candidate subdocument –The similarity score assigned by the IR system –The position of the candidate subdocument in the rank generated –The length (in characters) of the suspicious and the candidate subdocument

The Proposed Approach Suspicious Documents Original Documents Norm. Susp. Documents Norm. Orig. Documents Language Normalization Suspicious Document for each Index Retrieval Candidate Documents Training Corpus Feature Selection + Classifier Training Classification Model Plagiarism Analysis Preliminary Result Post-Processing Final Result (1) (2) (4) (5) (3)

(4) Plagiarism Analysis Submit the test instances to the trained classifier and let it decide whether the suspicious passage is, in fact, plagiarized from one of the candidate subdocuments SubDoc 1 SubDoc 2 SubDoc Classifier Plagiarized Or Non-Plagiarized class labels Suspicious Document Passage 1Passage 2Passage 5... Index Retrieval

The Proposed Approach Suspicious Documents Original Documents Norm. Susp. Documents Norm. Orig. Documents Language Normalization Suspicious Document for each Index Retrieval Candidate Documents Training Corpus Feature Selection + Classifier Training Classification Model Plagiarism Analysis Preliminary Result Post-Processing Final Result (1) (2) (4) (5) (3)

(5) Result Post-Processing Join the contiguous plagiarized passages detected by the method in order to decrease the granularity score The granularity score is a measure that assesses whether the plagiarism method reports a plagiarized passage as a whole or as several small plagiarized passages

Outline Introduction Related Work The Proposed Approach Experiments Summary and Future Work

Experiments Multilingual Test Collection –ECLaPA collection assembled from the Europarl Parallel Corpus (English, Portuguese and French) –An analogous monolingual corpus was also assembled –Available at Terrier IR System (Porter Stemmer + Stop-Word Removal) Weka (J48 classification algorithm) Google Translator (as language guesser) LEC Power Translator Evaluation Measures (PAN competition)

Experiments - Results Monolingual vs. Multilingual Recall was the most affected measure –Loss of information due to the translation process 86% of the overall score of the monolingual baseline ---MonolingualMultilingual% of Monolingual Recall % Precision % F-Measure % Granularity % Overall Score %

Experiments - Results Detailed analysis The larger the passage the easier the detection Plagiarized passages detected –Monolingual 90% vs. 77% Multilingual --- MonolingualMultilingual ShortMediumLargeShortMediumLarge Detected Total %

Outline Introduction Related Work The Proposed Approach Experiments Summary and Future Work

Summary We proposed and evaluated a new method for CLPA –Used a classification algorithm in order to decide whether a text passage is pagiarized or not We assembled an artificial cross-language plagiarism test collection to evaluate the method –It is freely available Cross-language experiment achieved 86% of the performance of the monolingual baseline

Future Work Improve the time spent during the analysis of each suspicious document –Analyze each suspicious passage in a different computer? Test other features during the classifier training phase Evaluate the method while detecting plagiarism between documents written in unrelated languages –English vs. Chinese/Japanese –Many plagiarism cases happen between these pairs of languages Citation analysis

Outline Introduction Related Work The Proposed Approach Experiments Summary and Future Work

A New Approach for Cross- Language Plagiarism Analysis Rafael Corezola Pereira, Viviane P. Moreira, and Renata Galante Universidade Federal do Rio Grande do Sul CLEF 2010 Questions?