Arabic STD 2006 Results Jonathan Fiscus, Jérôme Ajot, George Doddington December 14-15, Spoken Term Detection Workshop

Outline Motivation Language background –Challenges of processing –Written Arabic Orthography and Syntax Diacritization Evaluation results –Corpus statistics –Participants –Results Future directions

Motivation Why Include Arabic in STD? –The Arab Countries represents 4.8% of the world population 1 –Arabic a good complement to English and Mandarin –Suitable corpora resources exist –Community expertise is growing Aside from the STD technology questions, how to handle diacritics is a major issue –Diacritics provide better specificity for dialectal words but at a cost. Will it be worth it? –If diacritics are used, can they be reliably transcribed? 1

The Many Challenges of Arabic Language Research Arabic is not a single language but a family of languages –Each dialect is “like” a different language –STD ’06 focused on the two variants used in previous DARPA EARS evaluations: Modern Standard Arabic (MSA) in Broadcast News, e.g., Al Jezeera, Al Arabiya Colloquial Arabic in Conversational Telephone Speech: Levantine dialect While MSA is commonly written, dialectal Arabic is not –In fact, dialectal variations are not mutually intelligible The Arabic writing system is more complicated than English –28 letters consonants (each having several script glyphs) –8 Diacritics: 3 short vowels, 3 long vowels, an omitted vowel, double consonant –Diacritics can be omitted or optionally used to disambiguate text  Fluent readers predict vowels from context

Orthography and Syntax (K-6) Predicting Vowels Three Arabic syntactic classes –Verb (Fi’il) – Actions connected to time –Noun (Ism) – Content words –Particles (Harf) – Everything else The vowels within a word are affected by Agents –Agents can be: Word position within sentences Preceding morphemes that determine case, mood, accusative, state (Sign of Dammah, Fatha, Kashra, Sukoon), etc. Preceding particles –Agents are orthographically realized through vowels This is why diacritics can be left out because they can be predicted Abstracted from الشَّرْعِيِّة الدَّوْلِيِّة الشرعية الدولية

Diacritic Usage The 8 diacritics –Long Vowels Short Vowels Fathatan -/ā/ Fatha -/a/ Dammatan -/ū/ Damma - /u/ Kasratan -/ī/Kasra - /i/ –Other Diacritics Shadda – Consonant doubling Sukoon – Lack of a vowel Diacritized texts  Filtered(Diacritized texts) –Diacritics are sometimes used to disambiguate words Long Vowels were in the non-diacritized training data Caused a mismatch between terms and training resources –Expedient solution: throw out 77 terms with long vowels

Evaluation Corpora STD Arabic Data Sources –From the Rich Transcription 2004 Test Set BNews: Al Jazeera, Dubai TV, (~1 hour) CTS: Levantine Fisher Data collection (~1 hour) –This was too small, but all we had available Data was originally transcribed by LDC –Appen corrected and added diacritics to the transcripts

Appen’s Diacritizations Appen corrected and added diacritics to the DevSet and EvalSet transcripts. –2 Pass process: transcription and QA –20% was dually transcribed Findings: –The corrected, undiacritized transcripts differed from LDC by 5.0% for BNews, 4.7% for CTS –13.7% of the words have 2 or more diacritized variants (same underlying consonants, different vowels) These may be real differences –For the 20% dually transcribed data Inter-transcriber error rate for the diacritized transcripts was 17.0% for BNews and 19.7% for CTS Inter-transcriber error rate for the NON-diacritized transcripts was 4.5% for BNews and 8.8% for CTS 12.5% and 10.5% is the lower bound on disagreement in diacritization Conclusion: –The current level of diacritic ambiguity is not conducive to evaluations This was not an unexpected result Would need careful annotation guidelines to improve consistency –Mid-eval correction: allow both diacritized and non-diacritized term systems

Arabic Term Profile Followed the same selection protocol English –Except no trigrams were selected The annotator became too frustrated because trigrams in Arabic are mostly sentences. –Nahia Zorub selected DevSet terms, Essa Zorub selected the EvalSet Selection used diacritized transcripts –Non-diacritized terms derived by removing the diacritics from the diacritized terms – Whoops! –77 non-diacritized with long vowels thrown out Lower density rates

STD 2006 Arabic Results DiacritizedNon-Diacritized BBN1p * BUT1p, 3c2c DOD1p, 1c * BBN and DOD only processed the CTS data Diacritized results are not comparable to the Non-Diacritized

Diacritized Arabic CTS SiteIndexing Rate (Hp/Hs) ATWVMTWV BUT DOD

Non-Diacritized Arabic CTS SiteIndexing Rate (Hp/Hs) ATWVMTWV BBN

Diacritized Arabic BNews SiteIndexing Rate (Hp/Hs) ATWVMTWV BUT

Conclusions Considerable problems were uncovered working with the transcripts –Diacritized vs. Non-Diacritized distinction isn’t clear –Diacritic annotation has a low inter-annotator agreement This probably swamps the specificity benefit Biggest roadblock to moving forward: –Deciding how to handle diacritics