1. Resource Creation for Training and Testing of Transliteration Systems for Indian Languages Sowmya V.B. *, Monojit Choudhury *, Kalika Bali *, Tirthankar Dasgupta , Anupam Basu  *Microsoft Research Lab India, Bangalore, India  Society for Natural language Technology Research, Kolkata, India

2. Outline Transliteration for Indic Languages – Back transliteration for IME The Methodology for Collection and Transcription Data Analysis – Spelling Variation – Code-Mixing Conclusion

3. Transliteration Transliteration is the process of mapping a written word from a language-script pair to another language-script pair. Back-transliteration used for Indic Input Method Editors Example: – “ परम ”  “param” Forward Transliteration – “ शेयर ”  “share” Backward/Reverse Transliteration

4. MS Indic Language Input Tool

5. Methodology for Collection Three Languages: Hindi, Bangla and Telugu 18-20 near-native speakers for each language Users of Roman script for Indic language for email, chat, text etc For Hindi, regional variations represented in the demographics Mode of Collection – No “look and type” – controlled and uncontrolled – Collect natural user data

6. Methodology for Collection Dictation (Controlled) : – a set of 550 sentences for each language ranging from news corpus to blogs and other web content. – The selected sentences covered as many of the valid letter-letter combinations for that particular language as possible. – Recorded by native speakers of the language. – Every user was given 75 sentences for transcription. 50 sentences were common to all users and 25 were unique to a given user.

7. Methodology for Collection Scenario Writing (Uncontrolled) : – Users asked to choose two from topics ranging from popular movies to current news – Mimics blogging or email – Can edit and no time constraint – 100 words per user

8. Methodology for Collection Chat (Uncontrolled) : – Users asked to chat with researcher on topics like plan of the day, the weather, etc – Real-time communications – No scope for intensive editing – 75 words per user

9. Methodology for Transcription Back-transliterated manually Transcribers were instructed to mark – Code-mixing – Numerals Transcribed Unicode data aligned at word- level with User data (ASCII) semi-automatically Mismatches aligned manually using a simple UI

10. Data Analysis Total of ~2600 words per language Mode of Data Collection BanglaHindiTelugu Dictation (Common) 64271293413360 Dictation (Unique) 401665926030 Scenario 337740444279 Chat 264826982276 Total 164682626825945

11. Spelling Variation A significant percentage of words show spelling variation Zipf’s law: number of variants of high frequency words will be large, whereas that of the low frequency words will be fewer No. of variations of word (x-axis) vs No. of words having that much variation

12. Spelling Variation

13. Mapping >50 graphemes to 26 alphabets Consonants show less variation than vowels – राज being written raj, raaj, raja, raaja Regional conventions – ప్రభుత్వం being written as prabhutvam, prabutvam, prabhuthvam

14. Code-Mixing Code-mixing, or the interspersing of English words in Indian language, is frequently observed in chat, blog and email texts “This is a cricket ball” yaha kriket ball hai Potential code-mixing Genuine code-mixing

15. Code-Mixing The average %age of genuine code-mixing for Bangla, Hindi and Telugu 8%, 11% and 12%, respectively 13 users for Bangla, 15 for Hindi and 16 for Telugu show less than 6% genuine code-mixing. 10 users for Hindi and 2 for Telugu had 100% genuine- to-potential code-mixing.

16. Code-Mixing Chat data had more cases of genuine code mixing compared to scenario data – across all languages. The extent of genuine code-mixing across users have a similar trend for all the languages. The ratio of genuine to potential code-mixing is less than 50% for a considerable number of Bangla users. This indicates that there is a high tendency for Bangla users to type in non-English sound-based spellings for English words.

17. Conclusion Design and creation of a dataset for Hindi, Bangla and Telugu transliteration data Can be used for systematic evaluation as well as training of Machine Transliteration based systems, IMEs and others Methodology can be used for transliteration dataset creation Currently in the process of expanding this to other languages like Kannada and Tamil Initial analysis shows certain linguistic and socio-linguistic basis for user variations Deeper analysis to understand the effect of these features on user data

18. QUESTIONS? Thank-You