1. Pen Research Jay Pittman Development Lead Tablet PC Handwriting Recognition Microsoft Corporation Jay Pittman Development Lead Tablet PC Handwriting Recognition Microsoft Corporation

2. TDNN Time-Delayed Neural Network Ink is cut into segments via simple algorithm For Latin script, we just cut at bottoms For Arabic or Hindi, we may need a different algorithm Features are computed per segment Chebychev coefficients Variety of size metrics: width, height, offsets to neighbors, etc. Each supported letter or character has an output node Many have 2 outputs, for beginning of character and continuation of character Very large training sets are collected from tens of thousands of native writers Time-Delayed Neural Network Ink is cut into segments via simple algorithm For Latin script, we just cut at bottoms For Arabic or Hindi, we may need a different algorithm Features are computed per segment Chebychev coefficients Variety of size metrics: width, height, offsets to neighbors, etc. Each supported letter or character has an output node Many have 2 outputs, for beginning of character and continuation of character Very large training sets are collected from tens of thousands of native writers

3. Beam Search Very similar to speech recognition systems System lexicon Simple list of words (like a spellchecker) Stored as a trie Plus regular expressions for numbers, dates, times, currency, phone numbers, etc. Low-scoring sequences are “culled” from the trie as we go Converts character recognizer into word recognizer We can recognize a word even if you mangle one of the letters Supports sloppier (and therefore faster) handwriting Very similar to speech recognition systems System lexicon Simple list of words (like a spellchecker) Stored as a trie Plus regular expressions for numbers, dates, times, currency, phone numbers, etc. Low-scoring sequences are “culled” from the trie as we go Converts character recognizer into word recognizer We can recognize a word even if you mangle one of the letters Supports sloppier (and therefore faster) handwriting

4. 88868226357 4446157 23 92315194720 71125287913 53187928576 … … … 138182143 17 574390 71657914415 Output Matrix dog68 clog57 dug51 doom42 divvy37 ooze35 cloy34 doxy29 client22 dozy13 Ink Segments Top 10 List d 92 a 88 b 14 c 86 o 77 a 73 l 76 t 5 g 68 t 8 b 6 o 65 g 57 t 12 TDNN Space TDNN a b d o g a b t t c l o g t Lexicon e a … … … … … Beam Search a* a d* d g* g o* o TDNN + Beam Search

5. Personalization New In Vista Shape adaptation Collect samples from you Simple idea: continue same training we do at Microsoft, but only on your samples Implicit Explicit Text adaptation Collect your personal words from Word and outgoing emails New In Vista Shape adaptation Collect samples from you Simple idea: continue same training we do at Microsoft, but only on your samples Implicit Explicit Text adaptation Collect your personal words from Word and outgoing emails Capital I? dev RTM KKOMO dogfooding Qi trie featurize Herry

6. Open System Recognizer API is published Any recognizer may support this API A non-Microsoft recognizer can be installed, and it will be invoked by the inking platform Non-Microsoft recognizers available now in Japan, China, and Russia Compete with my group Or cover languages we don’t cover Or cover non-text (music, math, chemistry) Sorry: recognition result API is very text oriented Strings in Unicode Perhaps your text might be XML Recognizer API is published Any recognizer may support this API A non-Microsoft recognizer can be installed, and it will be invoked by the inking platform Non-Microsoft recognizers available now in Japan, China, and Russia Compete with my group Or cover languages we don’t cover Or cover non-text (music, math, chemistry) Sorry: recognition result API is very text oriented Strings in Unicode Perhaps your text might be XML http://www.microsoft.com/downloads/details.aspx?FamilyID=69640b5c-0ee9-421e-8d5c-d40debee36c2&displaylang=en Microsoft Tablet PC / Tablet PC Platform / Programming the Tablet PC / Creating a Recognizer Start / All Programs / Microsoft Tablet PC Platform SDK / Microsoft Tablet PC Platform SDK Documentation

7. Latin Orthography XP has U.S. English, U.K. English, German, French, Spanish, and Italian Vista adds Dutch and Brazilian Portuguese Currently working on Swedish, Danish, Norwegian (Bokmal), Finnish, Polish, Czech, Portugal Portuguese, Catalan, Romanian, Croatian, and Serbian No ship estimates are available Serbian is written in both Latin and Cyrillic Making plans for the next batch: Probably Bahasa Indonese (Indonesian), Hungarian, Turkish, Slovak, Slovene, Lithuanian, Estonian, Latvian, Vietnamese, Tagalog (Filipino), others TBD World’s largest orthography Largest count of languages Largest combined count of literates XP has U.S. English, U.K. English, German, French, Spanish, and Italian Vista adds Dutch and Brazilian Portuguese Currently working on Swedish, Danish, Norwegian (Bokmal), Finnish, Polish, Czech, Portugal Portuguese, Catalan, Romanian, Croatian, and Serbian No ship estimates are available Serbian is written in both Latin and Cyrillic Making plans for the next batch: Probably Bahasa Indonese (Indonesian), Hungarian, Turkish, Slovak, Slovene, Lithuanian, Estonian, Latvian, Vietnamese, Tagalog (Filipino), others TBD World’s largest orthography Largest count of languages Largest combined count of literates

8. East Asian Orthographies Ideographic Orthographies Completely different code base Focus on large count of characters XP has Japanese, Chinese (Simplified), Chinese (Traditional), and Korean Vista adds personalization, and improves cursive recognition Completely different code base Focus on large count of characters XP has Japanese, Chinese (Simplified), Chinese (Traditional), and Korean Vista adds personalization, and improves cursive recognition

9. Cyrillic and Greek Orthographies Same TDNN + beam search technology works equally well in Greek and Cyrillic But we must collect new training sets Cyrillic has more upper/lowercase confusion Russian Collection completed Currently working on recognizer No estimate on shipping date Some work also underway in Serbian Cyrillic: Russian, Ukrainian, Bulgarian, Serbian, Byelorussian, Macedonian, Kazakh Other former-Soviet Turkic republics are in a state of transition back to Latin scripts Serbian is written in both Latin and Cyrillic Same TDNN + beam search technology works equally well in Greek and Cyrillic But we must collect new training sets Cyrillic has more upper/lowercase confusion Russian Collection completed Currently working on recognizer No estimate on shipping date Some work also underway in Serbian Cyrillic: Russian, Ukrainian, Bulgarian, Serbian, Byelorussian, Macedonian, Kazakh Other former-Soviet Turkic republics are in a state of transition back to Latin scripts Serbian is written in both Latin and Cyrillic

10. Greek ΕλληνικήέδωσανμουγλώσσαΤη Ink by Dr. John Drakopoulos I was given a language that is Hellenic

11. Russian Ink by Vladimir Smirnov Кремль. Кремль, Ото слышал всех я про Kremlin, Kremlin. I’ve heard from everyone about [it]

12. Bi-Directional Orthographies Arabic and Hebrew Text written right-to-left, but numbers written left-to-right Arabic has its own digits Hebrew uses “western” digits No uppercase / lowercase distinction Arabic is cursive-only Up to 4 forms per letter (initial, medial, final, isolated) Hebrew is print-only 5 letters have a separate final form Abjabs (consonant alphabets) We have collections underway Some initial experimentation in both languages No estimate on shipping date Arabic script variations used for Farsi (Persian), Urdu, Kurdish, Azeri (in Iran), Pashto, Darwi, Baluchi, Sindhi (in Pakistan), Kashmiri (in Pakistan ) Text written right-to-left, but numbers written left-to-right Arabic has its own digits Hebrew uses “western” digits No uppercase / lowercase distinction Arabic is cursive-only Up to 4 forms per letter (initial, medial, final, isolated) Hebrew is print-only 5 letters have a separate final form Abjabs (consonant alphabets) We have collections underway Some initial experimentation in both languages No estimate on shipping date Arabic script variations used for Farsi (Persian), Urdu, Kurdish, Azeri (in Iran), Pashto, Darwi, Baluchi, Sindhi (in Pakistan), Kashmiri (in Pakistan )

13. Bi-Directional Example Arabic ١٢٣ 123 عمره [his] age عاماً Year[s] Ink by Ahmed Kamal

14. Bi-Directional Example Hebrew מבבל. Plimpton 322 [is] from Babylonia. 322 פלימפטון Ink by Ethan Zoller

15. Brahmic Orthographies Left to right, no uppercase / lowercase Abugidas (syllabic alphabets) Default vowel (short “Ə” or “uh” sound) is not written All other vowels require a vowel sign added to the consonant This includes a “no-vowel sign” Called halant (“choked”) in Hindi, virama in other Indic languages Vowel sound follows consonant sound There are also independent vowels Hindi Devanagari collection in progress Some initial experimentation No estimate on shipping date Brahmic scripts: Devanagari (Hindi, Marathi), Thai, Bengali, Gujarati, Gurmukhi (Punjabi), Tamil, Telugu, Kannada, Malayalam, Oriya, Sinhala, Khmer, Lao, Tibetan, Myanmar, Sindhi (in India), Kashmiri (in India) Left to right, no uppercase / lowercase Abugidas (syllabic alphabets) Default vowel (short “Ə” or “uh” sound) is not written All other vowels require a vowel sign added to the consonant This includes a “no-vowel sign” Called halant (“choked”) in Hindi, virama in other Indic languages Vowel sound follows consonant sound There are also independent vowels Hindi Devanagari collection in progress Some initial experimentation No estimate on shipping date Brahmic scripts: Devanagari (Hindi, Marathi), Thai, Bengali, Gujarati, Gurmukhi (Punjabi), Tamil, Telugu, Kannada, Malayalam, Oriya, Sinhala, Khmer, Lao, Tibetan, Myanmar, Sindhi (in India), Kashmiri (in India)

16. Vowel Signs “KAA” “KU” “KUU” “KO” “KOW” “KE” “KEE” “KA” “KAI” “KAY” “KR” “K”

17. Consonant Clusters ST halant RING Unicode buffer: A transliteration of the English word “string”

18. Brahmic Comparison K Kh G Gh C Ch Devanagari: क ख ग घ च छ Bengali: ক খ গ ঘ চ ছ Gurmukhi: ਕ ਖ ਗ ਘ ਚ ਛ Gujarati: ક ખ ગ ઘ ચ છ Tamil: க ச Telugu: క ఖ గ ఘ చ ఛ Kannada: ಕ ಖ ಗ ಘ ಚ ಛ K Kh G Gh C Ch Devanagari: क ख ग घ च छ Bengali: ক খ গ ঘ চ ছ Gurmukhi: ਕ ਖ ਗ ਘ ਚ ਛ Gujarati: ક ખ ગ ઘ ચ છ Tamil: க ச Telugu: క ఖ గ ఘ చ ఛ Kannada: ಕ ಖ ಗ ಘ ಚ ಛ

19. Demo Title Name Title Group Name Title Group

20. Early Feedback As for Vista, I am in love with the B2 Tablet implementation! I can see where the effort is going! The TIP is awesome and the handwriting recognition is vastly improved. There is hardly a time where I have to correct my input - no matter how sloppy I am. Unsolicited feedback from a Vista Beta II user whose identity I do not know

21. © 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Backup Slides

23. Devanaguri Example प “PA” [R sign] period [abbreviation] headline ल “LA” period [abbreviation] headline म “MA” ि [vowel sign] “I” ा [vowel sign] “AA” ि [vowel sign] “I” Pr. Ltd. [Private Limited]