1. PLT 2007 CSIS Shorthand Handwriting Recognition for Pen-Centric Interfaces Charles C. Tappert 1 and Jean R. Ward 2 1 School of CSIS, Pace University, New York, USA 2 Pen Computing Consultant, Massachusetts, USA

2. PLT 2007 CSIS Thesis: Pen-Centric, Chatroom-Like Shorthand Interfaces Will provide critical infrastructure for many pen-centric applications Will provide fast text input Will have greatest impact on applications running on small mobile devices

3. PLT 2007 CSIS Agenda Handwriting –Fundamental Property of Writing –Handwriting Recognition Difficulties Historical Shorthand Alphabets Online (Pen-Centric) Handwriting Recognition –Online more accurate than Offline Recognition –Online Info Can Complicate Recognition Process –Design Tradeoffs/Decisions Pen-Centric Shorthand Alphabets Pen-Centric Word/Phrase Shorthand Allegro/Chatroom Experimental Shorthand System

4. PLT 2007 CSIS Fundamental Property of Writing Differences between different characters are more significant than differences between different drawings of the same character This makes handwritten communication possible

5. PLT 2007 CSIS Fundamental Property of Writing Property holds within subalphabets of uppercase, lowercase, and digits, but not across them “I”, “l”, and “1” written with single vertical stroke “O” and “0” written similarly with an oval

6. PLT 2007 CSIS Handwriting Recognition Difficulties Shape, size, and slant variation Similarly shaped characters – U and V Careless writing –in the extreme, almost illegible writing Resolving difficult ambiguities requires sophisticated recognition algorithms, syntax/semantics

7. PLT 2007 CSIS Historical Shorthand Alphabets (prior to pen computing) Famous writings were written in shorthand –Cicero’s orations –Martin Luther’s sermons –Shakespeare’s and George Bernard Shaw’s plays We focus on shorthand appropriate for PDAs Two main types of shorthand –Non-geometric shorthand –Geometric shorthand Small number of basic shapes Shapes reused in multiple orientations

8. PLT 2007 CSIS Tironian Alphabet, 63 B.C.

9. PLT 2007 CSIS Stenographie Alphabet, 1602

10. PLT 2007 CSIS Stenographie Alphabet, 1602 Geometric shorthand – basic shapes/orientations

11. PLT 2007 CSIS Moon Alphabet, 1894 Geometric shorthand – basic shapes/orientations

12. PLT 2007 CSIS Other Historical Shorthand Systems Phonetic alphabets –Pitman (1837) –Gregg (1885) Systems for the blind –Braille (1824) Cursive shorthands –Gabelsberger (1834)

13. PLT 2007 CSIS Online (Pen-Centric) Handwriting Recognition Machine recognizes the writing as the user writes Digitizer equipment captures the dynamic information of the writing –Stroke number, order, direction, speed –A stroke is the writing from pen down to pen up

14. PLT 2007 CSIS Online (Pen-Centric) more accurate than Offline (Static) Recognition Can use both dynamic and static information Can often distinguish between similarly shaped characters –E.g., 5 versus S where the 5 is usually written with two strokes and the S with one stroke

15. PLT 2007 CSIS Online Information Can Complicate Recognition Process Segmentation ambiguities –Character-within-character problem – cl versus d Large number of possible variations –E can be written with one, two, three, or four strokes, and with various stroke orders and directions –Four-stroke E has 384 variations (4! stroke orders x 2 4 stroke directions)

16. PLT 2007 CSIS Design Tradeoffs/Decisions No constraints on the user –Machine recognizes user's normal writing User severely constrained –Must write in particular style such as handprint –Must write strokes in particular order, direction, and graphical specification Simplest is one stroke per character, one stroke direction, one shape

17. PLT 2007 CSIS Pen-Centric Shorthand Alphabets Some of the earliest were for CAD/CAM Others developed for text input on PDAs We review geometric and non-geometric shorthands appropriate for small devices Historical alphabets presented above could be used for machine recognition In addition to shape and orientation, stroke direction can differentiate among symbols

18. PLT 2007 CSIS Allen Alphabet

19. PLT 2007 CSIS Allen Alphabet Basic Shapes and Orientations

20. PLT 2007 CSIS Goldberg Alphabet

21. PLT 2007 CSIS Goldberg Alphabet Basic Shapes and Orientations

22. PLT 2007 CSIS Graffiti Alphabet ( non geometric )

23. PLT 2007 CSIS Allegro Alphabet (non geometric)

24. PLT 2007 CSIS Simplified Design Tradeoffs/Decisions for Graffiti and Allegro PDA Alphabets Small alphabet –one case rather than both upper and lowercase Small number of writing variations per letter –preferably only one One stroke per character (character = stroke) –allows machine to recognize each character upon pen lift Separate writing areas for letters and digits –avoids confusion of similarly shaped letters and digits

25. PLT 2007 CSIS Graffiti and Allegro Commercially Successful Shorthands High correspondence to Roman alphabet –Easier to learn –Graffiti used in Palm OS devices notably the Palm Pilot and Handspring models –Allegro used in Microsoft Windows devices Geometric alphabets not successful

26. PLT 2007 CSIS Pen-Centric Word/Phrase Shorthand e.g., Chatroom Shorthand Further increase speed of text entry Potential applications –Where input speed important –Where word/phrase abbreviations occur frequently – e.g., email

27. PLT 2007 CSIS Allegro/Chatroom Shorthand System Developed for M.S. dissertation –Student was hearing impaired –Developed as output component of communication system Handwriting to text to speech Two input writing areas –One for Allegro (all-purpose) –One for chatroom-like words/phrases (e.g., CUL, F2F)

28. PLT 2007 CSIS Allegro/Chatroom Shorthand System

29. PLT 2007 CSIS Allegro/Chatroom Shorthand System

30. PLT 2007 CSIS Allegro/Chatroom Shorthand System Preliminary Experimental Results Allegro/Chatroom pen-centric shorthand input faster than typing text and comparable to typing text and chatroom shorthand characters

31. PLT 2007 CSIS Conclusions: Pen-Centric, Chatroom-Like Shorthand Interfaces Will provide critical infrastructure for many pen-centric applications Will provide fast text input Will have greatest impact on applications running on small mobile devices