1. Interaction Techniques with Mobile Devices Jingtao Wang jingtaow@cs.berkeley.edu March 6th, 2006 Guest Lecture for CS160

2. Agenda Why Mobile Devices Matters Ubiquitous Computing Key Challenges in Designing Mobile Applications Input Techniques for Mobile Devices Output Techniques for Mobile Devices Interact With Other Devices

3. Why Mobile Devices Matters 6.5 billion people in the world 1.5 billion cell phones worldwide 500 million PCs (?) 46 million PDAs 1 million TabletPCs Challenge: How can handheld devices improve the user interfaces of everything else, and not just be another gadget to be learned

4. Agenda Why Mobile Devices Matters Ubiquitous Computing Key Challenges in Designing Mobile Applications Input Techniques for Mobile Devices Output Techniques for Mobile Devices Interact With Other Devices

5. Mark Weiser (1952 – 1999) Introduced the idea of ubiquitous computing

6. Weisers Vision The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it

7. Weisers Vision: Pervasive Mainframe many people share 1 computer PC 1 person with 1 computer Ubicomp many computers server each person

8. Agenda Why Mobile Devices Matters Ubiquitous Computing Key Challenges in Designing Mobile Applications Input Techniques for Mobile Devices Output Techniques for Mobile Devices Interact With Other Devices

9. Key Challenges in Making Mobile Applications Limited Physical Resources CPU, Memory, Screen Size, Input Devices, Battery Life etc Diversified Context of Use Different Activities Limited Attention

10. Limited Physical Resources A mobile device usually has 1/100 CPU power, 1/30 Screen resources, 1/20 Memory, and extremely limited input devices when compared with desktops in the same era. Small Screen Geography is different a. Large Screen b. Small Screen

11. Diversified Context of Use

12. Different Activities People use small-screen devices for different activities than desktops; dont assume you understand these activities already

13. Limited Attention Dont assume your applications have peoples full attention; theyre doing something else while using your device.

14. Context, Activity, Attention There is more opportunity for purpose-specific or context-specific devices than for general-purpose solutions that try to work for everyone in any situation.

15. One Sentence Summary Mobilize, Dont Miniaturize ! There is no silver bullet in designing mobile applications, but there is one sentence you should remember -

16. Agenda Why Mobile Devices Matters Ubiquitous Computing Key Challenges in Designing Mobile Applications Input Techniques for Mobile Devices Output Techniques for Mobile Devices

17. Input Techniques for Mobile Devices Pointing Text Input (Virtual) Keyboard Input Handwriting Input Speech Input Marker Based Input

18. Common Pointing/Navigation Techniques iPod Dialpad TrackPoint JogDial Touch Screen Four-directional keypad

19. TinyMotion – Camera Phone Based Pointing Detecting the movements of cell phones in real time by analyzing image sequences captured by the built-in camera. Typical movements include - horizontal and vertical movements, rotational movements and tilt movements.

20. Input Techniques for Mobile Devices Pointing Sensor Augmented Input Text Input (Virtual) Keyboard Input Handwriting Input Speech Input Marker Based Input

21. (Virtual) Keyboard Input

22. How to Make QWERTY Keyboards Portable ? Break

23. Making QWERTY Keyboards Portable Reducing the Absolute Size Reducing the Number of Keys Making Keyboards Foldable Projective Keyboard

24. From http://www.vkb-tech.com

25. Projective Keyboard – Working Mechanisms 1. Template creation 2. Reference plane illumination 3. Map reflection coordinates 4. Interpretation and communication

26. Can We Perform Better Than QWERTY? Originally QWERTY layout is manually optimized for two handed, alternative typing to minimize mechanical jam OPTI ATOMIK OPTI II FITALY

27. Theories Behind Quantitative Keyboard Layout Optimization Fitts Law Digraph Distribution Model in a Language

28. Can We Use Less Buttons than a Full QWERTY? 12-button Keypad 15-button Keyboard Half Keyboard

29. Disambiguation Methods for Reduced Size Keyboard The QWERTY keyboard itself is ambiguous! ( A vs. a, 3 vs. #) Pressing Several Keys together (shift key) Multiple Key Press Multi-Tap (22.5 wpm*) Two-Key Input (25.0wpm*) Dictionary/Statistics Based Disambiguation Methods T9/T15 (45.7 wpm*) LetterWise *Performance Upper Bound Estimation from Silfverberg 2000

30. FingerSense – Button Disambiguation by Fingertip Identification Differentiating a pressing action by identifying the actual finger involved Can be Faster than Regular Tapping When the Adjacent Tapping Involves Different Fingers and Different Buttons (59% on a phone keypad)

31. Input Techniques for Mobile Devices Pointing Sensor Augmented Input Text Input (Virtual) Keyboard Input Handwriting Input Speech Input Marker Based Input

32. Handwriting Input 1938 George Hansel, U.S. Patent 2,143,875, machine recognition of handwriting 1957 T. L. Dimond's stylator - the first on-line handwriting recognizer prototype Newton, Palm Pilot, PocketPC, CrossPad, TabletPC

33. Handwriting Recognition - Terminology Printed Character Recognition (OCR) Relatively mature these days, key challenges – layout analysis, fonts recovery, robust recognition for low quality, low resolution input Major Usage – Digital Library Handwritten Character Recognition Online HWR (With Temporal info) Character, Word, Sentence Level Offline HWR (Using raster image as input, no temporal info) Major Usage : Bank Check Recognition, Postal Automation

34. Word/Sentence Level Recognizer Build on Top of Character Recognizer General Strategy Over Segmentation Call Character/Component Recognizer, Get a List of Candidates with Scores Apply Geometry Spatial Information ( size, component gap ) Language Information (Dictionary, Language Model etc) to Each Sub Path Use Hypnosis Search (Dynamic Programming, A* etc) to Determine the Best Possible Path

35. Challenges in Online Handwriting Recognition Character Set/Dictionary Size (Especially Asian Languages!) Cursive Writing Styles/Broken Strokes/Duplicate Strokes/Omitted Components Stroke Order Variations Limited memory and CPU Power in Small Devices

36. Some Prototype Recognizers from IBM

37. New Form Factors - Anoto Pen Commercial Product is Available In the U.S. Market (Logitech IO Pen) Uses A Camera Mounted Beside the Tip of the Pen and Preprinted Dot Patterns to Detect Pen Movment

38. SHARK – Shorthand Writing on Stylus Keyboard A Combination of Virtual Keyboard and Handwriting Recognition Writing Shape of a Word (Shorthand) is Defined By the on Screen Location of Characters in the Word

39. EdgeWrite Input An EdgeWrite user enters text by traversing the edges and diagonals of a square hole imposed over the usual text input area Faster and More Reliable Than Regular Graffiti Especial Useful for People with Motor and Muscle Disabilities

40. Input Techniques for Mobile Devices Pointing Sensor Augmented Input Text Input (Virtual) Keyboard Input Handwriting Input Speech Input Marker Based Input

41. Input Techniques for Mobile Devices Pointing Sensor Augmented Input Text Input (Virtual) Keyboard Input Handwriting Input Speech Input Marker Based Input

42. Emerging Marker Based Interactions on Camera Phones

43. Towards More Sensitive Mobile Devices

44. Agenda Why Mobile Devices Matters Ubiquitous Computing Key Challenges in Designing Mobile Applications Input Techniques for Mobile Devices Output Techniques for Mobile Devices Interact With Other Devices

45. Peephole Displays (With Demo)

46. Zoomable Interface on Mobile Devices ZoneZoom By Microsoft Take advantage of spatial memory VS.

47. Halo - A Virtual Periphery for Mobile Devices Provding Visual Cue for Objects Located Out of the Small Screen

48. Agenda Why Mobile Devices Matters Ubiquitous Computing Key Challenges in Designing Mobile Applications Input Techniques for Mobile Devices Output Techniques for Mobile Devices Interact With Other Devices

49. Using Mobile Devices with Desktop Computers Pebbles Project at CMU Using a PDA as additional keypad, touch pad, scroll wheel and controller of PointPoint slides for desktop Applications http://www.pebbles.hcii.cmu.edu/

50. Using Mobile Devices with Laptops Wang 2002

51. Using Mobile Devices with Large Displays Ballagas 2005

52. Question and Answer

53. Backup Slides

54. Electromyographic (EMG) Keyboard NASA Virtual Keyboard SenseBoardKeyboard

55. The General Flow of Handwriting Recognition

56. Major Players in this Area (Embedded) English ART - ART Recognizer CIT - Jot IBM - Derived from Multi-lingual version Microsoft - Transcriber ( Licensed version of Calligrapher) & Own Single character recognizer Motorola Paragraph - Calligrapher Chinese/Japanese FineArt - GoGoPen Hanwang - more than 70% PDA market share in mainland China IBM Embedded HWR Motorola Lexicus - DragonPen PenPower - most influencial in Taiwan

57. Sensor Augmented Pointing