1. PhonePoint Pen: Using Mobile Phones to Write in Air Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Romit Roy Choudhury ACM MobiHeld 2009

2. Noting small pieces of information, quickly and effortlessly can be useful But aren’t there several solutions... Buy Milk

3. State of the Art  Sticky notes … Difficult to organize electronically  Difficult to organize electronically  Not always handy  Typing on phone keyboards Tiny keyboard sizes  Tiny keyboard sizes  Small inter-key spacing  Audio recording Cannot sketch diagrams  Cannot sketch diagrams  Time consuming to browse through voice messages

4. So, need a solution that is  Easy to use  Always-with-me  Allows sketching  Online/Searchable

5. Proposed Approach: PhonePoint Pen  User holds phone like a pen  Writes messages in air  Built-in accelerometer  Captures hand movement  Converts to image / text  Display text on screen  Also sent to user’s email address

6. Use Cases  Mashing with cameras  Superimposing text on pictures  Sketching  Giving directions to your home  Exchanging figures on the fly  One handed use  E.g., carrying a suitcase Duke University

7. Use Cases  Applications in healthcare  Hand movement disabilities  Speech problems after surgery  Military and emergency response  Quick and silent updates

8. Prototype Current prototype on Nokia N95 Equipped with 3-axes accelerometer Programmed usign Python + MATLAB several design challenges emerge …

9. Design Challenges (1)  Hands rotate while writing  Accelerometers only measure linear acceleration  Rotation injects ambiguity (Wii uses gyroscope) Proposed Approach: Hold phone in non-rotating grip Hold phone in non-rotating grip Determine angular orientation during the pause Determine angular orientation during the pause

10. Design Challenges (2)  Background Vibration (Noise)  Sensitive accelerometers  Jitter from hand vibrations Proposed Approach: Smooth the accelerometer readings with moving average Smooth the accelerometer readings with moving average Suppress acceleration values < threshold Suppress acceleration values < threshold

11. Design Challenges (3)  Computing Displacement  Ambiguity when acceleration is zero  Can be constant velocity or stationary  Worse with noise Proposed Approach: Detect change of direction Detect change of direction Reset velocity between strokes Reset velocity between strokes Reset

12. Design Challenges (4)  Lifting Pen from the Paper  e.g., dotting the “i” and crossing the “t” Proposed Approach: Impulse on the Z axis during the lift Impulse on the Z axis during the lift Off the Record – On the Record Off the Record – On the Record P Z-axis spikes

13. Design Challenges (5)  Character transition  e.g., B vs 13 Proposed Approach: Use delimited such as “.” Use delimited such as “.” 1.3 for 13 1.3 for 13

14. Drawing a Rectangle Raw Accelerometer Reading Raw Accelerometer Reading

15. Noise Smoothing using Moving Average Background Noise Suppression Velocity Plot after Avoiding Velocity Drifts Final Rectangle

16. Raw Accelerometer Data for M and i

17. Character Recognition

18. More PhonePen words …

19. 20 words tested with online OCR web service. 78% accuracy 20 words tested with online OCR web service. 78% accuracy

20. Results

21. Video Demo

22. Of Course, Not a Product Yet  Lowercase character recognition  Cursive handwriting more complicated  Need smaller hand movements in air  Youtube comment: looking crazy waving hands  Slashdot: Nearby people can get injured  Need increased robustness  20% failure when writing in air  5% failure when writing on a surface

23. Of Course, Not a Product Yet  What about longer messages  Sentences? Email replies?  Erasing alphabets  Shake your hands like using a eraser Ongoing work is pursuing these directions … Ongoing work is pursuing these directions …

24. Swipe your credit card and sign in air … The vision is:

25. Thanks Visit Systems Networking Research Group (SyNRG) @ Duke University Google “synrg duke”

26. Thank You! email: romit@ee.duke.eduromit@ee.duke.edu Web: http://synrg.ee.duke.edu/http://synrg.ee.duke.edu/

27. Noting small pieces of information, quickly and effortlessly can be useful So, what are the options... Call John @ 6658 Buy Milk Right on 3rd traffic light

28. Velocity Plot after Avoiding Velocity Drifts Final Rectangle