1. Visions for Mobile Devices Beyond Their Current Role Brad A. Myers Carnegie Mellon University Human Computer Interaction Institute bam@cs.cmu.edu http://www.bam.hcii.cmu.edu/ Brad A. Myers Carnegie Mellon University Human Computer Interaction Institute bam@cs.cmu.edu http://www.bam.hcii.cmu.edu/

2. Our Research on Handhelds, 1 Pebbles project Since 1997 Presented at Faculty Summit 2002, 2003, 2004 http://www.pebbles.hcii.cmu.edu How handhelds can be used together with desktop computers Remote Commander Shortcutter SlideShow Commander Pebbles project Since 1997 Presented at Faculty Summit 2002, 2003, 2004 http://www.pebbles.hcii.cmu.edu How handhelds can be used together with desktop computers Remote Commander Shortcutter SlideShow Commander

3. Our Research on Handhelds, 2 Pebbles Personal Universal Controller (PUC) Automatically create remote control panels on PocketPC and Phone for appliances Appliance description in XML Interfaces that are consistent for the user Automatically combine interfaces for multiple appliances Pebbles Personal Universal Controller (PUC) Automatically create remote control panels on PocketPC and Phone for appliances Appliance description in XML Interfaces that are consistent for the user Automatically combine interfaces for multiple appliances

4. Thoughts on Research Topics for the Future of Handhelds

5. Data Integration Multiple antennas in all devices Cellphone + BlueTooth + WiFi  Ability to be constantly connected Not just once a day for synchronization  Can re-think the data model Integration of all of the user’s data Small to large pieces of data Cut-and-paste across devices Making short notes and short annotations (meta- notes) a fundamental part of data that is stored Intelligent and automatic consistency of data  Never re-enter data more than once anywhere! Multiple antennas in all devices Cellphone + BlueTooth + WiFi  Ability to be constantly connected Not just once a day for synchronization  Can re-think the data model Integration of all of the user’s data Small to large pieces of data Cut-and-paste across devices Making short notes and short annotations (meta- notes) a fundamental part of data that is stored Intelligent and automatic consistency of data  Never re-enter data more than once anywhere!

6. Where is the Data? On desktop, “in the network”, on body, synchronized everywhere? Storage is basically free but still have the synchronization problem Could put on network, but then loose reliability, worry about security/theft Worry if lose mobile device with data Make control of what is shared with others easier than emailing On desktop, “in the network”, on body, synchronized everywhere? Storage is basically free but still have the synchronization problem Could put on network, but then loose reliability, worry about security/theft Worry if lose mobile device with data Make control of what is shared with others easier than emailing

7. Application Integration Portable applications across different form factors Run same application on handheld as on PC? Web pages Word  Pocket Word Automatic adaptation to I/O capabilities Even dynamically as application runs Would help with innovating in hardware since might just work on the new device Portable applications across different form factors Run same application on handheld as on PC? Web pages Word  Pocket Word Automatic adaptation to I/O capabilities Even dynamically as application runs Would help with innovating in hardware since might just work on the new device

8. Entering Data Text entry still hard Hand-helds have voice recorders, cameras, styluses for strokes and handwriting Easier on a phone to take a picture than to type Rich data types input and output Need better ability to handle all these types as easily as text Handwriting = picture = typed = spoken Anywhere can use one can use another Pictures as the answer to questions Use context, location, etc. so enter less information Intelligent interfaces to fill in field with right values? Use history, etc. to learn from user Text entry still hard Hand-helds have voice recorders, cameras, styluses for strokes and handwriting Easier on a phone to take a picture than to type Rich data types input and output Need better ability to handle all these types as easily as text Handwriting = picture = typed = spoken Anywhere can use one can use another Pictures as the answer to questions Use context, location, etc. so enter less information Intelligent interfaces to fill in field with right values? Use history, etc. to learn from user

9. Handheld and PC Augment Each Other When Nearby Touch screen, camera, voice recorder, phone modem, as PC’s I/O PC’s keyboard, mouse, big screen for handheld’s I/O Opportunistic annexing of anything in the environment (displays, keyboards, antennas, networks, etc.) Office, conference room, home, automobile E.g., Use car’s speakers, microphones, heads-up or nav- system display http://www.cc.gatech.edu/projects/PIE/projects/oppannex.shtml UI research issues: where best for: Control: across appliances/devices, input methods Output: displays on TV vs. front panel vs. handheld Touch screen, camera, voice recorder, phone modem, as PC’s I/O PC’s keyboard, mouse, big screen for handheld’s I/O Opportunistic annexing of anything in the environment (displays, keyboards, antennas, networks, etc.) Office, conference room, home, automobile E.g., Use car’s speakers, microphones, heads-up or nav- system display http://www.cc.gatech.edu/projects/PIE/projects/oppannex.shtml UI research issues: where best for: Control: across appliances/devices, input methods Output: displays on TV vs. front panel vs. handheld

10. Form Factors For Handhelds Fundamental human sizes: In Hand Phone, PDA -> smaller, pocket On Arm Tablet On Wrist Watch On Tables Clamshell Windows CE  not successful New Mini-pcs Laptop Wearables Sensors like BodyMedia Tradeoffs: portability, battery life, screen size (readability), text entry Fundamental human sizes: In Hand Phone, PDA -> smaller, pocket On Arm Tablet On Wrist Watch On Tables Clamshell Windows CE  not successful New Mini-pcs Laptop Wearables Sensors like BodyMedia Tradeoffs: portability, battery life, screen size (readability), text entry

11. Special Purpose Devices vs. Convergence Mobile game device, PDA, phone, pager, still camera, video camera, music player, movie player, watch, flash memory, … Star Trek: Tricorder, Communicator, Phaser, Medical scanner, portable memories Or one device that does it all? Marketing issues – will people create content (games, apps, movies) for your device? HCI issues Good game control UI ≠ good phone control Larger display ≠ smaller device Battery life Mobile game device, PDA, phone, pager, still camera, video camera, music player, movie player, watch, flash memory, … Star Trek: Tricorder, Communicator, Phaser, Medical scanner, portable memories Or one device that does it all? Marketing issues – will people create content (games, apps, movies) for your device? HCI issues Good game control UI ≠ good phone control Larger display ≠ smaller device Battery life

12. Convergence vs. Special Purpose, cont. If many devices, still should adhere to “enter once” principle Scattered around environment, vs. carry around in pocket/on body If many devices, still should adhere to “enter once” principle Scattered around environment, vs. carry around in pocket/on body

13. Idea: Disassemble the Parts Each function in separate device Speaker in ear, microphone, display on arm, processor in pocket, memory, antennas Integrate with watch display, input E.g., Intel “personal server” Addresses the one v. many issue? Specialized I/O mechanisms (sold separately?) attach to common core Game controller/game media reader Camera lens Each function in separate device Speaker in ear, microphone, display on arm, processor in pocket, memory, antennas Integrate with watch display, input E.g., Intel “personal server” Addresses the one v. many issue? Specialized I/O mechanisms (sold separately?) attach to common core Game controller/game media reader Camera lens

14. Other Research Areas for Handhelds New UIs for Handhelds Not just shrunk desktops More uses for cameras Handicapped access Digital inclusion using handhelds Security in general New UIs for Handhelds Not just shrunk desktops More uses for cameras Handicapped access Digital inclusion using handhelds Security in general

15. Summary Still lots of interesting and important research topics in the handheld / mobile area! Challenge: How can handheld devices improve the user interfaces of everything else, and not just be another gadget to be learned 6.5 billion people in the world 1.5 billion cell phones worldwide, vs. 500 million PCs (?) vs. only 46 million PDAs vs. 1 million TabletPCs Still lots of interesting and important research topics in the handheld / mobile area! Challenge: How can handheld devices improve the user interfaces of everything else, and not just be another gadget to be learned 6.5 billion people in the world 1.5 billion cell phones worldwide, vs. 500 million PCs (?) vs. only 46 million PDAs vs. 1 million TabletPCs

16. Visions for Mobile Devices Beyond Their Current Role Brad A. Myers Carnegie Mellon University Human Computer Interaction Institute bam@cs.cmu.edu http://www.bam.hcii.cmu.edu/ Brad A. Myers Carnegie Mellon University Human Computer Interaction Institute bam@cs.cmu.edu http://www.bam.hcii.cmu.edu/

18. -- Extras / Not Used --

19. Handhelds as Platform When Can’t Afford a PC Less-developed countries How use mobile phones for what we would use a PC for? What makes sense to try to do? Already bought a cell phone, computation is free, what can be done with it? Computers used for: Communication, entertainment, business stuff, … Collaboration through handheld as a way to bring experts to less-developed countries Applications in Health care, Education, Agriculture, etc. Specialized applications  how create apps. cheaper? Easier to program/customize Less-developed countries How use mobile phones for what we would use a PC for? What makes sense to try to do? Already bought a cell phone, computation is free, what can be done with it? Computers used for: Communication, entertainment, business stuff, … Collaboration through handheld as a way to bring experts to less-developed countries Applications in Health care, Education, Agriculture, etc. Specialized applications  how create apps. cheaper? Easier to program/customize

20. Text Entry on Handhelds Still Hard Lots of research, still not solved Text Entry MethodSpeed Multitap10 wpm average, 29 wpm tops T920 wpm tops Treo-Rim tiny keyboard30 wpm, up to 60 wpm Twiddler45 wpm average, 70 wpm tops (non-looking) QWERTY keyboard90 average; 192 wpm world-record Dvorjak keyboard90 average, 204 wpm world-record Normal Speaking175 wpm American Sign Language175 wpm Court report (chording)300 wpm (world record) EdgeWriteQuikWritingSharkDasherXnav

21. More Uses for Camera Scanner for text, UPC reader, sign translation Sensor for movement, Mouse replacement Gesture recognition Where am I? (augment GPS) Games that use camera as a sensor Face recognition Plant recognition, disease recognition, … Type on table with camera detecting virtual keys Collaboration support: show the objects discussed But needs much better integration  still too hard to send/receive pictures Scanner for text, UPC reader, sign translation Sensor for movement, Mouse replacement Gesture recognition Where am I? (augment GPS) Games that use camera as a sensor Face recognition Plant recognition, disease recognition, … Type on table with camera detecting virtual keys Collaboration support: show the objects discussed But needs much better integration  still too hard to send/receive pictures

22. New User Interface Metaphor? Completely new UI metaphor that is appropriate for phones/mobile devices Maybe not just applying the 25-year-old desktop UI to the phone What could be done? Haptic interfaces for mobile devices Besides just vibrate More appropriate than for desktop Completely new UI metaphor that is appropriate for phones/mobile devices Maybe not just applying the 25-year-old desktop UI to the phone What could be done? Haptic interfaces for mobile devices Besides just vibrate More appropriate than for desktop

23. Other Research Issues Handicapped access Blind: Braille handhelds?, talking Muscular difficulties Cognitive difficulties Security in general Safety of data Trustedness of devices Transmission of data Handicapped access Blind: Braille handhelds?, talking Muscular difficulties Cognitive difficulties Security in general Safety of data Trustedness of devices Transmission of data