1. Using Hand-Held Computers and PCs Together: The Pebbles Project Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles Brad A. Myers bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles

2. Brad Myers 2 Multiple Devices l Most of our time spent in places where there is embedded technology l Offices, meeting rooms, classrooms, homes l Often multiple devices will be available: l Mobile phone and PDA and a PC l Multiple people’s PDAs l PDAs in a “Smart Room” l Most of our time spent in places where there is embedded technology l Offices, meeting rooms, classrooms, homes l Often multiple devices will be available: l Mobile phone and PDA and a PC l Multiple people’s PDAs l PDAs in a “Smart Room”

3. Brad Myers 3 Handhelds will be communicating l 802.11 l BlueTooth l Cell-phone network l (Infrared) l 802.11 l BlueTooth l Cell-phone network l (Infrared)

4. Brad Myers 4 Premises of our Research “With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be in close, interactive communication.” — Brad Myers, “Using Hand-Held Devices and PCs Together,” Comm. ACM, Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.

5. Brad Myers 5 Research Agenda How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices?

6. Brad Myers 6 Research Agenda How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? “Multi-Machine User Interfaces”

7. Brad Myers 7 Example: Power Point Control l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, list of titles, timer, etc. l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, list of titles, timer, etc.

8. Brad Myers 8 Pebbles is: PEBBLESPEBBLES PEBBLESPEBBLES DAs for ntry of oth ytes and ocations from xternalources. DAs for ntry of oth ytes and ocations from xternalources. http://www.pebbles.hcii.cmu.edu/

9. Brad Myers 9 Handhelds in an Office l Augment desktop applications l Use multiple devices at the same time l Augment desktop applications l Use multiple devices at the same time

10. Brad Myers 10 Handhelds in Meetings l Attendees use handhelds to interact and annotate presentation l Augment collaboration l Take notes l Attendees use handhelds to interact and annotate presentation l Augment collaboration l Take notes

11. Brad Myers 11 Handhelds in Classrooms l Data projectors for instructor’s slides l Students could have computers for: l Notetaking linked to instructor’s slides l In-class testing l Running simulations l Data projectors for instructor’s slides l Students could have computers for: l Notetaking linked to instructor’s slides l In-class testing l Running simulations

12. Brad Myers 12 Handhelds in Homes l Interact with embedded computation l “Smart homes” l Not just speech and vision as interfaces l Interact with appliances, lights, etc. l “Personal Universal Controller” (PUC) l Interact with embedded computation l “Smart homes” l Not just speech and vision as interfaces l Interact with appliances, lights, etc. l “Personal Universal Controller” (PUC)

13. Office Use (For Individuals) l How can handhelds augment desktop applications?

14. Brad Myers 14 Applications for Individuals l Extra input and output devices have been shown to be useful l But can be expensive and hard to configure l People have PDAs and are attached to PC l For example, cradles for recharging l Customizable, extensible l Extend desktop applications l Extra input and output devices have been shown to be useful l But can be expensive and hard to configure l People have PDAs and are attached to PC l For example, cradles for recharging l Customizable, extensible l Extend desktop applications

15. Brad Myers 15 Scrolling with the PDA l For scrolling using the non-dominant hand l Studies showed parallel and efficient uses of both hands together l Generates Windows scrolling events l For scrolling using the non-dominant hand l Studies showed parallel and efficient uses of both hands together l Generates Windows scrolling events

16. Brad Myers 16 Results of Study of Scrolling with PDA l Scrolling with buttons on PDA was fastest l PDA scrollers similar to mouse speed l Using 2 hands is effective! l Scrolling with buttons on PDA was fastest l PDA scrollers similar to mouse speed l Using 2 hands is effective! Winner!

17. Brad Myers 17 Remote Clipboard l Transfer information between PDA and PC l Connects their clipboards together l Transfer content or reference l Works with all applications l Also between multiple computers l Transfer information between PDA and PC l Connects their clipboards together l Transfer content or reference l Works with all applications l Also between multiple computers

18. Brad Myers 18 Shortcutter l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l Palm or PocketPC l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l Palm or PocketPC

19. Brad Myers 19 Shortcutter Widgets l Buttons l Sliders l Knobs l Mouse pad l Graffiti Pad (Palm) l Gesture panel l Buttons l Sliders l Knobs l Mouse pad l Graffiti Pad (Palm) l Gesture panel

20. Brad Myers 20 Shortcutter Actions l Send any keyboard key, mouse button, scrolling action or string to PC l Open a file or URL l Run an application l Invoke any PC menu or button l Windows message l Recorded l Switch to a different Shortcutter panel l Control the Mouse l Send any keyboard key, mouse button, scrolling action or string to PC l Open a file or URL l Run an application l Invoke any PC menu or button l Windows message l Recorded l Switch to a different Shortcutter panel l Control the Mouse

21. Brad Myers 21 Shortcutter Actions, cont. l Control external devices through PC’s serial port l Directly (e.g., projectors) l X-10 for electrical devices l Macro l Can be multi-application l Application-specific l Same button, different messages l Useful for application sets: browsers, compilers l Control external devices through PC’s serial port l Directly (e.g., projectors) l X-10 for electrical devices l Macro l Can be multi-application l Application-specific l Same button, different messages l Useful for application sets: browsers, compilers

22. Brad Myers 22 More Scenarios of Use l Lean-back mail reading l Controlling WinAmp l … and many others l Lean-back mail reading l Controlling WinAmp l … and many others

23. Brad Myers 23 Study of Individual Use l Time to tap on button depended on size l Few errors l People often didn’t look at PDA l Time to tap on button depended on size l Few errors l People often didn’t look at PDA

24. Brad Myers 24 Study of Individual Use l Moving hands to both PDA and mouse only about 15% slower than just moving to the mouse 1H Keyboard->Mouse728 1H Keyboard->PDA744 1H Mouse->Keyboard701 1H PDA->Keyboard639 Keyboard -> Mouse&PDA83815.1% Mouse&PDA -> Keyboard79112.8% % slower msec

25. Use in Meetings l Enhance group’s collaboration and control

26. Brad Myers 26 Original Application: Remote Commander l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l Perform all mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l Perform all mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard

27. Brad Myers 27 PocketPC version l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons

28. Brad Myers 28 Scribble l Multiple people draw on top of whatever on PC screen, not just PowerPoint l Each user has own cursor and color l Save by PrintScreen l Erase by refresh l Multiple people draw on top of whatever on PC screen, not just PowerPoint l Each user has own cursor and color l Save by PrintScreen l Erase by refresh

29. Brad Myers 29 SlideShow Commander l For PowerPoint l PC Only: Windows 95, 98, NT, 2000, ME, XP l Use PC to give the presentation l Full features of PowerPoint l Use hand-held as “remote control” for PC l Can wander away from the keyboard l Two-way communication l View, Navigate, Highlight, Time l For PowerPoint l PC Only: Windows 95, 98, NT, 2000, ME, XP l Use PC to give the presentation l Full features of PowerPoint l Use hand-held as “remote control” for PC l Can wander away from the keyboard l Two-way communication l View, Navigate, Highlight, Time

30. Brad Myers 30 View on Hand-Held: slide l See thumb-nail of current slide l Black and white or color l See thumb-nail of current slide l Black and white or color

31. Brad Myers 31 View on Hand-Held: notes l See the notes of the current slide

32. Brad Myers 32 View on Hand-Held: titles l See the list of titles

33. Brad Myers 33 View on handheld: Time l Multi-function timer l View large l And in corner l Multi-function timer l View large l And in corner

34. Brad Myers 34 Control Demonstrations on Palm l Control PC applications and external devices l Uses Shortcutter l Easy demo and resume show l Control PC applications and external devices l Uses Shortcutter l Easy demo and resume show

35. Brad Myers 35 Control Demonstrations on PocketPC l Task list of PC applications l Tap brings one to front l PowerPoint continues behind – easy to resume l Task list of PC applications l Tap brings one to front l PowerPoint continues behind – easy to resume

36. Brad Myers 36 Commercialized l SlideShow Commander commercialized by: l Available at Office Depot, CompUSA, etc. l Palm and PocketPC l SlideShow Commander commercialized by: l Available at Office Depot, CompUSA, etc. l Palm and PocketPC Synergy Solutions, Inc. http://www.synsolutions.com/ Synergy Solutions, Inc. http://www.synsolutions.com/ http://www.slideshowcommander.com/

37. Brad Myers 37 Studies of Presentations l Summer study of 2 HCI Master’s students l Contextual Inquiry of 9 presentations l Audience: 10 to hundreds l 7 used PowerPoint, 4 used NetMeeting l Found 220 “breakdowns” l Most were minor problems l Averaged 8.7 lost minutes per talk (14.5%) l Designed SlideShow Commander to eliminate some of these problems l Summer study of 2 HCI Master’s students l Contextual Inquiry of 9 presentations l Audience: 10 to hundreds l 7 used PowerPoint, 4 used NetMeeting l Found 220 “breakdowns” l Most were minor problems l Averaged 8.7 lost minutes per talk (14.5%) l Designed SlideShow Commander to eliminate some of these problems

38. Brad Myers 38 MultiCursor l For special applications that are aware of multiple inputs l E.g.: Shared Whiteboard l Single Display Groupware l Multiple people, one display l A number of interesting issues: l Palettes, widgets l Section handles l Undo l For special applications that are aware of multiple inputs l E.g.: Shared Whiteboard l Single Display Groupware l Multiple people, one display l A number of interesting issues: l Palettes, widgets l Section handles l Undo

39. Brad Myers 39 Chat l Communicate to another PDA user through the PC l PC serves as a conduit l For side notes and messages l For example, in negotiation meetings l Send to all or to a specific person l Communicate to another PDA user through the PC l PC serves as a conduit l For side notes and messages l For example, in negotiation meetings l Send to all or to a specific person

40. Brad Myers 40 Command Post of the Future l Large DARPA funded project l Make commanders more effective l Large DARPA funded project l Make commanders more effective

41. Brad Myers 41 Study of Laser Pointers l Studied properties of laser pointer interaction techniques l Hand-wiggle  8 pixels l Delay until moving average stable  1.5 sec l Delay until target acquired  1.5 sec l Different devices and poses do not help much l Studied properties of laser pointer interaction techniques l Hand-wiggle  8 pixels l Delay until moving average stable  1.5 sec l Delay until target acquired  1.5 sec l Different devices and poses do not help much

42. Brad Myers 42 Tap Speed Study l Tapping directly on SmartBoard is fastest l Semantic Snarfing next l But high error rate l Then regular mouse l Laser pointing slowest l Used separate physical button on handheld l Tapping directly on SmartBoard is fastest l Semantic Snarfing next l But high error rate l Then regular mouse l Laser pointing slowest l Used separate physical button on handheld

43. Brad Myers 43 “Semantic Snarfing” l Interacting at a distance l Grab contents to handheld l Picture l Menus l Text l Re-visualization l “Magic Lenses” l Interacting at a distance l Grab contents to handheld l Picture l Menus l Text l Re-visualization l “Magic Lenses”

44. Classroom Use l Using handheld computers in classrooms l For testing l Improve large lectures l Provide immediate feedback to instructor l Using handheld computers in classrooms l For testing l Improve large lectures l Provide immediate feedback to instructor

45. Brad Myers 45 Hardware l HP donated 110 Jornada 680 and 100 720 computers l Windows CE l Lucent donated Wavelan wireless cards l HP donated 110 Jornada 680 and 100 720 computers l Windows CE l Lucent donated Wavelan wireless cards

46. Brad Myers 46 Context l Collaborator: Prof. David Yaron of Chemistry l Chemistry 09-106: Modern Chemistry II l Spring, 2000 and Spring, 2001 l About 90 students l Mostly freshmen l Loaned one Jornada and Wavelan card to each student for the whole semester l So would get used to using it l Offset technical difficulties with benefits to student l Collaborator: Prof. David Yaron of Chemistry l Chemistry 09-106: Modern Chemistry II l Spring, 2000 and Spring, 2001 l About 90 students l Mostly freshmen l Loaned one Jornada and Wavelan card to each student for the whole semester l So would get used to using it l Offset technical difficulties with benefits to student

47. Brad Myers 47 Wireless Andrew l CMU has almost complete coverage of 802.11b support l Wavelan l Donations from Lucent, etc. l Funding from state of Pennsylvania, etc. l Can use computers wirelessly in all classrooms and offices l Most lawns, dorm rooms, etc. l Windows, Mac, Linux, Windows CE support l CMU has almost complete coverage of 802.11b support l Wavelan l Donations from Lucent, etc. l Funding from state of Pennsylvania, etc. l Can use computers wirelessly in all classrooms and offices l Most lawns, dorm rooms, etc. l Windows, Mac, Linux, Windows CE support

48. Brad Myers 48 Pebbles Tests l Create through html pages l Arbitrary html in questions and answers l Pictures, formatting, links l Use FrontPage, etc. to author l Embed tags to show question and answers l Create through html pages l Arbitrary html in questions and answers l Pictures, formatting, links l Use FrontPage, etc. to author l Embed tags to show question and answers

49. Brad Myers 49 Concept Test l Students allowed to answer multiple times l Server stores all answers l Prof. Yaron displayed questions and answers on the board l Evidence that significantly contributes to learning — Mazur, E. (1997). Peer Instruction: A User's Manual, Prentice Hall. l Students allowed to answer multiple times l Server stores all answers l Prof. Yaron displayed questions and answers on the board l Evidence that significantly contributes to learning — Mazur, E. (1997). Peer Instruction: A User's Manual, Prentice Hall.

50. Brad Myers 50 Instructor’s View l In class, instructor sees a chart of answers l Optionally refreshes every 3 seconds l May be projected for whole class to see l Top displays instructions for students l In class, instructor sees a chart of answers l Optionally refreshes every 3 seconds l May be projected for whole class to see l Top displays instructions for students

51. Brad Myers 51 Results l Unfortunately, no data (yet) about effects on learning l Students thought concept tests were a valuable part of the class l Students significantly preferred using handhelds for concept tests. l Unfortunately, no data (yet) about effects on learning l Students thought concept tests were a valuable part of the class l Students significantly preferred using handhelds for concept tests. Raise HandsHold up cardsHandhelds 25%4%63%

52. Home Use l Personal Universal Controller l Help for people with muscular disabilities l Personal Universal Controller l Help for people with muscular disabilities

53. Brad Myers 53 Personal Universal Controller l Two-way communication l Appliances describe their functions l Handheld PUC creates interface based on descriptions l Handheld PUC controls the appliance l Appliance sends back status as feedback l Two-way communication l Appliances describe their functions l Handheld PUC creates interface based on descriptions l Handheld PUC controls the appliance l Appliance sends back status as feedback Specifications Control Feedback of Status

54. Brad Myers 54 Initial Experiments l Use real PocketPC l Pretend that controls devices l Use real PocketPC l Pretend that controls devices

55. Brad Myers 55 PocketPC results l For both appliances, users of actual interfaces: l Took about twice as long l Made at least twice as many mistakes as users of the handheld interfaces l Needed external help five times more often l For both appliances, users of actual interfaces: l Took about twice as long l Made at least twice as many mistakes as users of the handheld interfaces l Needed external help five times more often

56. Brad Myers 56 Current Steps l XML specification language and protocol for describing appliances l Create panels automatically from the specification l Find real appliances which we can control l XML specification language and protocol for describing appliances l Create panels automatically from the specification l Find real appliances which we can control Specification Language <spec xmlns="puc.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSc hema-instance" xmlns:xsd="http://www.w3.org/2001/XMLS chema" name="Audiophase 5 CD Stereo"> Off On Stereo Power Power Powr Pwr 10

57. Brad Myers 57 Current Work: Handhelds for People with Muscular Disabilities l Using handhelds as interface to PCs l People with Muscular Dystrophy have fine-motor control but lose gross motor control l Difficulties with mouse and keyboard, but stylus OK l Handhelds as interface to other devices l Control room lights, telephone, wheelchair, etc. l Using handhelds as interface to PCs l People with Muscular Dystrophy have fine-motor control but lose gross motor control l Difficulties with mouse and keyboard, but stylus OK l Handhelds as interface to other devices l Control room lights, telephone, wheelchair, etc.

58. Brad Myers 58 Remote Commander Adaptations l Faster Acceleration l More flexible tapping l Turn off key repeat l Multiple Keyboards l Faster Acceleration l More flexible tapping l Turn off key repeat l Multiple Keyboards

59. Brad Myers 59 Any PC app. PebblesPC Brad’s Palm Pilot Rob’s Pocket PC Ben’s Windows CE RemoteCmd MultiCursor On the PC PebblesDraw Various PDA apps Slideshow Commander PowerPoint OLE Automation Various dlls General Architecture Windows event stream Direct connection or sockets Serial, IR or sockets

60. Brad Myers 60 Protocols l Can use Pebbles protocols to develop your own application l Libraries for Palm, Windows CE, PC l Independence from communication medium l Use by: l PalmAmp from IronCreek Software l Intel research l others… l Windows messages or sockets l Can use Pebbles protocols to develop your own application l Libraries for Palm, Windows CE, PC l Independence from communication medium l Use by: l PalmAmp from IronCreek Software l Intel research l others… l Windows messages or sockets

61. Brad Myers 61 Downloads l Most of this software is available for free downloading l First release in Feb, 1998 l Just released version 5 (Feb, 2002) l Downloaded over 30,000 times l About 200 times a week l Most of this software is available for free downloading l First release in Feb, 1998 l Just released version 5 (Feb, 2002) l Downloaded over 30,000 times l About 200 times a week http://www.pebbles.hcii.cmu.edu/

62. Brad Myers 62 Conclusions l Handhelds are becoming ubiquitous l Connecting technologies will improve l Important to study how can be used when connected to computers and each other l “Multi-Machine User Interfaces” will be increasingly important l People will want to use the most convenient device for their information and control l Handhelds are becoming ubiquitous l Connecting technologies will improve l Important to study how can be used when connected to computers and each other l “Multi-Machine User Interfaces” will be increasingly important l People will want to use the most convenient device for their information and control

63. Brad Myers 63 Thanks to Our Sponsors! Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies IBM SMART Technologies, Inc. Synergy Solutions, Inc. Handango

64. Using Hand-Held Computers and PCs Together: The Pebbles Project Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles Brad A. Myers bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles