1. 1/20/20001 Human-Computer Interaction Research on the Endeavour Expedition James A. Landay Jack Chen, Jason Hong, Scott Klemmer, Francis Li, Mark Newman, Anoop Sinha Endeavour Retreat January 20, 2000

2. 1/20/20002 Several Projects on UIs of the Future Designer’s Outpost SUEDE Multimodal Design Assistant Context-aware PDA Infrastructure Context-based Information Agent

3. 1/20/20003 The Future: Information-centric User Interfaces

4. 1/20/20004 Designer’s Outpost: Tangible Tools for Information Design

5. 1/20/20005 Designer’s Outpost: Tangible Tools for Information Design

6. 1/20/20006 Designer’s Outpost: Tangible Tools for Information Design

7. 1/20/20007 Designer’s Outpost: Tangible Tools for Information Design

8. 1/20/20008 Designer’s Outpost: Tangible Tools for Information Design

9. 1/20/20009 Designer’s Outpost: Tangible Tools for Information Design Two 640x480 USB web cams –camera above desk captures ink / IDs –camera below desk occlusion free captures structure ITI Visionmaker Desk –1280 x 1024 resolution –direct pen input Cross iPen tablet –high-res ink capture

10. 1/20/200010 One Future: Our Access to Information will be via Speech

11. 1/20/200011 SUEDE: Low-fidelity Prototyping for Speech-based User Interfaces

12. 1/20/200012 SUEDE: Low-fidelity Prototyping for Speech-based User Interfaces

13. 1/20/200013 SUEDE: Low-fidelity Prototyping for Speech-based User Interfaces

14. 1/20/200014 A Better Future: Our Access to Information will be via Multimodal UIs How do we combine speech, gesture, etc. into a UI design? –rapid production of “rough cuts” informal (sketching / “Wizard of Oz”) iterative design (user testing/fast mods) –generate initial code UIs for multiple devices designer adds detail / improve UI Study –uses of these novel modes alone e.g., SUEDE –construction of multimodal apps e.g., a multimedia notebook Early stages so far –based on inferring models Model

15. 1/20/200015 The Best Future: Multimodal UIs that are Aware of User’s Context Applications can be aware of –location –who is the user –what are they doing –who is nearby… DARPA has funded us to purchase 25-50 PDAs w/ –wireless communications –wireless infrastructure –built-in bar code scanners Idea –deploy across Soda –build/study interesting applications –looking for students to get involved use Hill/Culler/Ninja infrastructure & Motes Symbol SPT 1700

16. 1/20/200016 A Context-based Information Agent: Motivation Proactive Meeting Support –Capture human-to-human communication and context –Use communication and context to proactively find relevant info –Present the most useful info as non- intrusively as possible Searching for relevant items… ………… … …!! January

17. 1/20/200017 Design Space Input Presentation Search

18. 1/20/200018 Design Space - Input Human-Human Communication –Speech, Ink, Vision, Text Context –Who you are? –Who is speaking? –Who else is here? –Where am I? –When is it? –What calendar event am I in? –What todo item am I doing? –How busy am I? –… Input Presentation Search

19. 1/20/200019 Design Space - Search Structuring queries from input –Continuous streams of input –Simple approach - spot keywords –But, given richer sources of input, can we formulate better queries? –Which kinds of communication and context are useful for this task? Information sources to search –Personal Information (local files) –Group Information (local web pages) –Global Information (web search) Input Presentation Search

20. 1/20/200020 Design Space - Presentation Synchronous / Asynchronous –Show me as I do my work –Show me after I get back from lunch Aim for minimal attention –Attention should be on the task Tailor output to context –I'm busy, don't bother me at all –Show me only if it's really important Input Presentation Search

21. 1/20/200021 Low-Fidelity Prototype Run a low-fidelity prototype –Quick way of testing system that doesn't exist yet –Parts requiring lots of programming simulated by human First iteration –Speech-based agent, listens in on conversations –Uses web search engines –Presents combined search results –Basically an alternative front-end for web search engines Web search

22. 1/20/200022 Low-Fidelity Prototype Combined results –Categorized by time –Categorized by topic –Material explicitly referenced Endeavour homepage "Do you have a link to that?" "There's a paper from Xerox PARC…" –Related material not explicitly mentioned UW Portolano MIT Oxygen Project

23. 1/20/200023 Low-Fidelity Prototype Results Some survey results –People liked the general concept –Search results rated ok, not great –Didn't want to spend too much time (spent more time than we expected) –Control of when agent was running (turn on and off) –Real-time results –Multiple ways of organizing, and accessing, and filtering the info

24. 1/20/200024 Design Space – Refined Input Presentation Context Communication Ink, Text, Speech Who, Where, When, How, What Search Personal Info Group Info Public Info Calendar, Contacts, Email, Personal webpages, Notes Calendar, Group Notes, Contacts, Group webpages Webpages, Newsgroups, Digital Libraries

25. 1/20/200025 Design Space - Refined Input Presentation Context Communication Ink, Text, Speech Who, Where, When, What Search Personal Info Group Info Public Info Calendar, Contacts, Email, Personal webpages, Notes Calendar, Group Notes, Contacts, Group webpages Webpages, Newsgroups, Digital Libraries

26. 1/20/200026 Prototype Implementation –Speech-based input (IBM ViaVoice) –Start with keyword-based search –Uses Google search engine Problems –Speech recognition is poor –"Interesting" keywords not in dictionary Still in the process of improving the software

27. 1/20/200027

28. 1/20/200028 Continuing Work Better organization and navigation of results Peripheral displays Figuring out "right" rate to update results Another low-fi prototype using "precise" speech recognition Improving recognition rate –ICSI Speech Recognition Engine / ViaVoice 2000 –Crawl local web pages to expand dictionary

29. 1/20/200029 Comments?

30. 1/20/200030 Related Work Cyberguide and others Letizia Remembrance Agent MSR Implicit Queries XLibris