1. Frontiers in User- Computer Interaction Brown University and the NSF Science and Technology Center for Computer Graphics and Scientific Visualization June 1999 Andries van Dam

2. Roadmap for Talk Introduction – background and goals of UIs – pros and cons of WIMP (Windows, Icons, Menus, and Point-and-click) GUIs (Graphical User Interface), e.g., MS Windows – motivation for post-WIMP UIs Post-WIMP interaction: From gesture and speech recognition to Multi-modal and Perceptual UIs (PUIs) Research problems Note: Interaction is hard to describe with static slides, even with video. You really must experience it…

3. Computer vs. Human Performance Compute Graphics Computing CapacityHuman Capacity t Goal: increase bandwidth to the brain UI t

4. None! UIs are a necessary evil Want human-style interactions with other humans, intelligent assistants, objects in the real and/or virtual world – intelligent assistant (IA) may be manifested as an agent with a “social interface”/avatar (see Apple’s Knowledge Navigator video) – prediction: when IAs become vastly superior to what is available today, they will mediate the majority of transactions The Ultimate User Interface

5. Intelligent Assistants: Understand You and Your Context Combine continuous speech recognition and natural language understanding, knowledge management and processing May have customizable social interfaces/avatars May run both synchronously (under direct user control) and asynchronously (communicating with other agents)

6. Today’s Best UI Goal: Transparency – minimal cognitive processing, relying mostly on motor skills and subconscious perception – focus on task, not technology (e.g., bicycle, car, and our Sketch examples. MS PowerPoint is a negative example!) – ease of learning is not the same as ease of use – context determines how much of an investment the user is willing to make (e.g., playing a musical instrument)

7. What to Use Moore’s Law for? Computers are now fast enough for most applications Increasingly, we can afford to concentrate on human productivity rather than machine productivity (8 CDs for Office 2000!) How? Use compute power to make dramatic improvements in user interfaces and capabilities (new modes of expression) – today, majority of code in desktop productivity applications is in the GUI (e.g., MS Office), and we’re only tinkering – need radical departure, e.g., Raj Reddy in the early 80s proposed SILK interfaces -- not yet here ospeech, images, language, knowledge base

8. History of UIs: Four Generations 1960’s Batch 1970’s Command Line on  # CRT 1980-90’s WIMP GUI 1999? Post-WIMP Each paradigm shift led to large increase in user base

9. DOS Command Line Interface

10. GUIs Reduce Typing

11. 1984 Apple Macintosh Screen

12. An Unanticipated and Revolutionary Result of GUIs Even young children who can’t yet read and write are productive computer users

13. Advantages of WIMP GUIs Xerox PARC’s legacy provides a “standard” => ease of x for users But not everyone can or wants to use a mouse... Layers of support software => ease of implementation, maintainability – toolkits – interface builders – UIMS

14. Limitations of WIMP GUI Imposes sequential “ping-pong” dialog model: mouse and keyboard input, 2D graphics (sound?) output – deterministic and discrete – difficult to handle simultaneous input, even two mice – pure WIMP doesn’t use other senses: hearing, touch,... – >50% of our neurons in visual cortex, but as humans it is very difficult for us to communicate without speech, sound... Not usable for immersive VR (e.g., headmounted display or cave) where you are “in” the scene: no keyboard, mouse…

15. Impedance-matching Limitations of WIMP GUI Limited Vision (Flat, 2D) No Speech No Gestures One Hand Tied Behind Back Limited Audio Limited Tactile

16. Brown’s 2D NotePads: Simple Post-WIMP UIs via Gestures Both based on today’s tablet technology; visual plus tactile feedback NotePad – for note-taking, annotation – extend WIMP GUI with appropriate marking menus and gestures – designed for both stand-alone use and integration into a larger environment such as a Cave Musical Notepad – for musical notation and simple composition – purely gestural – electronic music paper integrated with full computer editing and MIDI facilities

17. Environment of Tomorrow: Push/Pull Driven by Moore’s Law, clever device engineering (e.g., micro- and nano-technology) we’ll have a profusion of form factors: – wearable computers, information appliances, e-books (based on digital paper) – smart rooms, furniture, clothing, jewelry … (Gershenfeld’s “When Things Start to Think”) – office/home (inexpensive) Virtual Reality – micro- and even nano-devices sensors & actuators/“displays” Ubiquitous/pervasive computing with many devices per user – personal computing won’t depend on “the personal computer” – users will be mobile, location- and device-independent, to concentrate on task Next century: bionic humans, neural implants; human- machine fusion? (Kurzweil’s “Age of Spiritual Machines”)

18. Rear-Projected ActiveDesk

19. UNC Vision of Augmented Reality

20. Augmented Reality Prototype

21. MIT’s Wearable Computing Group (1998)

22. Augmented Reality: LCD Projection (1998) Thad Starner, MIT Media Lab, Wearable Computing group Micro Optical Corporation, 320x240x8 LCD Display High- resolution Virtual Retinal Display (laser-based) at Dr. Furness’ HITLab @ U. of Washington

23. Post-WIMP User Interface Characteristics (1/2) Multiple simultaneous devices, sensory channels, or humans – 2-handed, with gesture recognition (e.g., Brown’s Sketching demos) – multi-modal UI: multiple integrated and reinforcing modes ooutput: display + spatial sound oinput: gesture + speech recognition – perceptual UI: goal is to imitate multi-modal human- human dialog otransparent, passive sensing -- computer is aware of and models user state (location, position, and possibly even mood) so it can respond intelligently – multiple humans can collaborate, compete, or be adversarial…

24. Post-WIMP User Interface Characteristics (2/2) High bandwidth, continuous input – body part tracking (head, gaze, hand...), e.g., for immersive VR (which requires head-tracked, wide- field-of-view, low latency, stereo display) – gesture and speech recognition => probabilistic disambiguation oex: handwriting, gesture recognition in 2D and 3D/VR Autonomous objects in active world are common Examples: “Put that there” demo from the 80s at MIT Media Lab, many VR demos

25. UI spectrum – direct control (direct manipulation, by-example...) – indirect control ( smart X, agents, social interfaces) WIMP enhanced by – 3D GUI “widgets” (e.g., 3D translate/rotate/scale boxes with handles) – speech & gesture recognition (tablets, wands, gloves, body and scene extraction via video-based computer vision technology…) – haptics (e.g., force feedback joysticks, augmented mice) – smart X, agents/wizards WIMP GUIs Will Be Augmented, Not Replaced

26. From HCI to HHI (Human-Human Interaction) Note: each human typically controls many devices and user interfaces

27. Multiple, Interconnected Devices and UIs per User (1/3) Office application – wall displays + personal notepad that provides private space in addition to any shared space(s) – user is video-tracked for identification, location, gaze, gesture – user’s speech is decoded: continuous speech recognition + natural language understanding + intelligent information processing allows dialog with an intelligent assistant – furniture: chair is instrumented to help detect posture, adjust to the user’s preferred position – user may wear prostheses (e.g., hearing aid), health monitors (pulse and blood pressure to monitor stress)

28. Multiple, Interconnected Devices and UIs per User (2/3) Health-care application – scaled-down office computing environment for the home – prostheses (today, heart pacemakers, hearing aids, cochlear implants, voice boxes, artificial joints and organs…) – electro-chemical monitors, probes (increasingly less obtrusive) osmart toilet to monitor bodily wastes – avatars of ohealth-care providers ofamily and friends, support group

29. Multiple, Interconnected Devices and UIs per User (3/3) Telecollaboration for multi-disciplinary industrial design – immersive VR environment – emphasis on small team collaboration – scene composed of CAD objects and objects in the environment, including co-workers orequires combination of geometry-based rendering (polygons, surfaces) and image-based rendering of geometry acquired via scene extraction based on computer vision techniques othis is a major paradigm shift in computer graphics – huge distributed systems problems, e.g., minimizing latency and its effects (cybersickness!)

30. The Cave for Immersive VR 8’ cubicle with projection screens as walls and floor

31. A Vision of Telecollaboration: A Normal Office Henry Fuchs, chief architect of this vision, in his office at UNC, Chapel Hill

32. A Vision of Telecollaboration: Overlapped Projected Displays Synchronized cameras, and projectors use imperceptible structured light to calibrate the room and its contents (depth and surface reflection properties of each pixel)

33. A Vision of Telecollaboration: Seeing and Manipulating Objects Designing a new head- mounted display (electro- mechanical- optical team design)

34. Shared CAD & Visualization Modeling Image Generation Scene Acquisition Interaction Networking Tracking Rendering Displays A Vision of Telecollaboration: “Being There” Together

35. Near-term Post-WIMP Office VR with Co- workers

36. Designing for the Future (1/2) Unlike with WIMP GUI for desktop applications, “one size fits all” doesn’t work: need many specialized UI’s and styles Design space is vastly larger and more complex, and needs a multidisciplinary aproach to create a design discipline – Science: how do humans see/hear/…/understand/remember? opsychophysics, perceptual psychology, cognitive science, social science, engineering and computer science (how to sense, store, manipulate, integrate and present efficiently) – Applied arts: Communication (newspapers, magazines, advertising…), Storytelling (art, theater, film and video, opera…), Design (graphic, industrial, architectural…), Education

37. Designing for the Future (2/2) 3D virtual worlds are far more complex than 2D “worlds,” and they are not the same as the real world. For example, – physical space and virtual space needn’t have the same metrics – rules for behavior can be simpler or more complex than real-world – social interfaces are not the same as people – cybersickness in VR (latency-induced) isn’t the same as motion sickness in the real world We should avoid the trap of designing “horseless carriages” – these are new media, neither the real world nor the 2D desktop – they will have their own idioms and metaphors, design discipline, and development environments

38. Some Research Challenges for Post-WIMP Interfaces (1/3) Much better device technology, greatly improving – accuracy, comfort, safety (avoid cybersickness) – both input (e.g., pointers, trackers) and output (e.g., visual and aural displays, haptics for force and tactile feedback) – non-obtrusiveness, spatial and temporal resolution, robustness (for handling, mutual interference, etc.) – cost effectiveness Metaphors for navigation, selection, manipulation, feedback – some based on the real world, others not (like teleportation) – animation for feedback

39. Some Research Challenges for Post-WIMP Interfaces (2/3) Augmentation of display with other sensory channels (e.g., haptics, sound); “unification” algorithms for mutual disambiguation and reinforcement, e.g., for gesture + voice Scaling of interaction techniques Platform independence (desktop vs. VR; small- team vs. large-scale simulations vs. online communities) – how can we factor out commonality? Integration of direct control with indirect control (e.g., through agents)

40. Some Research Challenges for Post-WIMP Interfaces (3/3) Powerful integrated development environment – visual programming and debugging tools?!? Usability studies, especially for longterm effects (negative training, safety...) Design discipline for post-WIMP UI design – interdisciplinary – based on verifiable theories, metrics – produce design handbooks with design patterns (rules) and exemplars of good design

41. Where are We Today?

42. Resources Basic UI textbooks – oriented towards traditional WIMP world but useful foundation material – Laurel, B. “The Art of Human-Computer Interface Design”, 1990 – Newman, W. M. and Lamming, M. G. "Interactive System Design", 1995 – Preece, Jenny. "Human-Computer Interaction", 1994 – Shneiderman, B. "Designing the User Interface", 1997 EC/NSF Research Frontiers in Virtual Environments and Human-Centered Computing Workshop (June 1999) report – to be published in ACM Computer Graphics SIGGRAPH issue, and IEEE Computer Graphics and Applications Resources Website: – http://www.cs.brown.edu/research/graphics/UIResources/ – further information about EC/NSF Workshop report publication will be posted here

43. Acknowledgements Bill Buxton (SGI Alias|Wavefront) Tom Furness (University of Washington Human Interface Technology Laboratory) Ben Shneiderman (University of Maryland) Rosemary M. Simpson (Brown University) Mathew Turk (Microsoft Research) Turner Whitted (Microsoft Research)

44. THE END