1 Practical ID: L6 Technology & interaction choices

2 This lecture will review some of the available interface technologies and styles of interaction available to the designer

3 Interaction style The principle means of interacting with computers and other interactive media include –Command line –Form-fill (or fill-in) –Direct manipulation, GUI –Voice –Tangible / touch input –Brain –By being there

4 CLI A command line interface uses lines of textual commands These are either typed at a prompt such as “C:\>” (MSDOS) or even “.” (DB II) Or delivered by a script “autoexec.bat”

off SET SOUND=C:\PROGRA~1\CREATIVE\CTSND SET BLASTER=A220 I5 D1 H5 P330 E620 T6 SET PATH=C:\WINDOWS;C:\ LH C:\WINDOWS\COMMAND\MSCDEX.EXE /D:123 LH C:\MOUSE\MOUSE.EXE DOSKEY CLS

6 Are they dead? No - witness Linux Still widely used close to hardware cf. networks, real time systems

7 On balance Skilled users report finding them faster than GUIs But –Need to know the available commands which relies on recall memory –It is easy to make syntax errors (entering instructions in the wrond order)

8 Some of the commands available within Unix

9 Form Fill(in) Structured input On-screen forms - lots of business applications Reborn with e-commerce / e-shopping

10 Direct manipulation Direct manipulation - often attributed to Shneiderman but actually invented by … Sutherland (1963) who developed SketchPad which supported the manipulation of objects using a lightpen Involves directly manipulating (e.g. double clicking) objects on the screen

11 The Xerox Star Xerox ‘Star’ workstation launched as the 8010 Star information system in April, 1981 Every GUI is based on the Star

12 Familiarity not metaphor Every user’s initial view of Star is the Desktop, which resembles the top of an office desk, together with surrounding furniture and equipment. On the screen are displayed pictures of familiar office objects, such as document folders, file drawers, in-baskets and out- baskets. These objects are displayed as small pictures or icons. The Desktop is the principal Star technique for realizing the physical office metaphor.

13 Next came the Lisa… The Apple Lisa was the first truly commercial release of the desktop metaphor GUI Suppose I’m writing a report for my boss and I want to prepare a chart to illustrate a certain point. With a few movements of the mouse I ‘tear off’ a sheet of Lisa Graph ‘paper’ and give it the heading ‘Annual Sales’… then I use the mouse to cut the graph … and put it in a temporary storage place called the ‘clipboard’. I can then ‘throw away’ the Lisa Graph ‘paper’. With the Lisa applications programs … you can switch your attention from a sheet of Lisa Write ‘paper’ to a sheet of Lisa Calc ‘paper’ and back with no problem, just as you could if they were two sheets of paper on your desk. - review by Williams (1983)

14 Mac OS 1 / 7/ 9

15 OS X (10.4)

16 Followed Microsoft with Windows 1 & 2

17 Windows 3, 95, 98

18 Windows ME / XP / Vista

19 Open source GUIs

20 Sugar

21 WIMPs - end of a paradigm? WIMP user interfaces and office applications very closely linked Nothing much as changed for 20+ years Isn’t there something faintly ridiculous having a desktop in your pocket (Windows for PDAs)?

22 GUIs are regarded as having gone as far as they can –“I have always been sceptical of UIs that try to mimic a physical environment in a close way (like the desktop). One reason is that the things that are the greatest about computers (like hyperlinks) do not exist in the real world, so why should we simulate real-world environments slavishly?” (Alan Kay, 1996) But still the vendors keep turning them out

23 What about other sensory modalities? A serious criticism of GUIs is that they are confined to the visual medium Where we have n senses (how senses do we have?)

24 Scent collar The small collar has been developed to be used in conjunction with virtual reality military training systems The collar (worn around the neck) releases smells to enhance the experience of (virtual) combat

25 “The place smells like damp earth and mouldy concrete, a bit like a cellar, reinforced by creepy visuals, and a rat scurrying by, the rustle of its feet chilling the spine. Crawling out of the strange, broken down ruins, you emerge in a forest, by a river, at night. The air is crisp and the aroma of pinewood is all about. After the dank, dark, ruined crawlspace, the change is so refreshing that it distracts you, and you fail to notice the sentry ahead, atop a building peeking over the tree line, standing at a machine gun and looking straight at you.”

26 Biometric measures GSR - galvanic skin response Blood pressure & blood volume Heart rate Are the easily measured unconscious physical characteristics They all indicate levels of arousal - that is, how interested, concerned, emotional about something

27 These can be used for medical purposes, e.g. the smart bra which can indicate the presence breast cancer And for more ludic purposes, e.g. the ear rings (next slide) indicate how aroused the wearer is …

28 Biometric wearables

29 Smart Eyewear The Inview goggles display a lap count and time elapsed on their lenses so swimmers can track their progress. Invented by industrial design student Katie Williams, the goggles use an in- built compass to spot when swimmers complete lengths.

30 A bionic arm Jesse Sullivan from Tennessee in the US is the world's first 'bionic man'. Scientists at the Rehabilitation Institute of Chicago (RIC) replaced his left arm.He was given a special "bionic" arm - a mechanical device directed by his own brain.

31 Voice VoiceXML is designed for creating audio dialogs that feature synthesized speech, digitized audio, recognition of spoken and DTMF (dual tone multi-frequency) key input, recording of spoken input, telephony, and mixed initiative conversations. Its major goal is to bring the advantages of Web-based development and content delivery to interactive voice response applications.

32 It is platform-independent and works like HTML applications. VoiceXML applications can used with traditional web applications, accessing the same data and performing the same essential tasks, even residing on the same machines. VoiceXML allows a programmer to write a basic voice application without having to know or learn anything about the voice hardware on which the application will run.

33 Physical interaction Distinguish between touch-based (like the iPhone) and tangible Touch screen have been around for years but there has been recent excitement about the Apple multi-touch UI and other “copy-cat” offerings from other mobile phone vendors And there is tangible interaction which is still largely in the research labs

34 Haptics –Proprioceptive - relating to sensory information about the state of the body (including cutaneous, kinaesthetic, and vestibular sensations). –Vestibular - pertaining to the perception of head position, acceleration, and deceleration. –Kinaesthetic -the feeling of motion. Relating to sensations originating in muscles, tendons and joints. –Cutaneous - pertaining to the skin itself or the skin as a sense organ. Includes sensation of pressure, temperature and pain.

35 –Tactile -pertaining to the cutaneous sense but more specifically the sensation or pressure rather than temperature or pain. –Force Feedback - relating to the mechanical production of information sensed by the human kinaesthetic system. –after Oakley, McGee, Brewster and Gray, (2000)

36 The iPhone was launched Jan with the touch iPod following Sept later In 2007 Nokia announced its touch-screen phone – code named Tube – with its vice- president Anssi Vanjoki admitting “If there’s something good in the world, we copy it with pride (Reported in the Independent , p. 9.)

37 All of the major phone vendors following suit … –Samsung’s Instinct and F700 –Omnia i900 –HTC Touch

38 Window 7 (the one after Vista) will be touch-based So what are they good for … Home, mobile (small screen), smart toys, home robots Desktop replacement - unlikely

39 Still evolving Twisting interface discussed in a paper entitled “I Sense a Disturbance in the Force” by Scott, Brown and Molloy, 2008 Whether developments like this make it out of the labs remain to be seen This is a very hot research area

40

41 And let’s not forget the Wii This is the Wii fitness pad

42 Watch this space for developments in gaming UIs (e.g. guitar hero) And the copies or clones from the other vendors Smart toys often have touch interfaces which are particularly suited to small children

43 And then there’s tangible UIs What’s this?

44 Tangible Bits at MIT People have developed sophisticated skills for sensing and manipulating our physical environments. … most of these skills are not employed by traditional GUI - Tangible Bits seeks to build upon these skills by giving physical form to digital information, seamlessly coupling the dual worlds of bits and atoms.

45 Guided by the Tangible Bits vision, we are designing TUIs which employ physical objects, surfaces, and spaces as tangible embodiments of digital information. These include foreground interactions with graspable objects and augmented surfaces, exploiting the human senses of touch and kinaesthesia.

46 Bricks The Bricks system is a good example of a graspable user interface. Bricks was developed by Fitzmaurice, Ishii and Buxton and reported in It was designed to allow the manipulations of digital objects by way of physical ‘bricks’. The bricks, which are approximately the size of LEGO™ bricks, are placed and operate on a large, horizontal computer display surface called the Active Desk.

47 The physical object (the brick) is mapped onto a digital object. Moving the former, moves the latter Rotating the former … But this requires a sensory-motor “desk”

48 This is later than Bricks but employs similar principles. A schematic representation of the operation of the actuated workbench (images have been extracted from Pangaro, Maynes-Aminzade and Ishii’s paper in UIST’02)

49 Brain control Direct brain-computer interfaces have been around for 5 years or so The early ones were intrusive (electrodes actually implanted in the brain) Current version have been based on fMRI

50 fMRI works by picking up tiny magnetic signals from oxygen bound to iron dissolved in the bloodstream. As brain activity increases in a particular area, the blood flow increases, bringing more oxygen with it and increasing the signal strength. The technique can measure specific areas of the brain with millimetric accuracy, and most subjects proved adept at switching thoughts 'on' and 'off' after just three 45-minute training sessions.

51 finally Interaction in the world Augmented reality, that is, overlying the real world with digital media, e.g. the next generation of sat nav This prototype head-up display (next slide) works like the system in military aircraft Spoken instructions & maps being replaced with a projected line to follow

52

53 AR is a major research area which a large number of very interesting developments

54 So, we have desktop based interaction (CLI, forms, GUI) And multimodal interaction - smell, voice and most importantly touch And on the horizon, AR and wearables

55 Some what questions to ask of your design Is the platform appropriate? Am I restricting the choice of input? What else?