1. Multimodal Interfaces in a Ubiquitous Computing Environment 3 rd UK-Ubinet Workshop —————— 9 th – 11 th February 2005 —————— Fausto. J. Sainz Salces, Dr. David Llewellyn-Jones, Prof. Madjid Merabti —————— School of Computing and Mathematical Statistics Liverpool John Moores University James Parsons Building Byrom Street Liverpool, L3 3AF, UK {cmsfsain, D.Llewellyn-Jones, M.Merabti}@livjm.ac.uk http://www.cms.livjm.ac.uk/PUCsec/

2. Disappearing Computers – Disappearing Interfaces The Ubiquitous Computing paradigm presents difficult user interface problems –Enforces a minimal interface –Requires increased usability Current user interfaces tend to concentrate on visual display techniques Need to explore alternative methods for transferring information and allowing control

3. Harnessing the Senses We felt the most appropriate sense for information presentation after vision was sound –Can alert the user without requiring their specific attention –Ambient interface ideal in a pervasive environment Considerable work exists in the field of audio interfaces –Earcons: tones or sequences of tones as a basis for building messages [1,2] –Auditory icons: everyday sounds that convey information about events by analogy with everyday sound producing events [3] –Voice: instructions are read either as a recorded or simulated voice Our work looks at the use of earcons in a Ubiquitous Computing Environment [1] M. Blattner, Sumikawa, D. & Greenberg, R., "Earcons and icons: Their structure and common design principles," Human Computer Interaction, vol. 4(1), pp. 11-44, 1989. [2] Brewster, S., Wright, P.C., Edwards A.D.N. (1994). “A detailed investigation into the effectiveness of earcons,” in Auditory Display. Sonification, audification and auditory display., vol. XVIII, G. Kramer, Ed. Santa Fe, U.S.A.: Addison-Wesley, 1994, pp. 471-498. [3] W. W. Gaver, "Using and creating auditory icons.," in Auditory Display. Sonification, audification and auditory display., vol. XVIII, G. Kramer, Ed. Santa Fe, U.S.A.: Addison-Wesley, 1994, pp. 417-446.

4. Multimodality The use of several means to present information simultaneously Combine the use of visual icons and earcons –Makes sure the message arrives at the receiver –Redundant info –Multiple channels (also increases accessibility) –More natural presentation method Undertook experiments to compare devices used to control household appliances –Small handheld device with a restricted form factor –Large screen ‘fixed’ laptop device Both devices were used with various combinations of visual, audio and multimodal interfaces

5. Interface Design We compared people’s perceptions and ability to use each device using 6 permutations of the interface –Visual– Four device control –Earcons– Six device control –Multimodal

6. Design Process An initial design stage was used to establish effective designs for each of the interfaces modalities –Human constraints –Machine constraints –Cognitive knowledge –User preference An iterative design method was used

7. Earcon Design Each earcon is comprised of two parts –The first motif represents the household appliance –The second motif represents the new state of the device (e.g. “on” or “off”) Motif 1 Motif 2 Hob off Hob on

8. Experiments Experiments were undertaken individually with two groups of users –20–33 year olds –60–84 year olds The intention was to observe a broad range of users, both relatively proficient users and computer novices

9. Results: Comparing Large & Small Overall, users made favourable comments about the smaller handheld devices The most common remark concerned the positive usability aspect of the palmtop “Small portable, clear, handy able to carry in one hand” However, people cited the small screen as being a problem The larger screen was seen as a positive aspect of the laptop device, but this was countered by its fixed nature Interestingly, the only comment concerning security, was in favour of the palmtop

10. Results: Device Ranking The comments are echoed by the preference ranking that subjects gave for the devices Screen Visual Palmtop Visual Screen Audio Palmtop Audio Palmtop Multimodal 0 10 20 30 40 50 60 Percent Ranked Lowest Screen Visual Palmtop Visual Screen Audio Screen Multimodal Palmtop Multimodal 0 10 20 30 40 50 60 Percent Ranked Highest

11. Results: NASA TLX Mental Demand measurement “How much mental demand and perceptual activity was required (e.g. thinking, deciding, calculating, remembering, looking, searching, etc.)? Was the task easy or demanding, simple or complex, exacting or forgiving?” AudioVisualMultimodal Mean 48.7323.0424.07 Mode 305 & 105 Std. Deviation 25.22922.14023.576 5101520253035404550556065707580859095 0 2 4 6 8 10 12 14 Frequency Audio 100 0510520253035404550556065707580859095 0 5 10 15 20 25 Frequency Visual 100 51015202530354045506070758085 0 5 10 15 20 25 30 Frequency Multimodal 100

12. Results: NASA TLX Performance measurements “How successful do you think you were in accomplishing the goals of the task set by the experimenter? How satisfied were you with your performance in accomplishing these goals?” AudioVisualMultimodal Mean 40.0021.9120.59 Mode 555 Std. Deviation 29.59323.96424.468 0 5 1015 20 25 3035 4045 50 556065 707580 8590 95 100 0 2 4 6 8 10 12 14 Frequency Audio 100 0510152025303540455055606570809095 0 10 20 30 40 Frequency Visual 100 0510152025303540455055607080859095 0 10 20 30 Frequency Multimodal 100

13. Results: NASA TLX Frustration measurement “How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?” AudioVisualMultimodal Mean 37.0622.2518.28 Mode 555 Std. Deviation 28.77322.76318.912 05101520253035404550556065707580859095 0 5 10 15 20 Frequency Audio 100 051015202530354045506070808590 0 10 20 30 40 Frequency Visual 100 0510152025000354045500065707580 0 10 20 30 40 Frequency Multimodal 100

14. Conclusions Users generally found the handheld device to be better than the larger screen laptop Many of the benefits were described in terms that might apply in a Ubiquitous Computing environment Multimodal designs were preferred –Earcons can be used to augment a limited user interface in a beneficial way –Earcons alone performed significantly worse than visual icons alone These results applied across both groups of subjects Further results concerning reaction times are in the process of being analysed

15. Multimodal Interfaces in a Ubiquitous Computing Environment 3 rd UK-Ubinet Workshop —————— 9 th – 11 th February 2005 —————— Fausto. J. Sainz Salces, Dr. David Llewellyn-Jones, Prof. Madjid Merabti —————— School of Computing and Mathematical Statistics Liverpool John Moores University James Parsons Building Byrom Street Liverpool, L3 3AF, UK {cmsfsain, D.Llewellyn-Jones, M.Merabti}@livjm.ac.uk http://www.cms.livjm.ac.uk/PUCsec/