1. Evaluation of a multimodal Virtual Personal Assistant Glória Branco
20th International Symposium 2006
Evaluation of a multimodal Virtual Personal Assistant Glória Branco
Sophie-Antipolis, March 23, 2006

2. FASiL Project
FASiL – “Flexible and Adaptive Multi-Modal Spoken Interface Language”
EU-IST funded, multimodal, multi-lingual, conversational application to management.
Objectives
“...to pilot a full multi-modal voice portal application that is 3G mobile network ready, along with tools for rapid development of new applications. FASiL targets the languages of UK English, Portuguese and Swedish… [with] intelligent, friendly adaptive multi-modal interaction.”

3. FASiL Consortium Inovação Inovação P T
FASiL: “Flexible and Adaptive Multi-Modal Spoken Interface Language”
Inovação P T
Inovação
generation

4. VPA Architecture

5. VPA Funcionalities Hear a summary of the Inbox.
Navigation: next, previous.
Select specific s : search by State (new, old), Sender, Date, Priority and Category.
Read, compose, reply, forward and delete s.
Recipient list management.
Summarisation.
Categorisation.

6. VPA Interface Output Input VUI Multimodal GUI Voice Screen PDA
Keyboard
Mouse
Touch
Stylus
Multimodal
VUI
GUI
Available in English, Swedish and Portuguese

7. Global Evaluation
“to iteratively gather information about the usability and accessibility of the system”
Set up of test environment
Task design, to cover the VPA functionalities.
Test mailbox populated with a restricted set of contacts and s.
Heuristic Evaluation
5 expert assessments by each language.
Experts in accessibility, usability and voice interaction.
User Tests
20 users for accessibility only for the English version (RNIB and RNID)
20 Swedish and English users and 12 Portuguese.
Experts in usage.

8. The Portuguese Trial Laboratory environment
the graphic interface was a web-based page, simulating a mobile phone. The users used a desktop PC with Internet access to interact with the GUI and a fixed phone to convey voice to the system.
12 native Portuguese speakers
8 males and 4 females
from 19 to 46 years (mean 30,6 years)
75% of the participants had high-level education and 16,7 % had mid-level education
ICT domain professionals and experienced users.
5 typical tasks
login and browsing mailbox
search for and reply to an
search and forward
administer and manage the recipient list
finding, reply and deleting an

9. Task results summary Interactions Spoken interaction (%) Task Time
Comp. %
VUI
GUI.
Correct resp.
No resp.
Misund.
Incor. resp T1 10
(7-18)
83,3
14,0
(9-49)
6,5
(0-22)
54,3
13,7
6,8
25,2 T2 7
(5-12) 75
11,5
(5-33)
4,5
(0-16)
60,2
11,4
8,5
19,9 T3 5
(3-16)
100
14,5
(1-26)
1,5
(0-20) 67
9,8
13,3 T4 6
(2-10) 50
16;5
(4-41) 2
(0-6)
55,84
7,4
18,6
18,2 T5
(4-18) 32
(9-66) 4
(0-11)
59,4
9,4
21,8

10. Post-test satisfaction questionnaire
Frequecie

11. Statistical analysis
Significant correlation (Spearman’s correlation coefficient) between the overall satisfaction and:
Quality of dialog:  = 0,87
Confidence:  = 0,79
Easy of use:  = 0,74
Interaction control:  = 0,73
Interaction quality (error recognition):  = 0,69
Significant correlation (Spearman’s correlation coefficient) between the overall satisfaction (subjective) and the concept accuracy (objective value of correct responses):  = 0,85.
No differences between females and males (Mann-Whitney test) as well as between the experimented or naïve users.

12. Users aproach The preferred modality was speech.
Natural language, using short phrases but with complex commands.
Speech input to convey the commands and graphical interface to read the messages and to scroll quickly through the contacts list.
More intensive use of the GUI to overcome the recognition problems and slowness of the system response.
Mixed initiative dialog

13. Interaction Example 1
U I want replace [recipient name] by carbon copy.
S Who would you like to send copy to?
U (barge-in) [recipient name]
S Send copy to [recipient name]
U I want change the recipient list.

14. Interaction Example 2 U mailbox S You have 4 e-mails
U New search. Find high priority messages from [recipient name]
S You have 1 new priority from [recipient name]
U Read it

15. Users apreciation
The conversational and multimodal VPA concept was attractive to all users and was seen as a key enabler supporting the growing user mobile attitude.
The VPA was seen as easy to use and intuitive. The Help part of the system was almost not used.
Users did not liked excessive confirmations.
The use of the TTS Portuguese voice was well accepted by the users.
Users liked voice-in and VUI and GUI-out in a small-screen environment.
The multimodality was seen as a very good capability to overcome recognition problems encountered in the VUI.

16. Future Use But, when asked about the future use
58% of the users said that they would not use the system in its current form.
Main reasons:
slow response time
recognition/understanding problems.

17. Failure?
Tell me “when it’s time” to stop!

18. NO!
Lessons learned
Multimodal interfaces can overcome the weaknesses of each modality and exploit the full strengths of combined modes.
Speed of feedback is very important. Users dislike latency or long periods of silence.
Improvements are needed to increase the recognition accuracy of the spoken components.
Natural language is working.

19. Thank you! Contactos: Glória Branco: gloria@ptinovacao.pt