1. NESPOLE! is a project which aims at providing a system capable of supporting communication in the field of e-commerce and e-service by resorting to automatic speech-to-speech translation and multimodal interaction. Multimodality in NESPOLE! is accomplished by the integration of speech and pen-based gestures (drawings performed on a shared Whiteboard). The NESPOLE! Multimodal Interface for Cross-lingual Communication - Experience and Lessons Learned Monolingual Data Collection (February 2002) Loredana Taddei Erica Costantini Alon Lavie l.taddei@aethra.it costanti@psico.units.it alavie@cs.cmu.edu NESPOLE! Project (http://nespole.itc.it) The first NESPOLE! Showcase was developed for the following tourist scenario: The customer aims at organizing a trip in Trentino. She starts by browsing APT web pages to get information. When the customer wants to know more about a particular topic or prefers to have a more direct contact, the speech-to-speech translation service allows her to interact in her own language with an APT agent. A videoconferencing session can be opened clicking a button and the dialog starts. Showcase Scenario The System NESPOLE! system allows users speaking different languages to interact in a natural way by means of: hearing their original voice and their translated speech; seeing the face and expression of their partner; sharing visual information (maps and web pages), even through pen-based gestures; monitoring the translation process and correcting it if and when something goes wrong. First Version of The Interface Throughout the first two years of the NESPOLE! project we took advantage of many opportunities to collect data concerning the system and the interface usability with actual users: most of the interface improvements leading to the current version were based on the comments and suggestions of these users and on our observation of their behavior. NESPOLE! Monitor Window “Remote speech translation”: the text of the last translated utterance of the remote speaker. “Recognized hypothesis string”: the text of the last recognized utterance of the local user. “String converted from IF”: textual representation resulting from the translation of the last utterance spoken by the local user back into their own language. ”Cancel Translation” button: generates a red flashing message on the monitor of the other party indicating that the incoming translated message should be ignored. Hallo I would like to take a trip to Trentino Hallo I would like to visit Trentino AeWhiteboard Window image loading; drawing functions (possibility to choose colors and shapes); rectangular or elliptical area selecting; URL opening. The user can share all these operations with the remote user as they were sitting around a table exchanging brochures and illustrative material. User Group Meeting (September 2001) Experimental on Multimodality (Summer 2001) Involved 77 people (65 volunteers as customers and 12 from APT tourist office agents). Objective: to collect impressions and suggestions about the system through a questionnaire of 16 statements. Result: 91% of the volunteers reported they would like to use the NESPOLE! system instead of telephone or e-mail to ask for tourist information and 100% of the tourist agents reported they would like to use the system at work. Involved 8 European and US companies. A questionnaire was distributed: it consisted in 14 statements which the subject was to give his/her attitude towards by using a five-grade scale plus four open-ended questions. Result: many comments and useful suggestions that had been implemented in the successive user interface. Involved 35 novice users speaking Italian, German or English. Objective: to compare the multi-modal version of the NESPOLE! system with a speech only version in which multi-modal resources (pen-based gestures) were not available. Result: multi-modality helped users to overcome ambiguities and system recognition and translation errors. User Studies Current Interface Hallo I would like to take a trip to Trentino Hallo I would like to visit Trentino IT S.U.: APT informazioni buongiorno EN R.S.T:APT information,Hallo EN S.H.: Hallo I would like to take a trip to Trentino EN S.U.: Hallo I would like to visit Trentino IT R.S.T.: Buongiorno vorrei fare un viaggio in Trentino NETMEETING WINDOW: added live video transmission. Progress bars and icons: indicating system status and audio flow. “Recognized hypothesis string” changed in “System hears” and made editable to allow the user to manually correct recognizer’s errors. NESPOLE MONITOR WINDOW: Audio enable/disable button. DIALOGUE HISTORY WINDOW: it contains all feedback text strings ordered on a temporal base. Quick map saving:the user can save the map with a progressive number by simply clicking a button. WHITEBOARD WINDOW: Image scrolling and zoom