Queen Mary, University of London Automatic generation of multimodal interaction models from behavioural data Marie-Luce Bourguet Computer Science Dept. Queen Mary, University of London

Multimodal Interaction Models Application Multimodal Engine … Speech recognition Gesture recognition

Modeling Multimodal Interaction “remove” pen down pen move pen up gesture delete speech remove reset

Modeling Multimodal Interaction: Synchronisation patterns pen down pen move pen up gesture delete speech remove reset “remove” pen down pen move pen up gesture delete speech remove reset pen down pen move pen up gesture delete speech remove reset

Modeling Multimodal Interaction: Mutual disambiguation pen down pen move pen up gesture delete speech remove reset pen down pen move pen up speech move reset

Design Tool

Simulation Tool

Automatic Generation of Interaction Models Training Module … Speech recognition Gesture recognition

Conclusion Building interaction models from user interaction data presents numerous advantages: It bypasses the difficult task of designing multimodal interaction models The resulting interaction models are highly personalised They should enable natural and robust interaction The FSM based formalism is well adapted because very simple analysis of the data should permit the automatic generation of interaction models.