1. Multimodal FooBilliard
adding two multimodal user interfaces to an existing 3d billiard game
Dominic Sina, Paul Frischknecht, Marian Briceag, Ulzhan Kakenova
Spring 2015 Final presentation
Paul

2. Outline Introduction The Game The Interfaces CASE CARE Model
Research Questions
Users Evaluation
Results
Demo
Paul

3. Introduction: Objectives
add two multimodal user-interfaces to an existing game
pose and attempt to answer research questions related to these UIs
by running a user-evaluation
Paul

4. Introduction: The Game
Open source, C & OpenGL, FooBilliard
the Windows Platform’s Speech Recognition API in a C# component and
the Windows Touch APIs (+custom swipe detection) in a C++ component
Dominic

5. Introduction: The Interfaces
Keyboard, Mouse, Voice Commands
Interface 2:
Touch, Voice Commands
Dominic

6. CASE Model
Concurrent - Using independent modalities in parallel for distinct tasks. The user can at the same time use voice to increase/decrease the power of the strike, and use touch gestures to change to zoom in/out; the same could be done by using the keyboard and mouse.
Alternate - Using combined modalities in sequential order for one task. You can position the ball by dragging it and then say “Put Here” to confirm your placement.
Synergistic - Using combined modalities in parallel for the same task. The ball will be placed at the fingers’ position when you hold your finger on the screen and say “Put Here”.
Exclusive - Using independent modalities in sequential way for distinct tasks. Each shot can be considered a distinct task. You can always choose whether to use voice or keyboard/touch command to shoot.
Concurrent - Using independent modalities in parallel for distinct tasks. The user can at the same time use voice to increase/decrease the power of the strike, and use touch gestures to change to zoom in/out; the same could be done by using the keyboard and mouse.
Alternate - Using combined modalities in sequential order for one task. You can position the ball by dragging it and then say “Put Here” to confirm your placement.
Synergistic - Using combined modalities in parallel for the same task. The ball will be placed at the fingers’ position when you hold your finger on the screen and say “Put Here”.
Exclusive - Using independent modalities in sequential way for distinct tasks. Each shot can be considered a distinct task. You can always choose whether to use voice or keyboard/touch command to shoot.

7. CARE Model
Complementarity - Multiple modalities are to be used within a temporal window to reach a given state. User touches the screen and holds the finger at a position and uses the voice command “Put Here”. The ball moves to the position where finger is pointing.
Assignment - Only one modality can be used to reach a given state. The list of all voice commands is only accessible through the voice commands “commands/help/what can I say”
Redundancy - Multiple modalities have the same expressive power and they are all used within the same temporal window - User can shoot by using voice and keyboard (only one shot is performed, the rest of the input is ignored).
Equivalence - Necessary and sufficient to use any one of the available modalities. User can shoot the ball using gestures or voice or keyboard.
Complementarity - Multiple modalities are to be used within a temporal window to reach a given state. User touches the screen and holds the finger at a position and uses the voice command “Put Here”. The ball moves to the position where finger is pointing.
Assignment - Only one modality can be used to reach a given state. The list of all voice commands is only accessible through the voice commands “commands/help/what can I say”.
Redundancy - Multiple modalities have the same expressive power and they are all used within the same temporal window - User can shoot by using voice and keyboard (only one shot is performed, the rest of the input is ignored).
Equivalence - Necessary and sufficient to use any one of the available modalities. User can shoot the ball using gestures or voice or keyboard.

8. The Research Question
1. Which interface do users learn better without instructions?
2. The usage of which interface gives better results in playing the game?
3. Which interface is more attractive/satisfying for users?

9. Evaluation with users Objectives: Evaluations: Quantitative:
to observe valuable problems
to test theoretical questions
Evaluations:
Quantitative:
○ After how much time was a certain command from a set of commands used for the first time and how often was this command used?*
○ How much score did the user make (how many balls had disappeared from the table)?
○ Satisfaction score
Qualitative:
○ Comments and answers from the user to the question “What would you improve?”
○ Videos of the users

10. Evaluation: quantitative
Interface 1: Keyboard, Mouse, Voice Commands
Interface 2: Touch, Voice Commands
Score (the number of balls)
Satisfaction
(from 1 to 10)
User 1 8
User 2 6
User 3 7
User 4 3 5
User 5
User 6* 12 10
Score (the number of balls)
Satisfaction
(from 1 to 10)
User 1 8
User 2 5 4
User 3 2 7
User 4 9
User 5 11
User 6* 12 10

11. Evaluation: discovered commands
Interface 1: Keyboard, Mouse, Voice Commands
Command
User 1
User 2
User 3
User 4
User 5
Shoot (Voice)
40s, 4x
13s, 8x
25s, 1x
80s, 4x
90s, 1x
Shoot (Keyboard)
80s, 3x
34s, 7x
210s,
32s, 5x
20s, 56x
Bird view (Voice)
80s, 10x
59s, 2x
220s, 1x
Bird view (Keyboard)
Stronger/Weaker (Mouse)
65s, 3x
6s, 4x
Commands (Voice)
60s,1x
55s,
26s, 1x
70s, 1x
Stronger/Weaker (Voice)
120s, 4x
20s, 2x
Cue (Keyboard)
Cue (Voice)
70s, 4x
180s, 2x
75s, 1x
Help - F1 (Keyboard)

12. Evaluation: discovered commands
Interface 2: Touch, Voice Commands
Shoot (Voice)
20s, 4x
13s, 6x
2s, 8x
Shoot (Gesture)
50s, 3x
15s, 14x
70s, 10x
4s, 35x
Zoom (Gesture)
50s, 1x
120s, 2x
Camera Move (Gesture)
15s, 16x
40s, 39x
Place the Ball (Gesture)
20s,4x
1s, 3x
1s, 5x
5s, 5x
Bird view (Voice)
80s,10x
180s, 2x
Commands (Voice)
140s, 1x
1s, 1x
Stronger/Weaker (Voice)
34s, 5x
130s, 2x
54s, 14x
Cue (Voice)
55s, 1x
30s, 2x
33s, 1x
Place the Ball (Voice)
28s, 2x

13. Results Which interface do people learn better without instructions?
Interface 2 (Touch, voice commands)
The usage of which interface gives better results in playing the game?
Which interface is more attractive/satisfying for users?
Interface 1 (touch, keyboard, mouse)
Interface 2 (touch, voice commands)
Overall Score (number of balls killed) 29 34
Average Satisfaction
6.8
7.5

14. Video
Dominic