1. Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer Lisa Anthony, Jie Yang, Kenneth R. Koedinger Human-Computer Interaction Institute Carnegie Mellon University, Pittsburgh, PA

2. Human-Computer Interaction Institute Anthony, Yang, Koedinger Programming constructs or special syntax Linearization of mental representation Difficult to revise structure in template-based editors Computer-Based Math Tools MATHEMATICA MICROSOFT EQUATION EDITOR Maple V, Mathematica, Matlab; Microsoft Equation Editor, …

3. Human-Computer Interaction Institute Anthony, Yang, Koedinger Problems: ► Large learning curve ► Tied to keyboard and mouse input Computer-Based Math Tools MATHEMATICA MICROSOFT EQUATION EDITOR Maple V, Mathematica, Matlab; Microsoft Equation Editor, …

4. Human-Computer Interaction Institute Anthony, Yang, Koedinger Notations already exist for paper-based math Affordance for spatial representation Especially good for students learning math Problem: recognition accuracy Pen-Based Computing

5. Human-Computer Interaction Institute Anthony, Yang, Koedinger Solution: Multimodal Input Increased robustness ► Better recognition accuracy with multiple input streams (Oviatt, 1999; pen gestures + speech) Consider both repair-only and simultaneous input in both streams

6. Human-Computer Interaction Institute Anthony, Yang, Koedinger Motivation for User Study Literature says typing is faster (Brown, 1988) ► Compared inputting paragraphs of English text ► Math is different domain Little evaluation done of pen-based equation input ► Systems constrained by recognition accuracy (Smithies et al, 2001)

7. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Design Input method Equation complexity ► Number of characters in equation ► Number of “complex” symbols (e.g., √ and ∑) Keyboard-and-mouseUsed keyboard and mouse with Microsoft Equation Editor (MSEE). Handwriting-onlyWrote in Windows Journal program; without recognition. Speech-onlyDictated into microphone; without recognition or visual feedback. Handwriting-plus- speech Both spoke and wrote in series or in parallel (user choice); without recognition. “Recognition” means system tries to interpret user input.

8. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Participants 48 paid participants (27 male, 21 female) Undergraduate/graduate, full-time/part-time students at Carnegie Mellon Native English speakers only Most (33 of 48) had no experience with MSEE before study

9. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Procedure Entered math equations on TabletPC ► 36 equations (7 + 2 practice per condition) ► Conditions counterbalanced across participants Instructions for each condition ► No prompting for specific ways of expressing equations ► 5 min “explore time” for MSEE

10. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Measures Time per equation Number of errors per equation (corrected and uncorrected) User preferences before and after session Equation entry screen in handwriting condition.

11. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Sample Stimuli

12. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Sample Stimuli ConditionStimuli VersionUser Input Keyboard-and- mouse Handwriting-only Speech-only “y minus four, over quantity, y squared minus-minus five y, plus four, equals nine” Handwriting-plus- speech (in parallel) “sum, of, c, subscript k squared, minus two, c, subscript k, minus ten, close parentheses”

13. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Sample Stimuli ConditionStimuli VersionUser Input Keyboard-and- mouse Handwriting-only Speech-only “y minus four, over quantity, y squared minus-minus five y, plus four, equals nine” Handwriting-plus- speech (in parallel) “sum, of, c, subscript k squared, minus two, c, subscript k, minus ten, close parentheses”

14. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Sample Stimuli ConditionStimuli VersionUser Input Keyboard-and- mouse Handwriting-only Speech-only “y minus four, over quantity, y squared minus-minus five y, plus four, equals nine” Handwriting-plus- speech (in parallel) “sum, of, c, subscript k squared, minus two, c, subscript k, minus ten, close parentheses”

15. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Sample Stimuli ConditionStimuli VersionUser Input Keyboard-and- mouse Handwriting-only Speech-only “y minus four, over quantity, y squared minus-minus five y, plus four, equals nine” Handwriting-plus- speech (in parallel) “sum, of, c, subscript k squared, minus two, c, subscript k, minus ten, close parentheses”

16. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Results Average time in seconds per equation by condition. Error bars show 95% confidence interval (CI).

17. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Results Mean number of user errors made per equation by condition. Error bars show 95% CI.

18. Human-Computer Interaction Institute Anthony, Yang, Koedinger User Study: Results Preference questionnaire rankings of each condition on a 5-point Likert scale. Error bars show 95% CI. Pre-testPost-test

19. Human-Computer Interaction Institute Anthony, Yang, Koedinger Conclusions Handwriting faster, more efficient, and more enjoyable to novice users than standard keyboard-and-mouse Handwriting-plus-speech faster and better liked than keyboard-and-mouse Handwriting-plus-speech not much worse than handwriting alone, so multimodal may be a winner for technology reasons

20. Human-Computer Interaction Institute Anthony, Yang, Koedinger Further Analyses Transcription of spoken input as corpus for generation of language model Consistency across and within users in handwriting and speech ► Ambiguity resolution ► Self correction ► Pausing and synchronization in multimodal input Greater variability within speech condition than within handwriting condition

21. Human-Computer Interaction Institute Anthony, Yang, Koedinger Further Analyses Transcription of spoken input as corpus for generation of language model Consistency across and within users in handwriting and speech ► Ambiguity resolution ► Self correction ► Pausing and synchronization in multimodal input Greater variability within speech condition than within handwriting condition (Supported by Oviatt et al, 2005)

22. Human-Computer Interaction Institute Anthony, Yang, Koedinger Questions? Project Webpage: http://www.cs.cmu.edu/~lanthony/research/multimodal/ Pittsburgh Science of Learning Center: http://www.learnlab.org/index.php

23. Human-Computer Interaction Institute Anthony, Yang, Koedinger References Brown, C.M.L.: Comparison of Typing and Handwriting in “Two-Finger Typists.” Proceedings of the Human Factors Society (1988) 381– 385. Oviatt, S.: Mutual Disambiguation of Recognition Errors in a Multimodal Architecture. Proceedings of the CHI Conference (1999) 576–583. Smithies, S., Novins, K., and Arvo, J.: Equation Entry and Editing via Handwriting and Gesture Recognition. Behaviour and Information Technology 20 (2001) 53–67. Hausmann, R.G.M. and Chi, M.T.H.: Can a Computer Interface Support Self-explaining? Cognitive Technology 7 (2002) 4–14.

24. Human-Computer Interaction Institute Anthony, Yang, Koedinger Other References in Paper Anderson, J.R., Corbett, A.T., Koedinger, K.R., and Pelletier, R.: Cognitive Tutors: Lessons Learned. The Journal of the Learning Sciences 4 (1995) 167–207. Blostein, D. and Grbavec, A.: Recognition of Mathematical Notation. In Handbook on Optical Character Recognition and Document Analysis, Wang, P.S.P. and Bunke, H. (eds) (1996) 557–582. Locke, J.L. and Fehr, F.S.: Subvocalization of Heard or Seen Words Prior to Spoken or Written Recall. American Journal of Psychology 85 (1972) 63–68. Microsoft.: Microsoft Word User’s Guide Version 6.0 (1993), Microsoft Press. Sweller, J.: Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science 12 (1988) 257–285.