1. May 14, 20086.870 Multimodal User Interface1 6.870 Final Project Webnnel: A channel-based Web navigation system Chen-Hsiang Yu and Oshani Seneviratne {chyu,oshani}@mit.edu

2. May 14, 20086.870 Multimodal User Interface2 Outline Introduction (Jones)Introduction (Jones) Motivations (Jones)Motivations (Jones) Related WorkRelated Work –Web automation and customization (Jones) –Speech recognition (Oshani) Our ApproachOur Approach –Web customization and automation (Jones) –Speech recognition (Oshani) –The integration of command extension with speech recognizer (Jones, Oshani) DemonstrationDemonstration Challenges (Oshani) & Future Work (Jones)Challenges (Oshani) & Future Work (Jones) Discussion (Jones)Discussion (Jones) ReferencesReferences

3. May 14, 20086.870 Multimodal User Interface3 Introduction I

4. May 14, 20086.870 Multimodal User Interface4 Motivations I

5. May 14, 20086.870 Multimodal User Interface5 Related Work Web Automation and CustomizationWeb Automation and Customization –Point 1 –Point 2 Speech RecognitionSpeech Recognition –Microsoft Vista Speech Recognition Engine –Apple Mac Speech Recognition Engine (But none of the above provide the level of customization offered by Webnnel!)

6. May 14, 20086.870 Multimodal User Interface6 Our Approach - Web Customization and Automation I

7. May 14, 20086.870 Multimodal User Interface7 Our Approach - Speech Recognition Used the Mac OS Speech Recognition EngineUsed the Mac OS Speech Recognition Engine Written in Objective C Highly flexible To add new commands you have to… –Allocate and initialize an instance of NSSpeechRecognizer. –Set the commands that the object should listen for using the setCommands: method. –Set a delegate for the NSSpeechRecognizer object that implements the speechRecognizer:didRecognizeCommand

8. May 14, 20086.870 Multimodal User Interface8 Our Approach - Integration Web Customization and Automation:Web Customization and Automation: Apple Scripts:Apple Scripts: –Acts as the “glue” between the speech recognition and the Webnnel Firefox Extension –Custom scripts for each speech command –Perform keystrokes at the Webnnel command prompt upon recognition

9. May 14, 20086.870 Multimodal User Interface9 Demonstration I

10. May 14, 20086.870 Multimodal User Interface10 User Study Conducted a qualitative study on 4 usersConducted a qualitative study on 4 users Asked the users to perform 2 tasks using the Webnnel speech recognition systemAsked the users to perform 2 tasks using the Webnnel speech recognition system –Task 1: Go to a certain website –Task 2: Go to their web-based email system

11. May 14, 20086.870 Multimodal User Interface11 User Study (cont) Recognition Accuracy (from the 16 commands we asked them to test the system with):Recognition Accuracy (from the 16 commands we asked them to test the system with):

12. May 14, 20086.870 Multimodal User Interface12 User Study (cont) General Comments from the users:General Comments from the users: –Commands are natural and easy to remember –Liked the tag system –Shorter the command it’s better –There should be ways to enter the URL directly in to the address bar as well

13. May 14, 20086.870 Multimodal User Interface13 Challenges Early experimentation on CMU-Sphinx4 Java based speech recognition failedEarly experimentation on CMU-Sphinx4 Java based speech recognition failed –Too many configuration parameters to consider –Our custom language model and grammar had a very poor recognition accuracy Achieving cross platform compatibility:Achieving cross platform compatibility: –Compared to the Mac OS, Windows (XP, Vista) and Linux (Ubuntu 7.10) did not have good support for speech recognition. –The quality of the microphones vary across different computers Introducing many speech commands generally lowers the accuracy of the entire systemIntroducing many speech commands generally lowers the accuracy of the entire system Having a stress ball around was very handy while testing the speech recognition :)Having a stress ball around was very handy while testing the speech recognition :)

14. May 14, 20086.870 Multimodal User Interface14 Future Work Porting the speech recognition aspect of the Webnnel system to other platforms.Porting the speech recognition aspect of the Webnnel system to other platforms.

15. May 14, 20086.870 Multimodal User Interface15 References 1.Avot mV, http://www.avotmedia.com/ 2.Bigham, J. P., and Ladner, R. E. Accessmonkey: a collaborative scripting framework for web users and developers. In W4A '07, ACM Press, pp. 25-34, 2007. 3.Bolin, M., Webber, M., Rha, P., Wilson, T. and Miller, R.C. Automation and customization of rendered web pages, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005. 4.Apple Speech Recognition Engine, http://developer.apple.com/documentation/Cocoa/Conceptual/Speech/Articles/RecognizeSpeech.h tml 5.CMU-Sphinx Speech Recognition Engine, http://cmusphinx.sourceforge.net/html/cmusphinx.php 6.Greasemonkey, https://addons.mozilla.org/en-US/firefox/addon/748 7.Joost, http://www.joost.com/ 8.Microsoft Windows Vista Speech Recognition system http://www.microsoft.com/enable/products/windowsvista/speech.aspx 9.Mogulus, http://www.mogulus.com/ 10.Petrie, H., Hamilton, F. and King, N. Tension, what tension? Website accessibility and visual design. Proceedings of the 2004 international cross-disciplinaryworkshop on Web accessibility (W4A), pp. 13-18, 2004.

16. May 14, 20086.870 Multimodal User Interface16 References (cont.) 10.Richards, J. and Hanson, V. Web accessibility: a broader view. Proceedings of the 13th international conference on World Wide Web, pp. 72-79, 2004.

17. May 14, 20086.870 Multimodal User Interface17 Any Questions? {chyu,oshani}@mit.edu