September 28, 2004 Programming By Voice Andrew Begel Advisor: Prof. Susan L. Graham University of California, Berkeley VL/HCC 2004

September 28, 2004VL/HCC 2004 Graduate Student Consortium Programming by Voice for int i equals zero i less than ten i plus plus 1

September 28, 2004VL/HCC 2004 Graduate Student Consortium Programming by Voice for (int i = 0; i < 10; i++ ) { ▌ } for int i equals zero i less than ten i plus plus 1

September 28, 2004VL/HCC 2004 Graduate Student Consortium Ambiguities for (int i = 0; i < 10; i++ ) { ▌ } 4 int eye equals 0 aye less then ten i plus plus KW or #? Spelling of ID? KW or ID? 2

September 28, 2004VL/HCC 2004 Graduate Student Consortium Sometimes it’s hard! for times ate equals file two load equals one 3

September 28, 2004VL/HCC 2004 Graduate Student Consortium for (times; ate == file; to().load = 1) { ▌ } for times ate equals file two load equals one 3 Many Valid Interpretations!

September 28, 2004VL/HCC 2004 Graduate Student Consortium 4 * 8 = file; toload = won ▌ for (times; ate == file; to().load = 1) { ▌ } for times ate equals file two load equals one 3 Many Valid Interpretations!

September 28, 2004VL/HCC 2004 Graduate Student Consortium Many Valid Interpretations! 4 * 8 = file; toload = won ▌ for (times; ate == file; to().load = 1) { ▌ } fore.times(8).equalsFile(2, load == 1) ▌ for times ate equals file two load equals one 3

September 28, 2004VL/HCC 2004 Graduate Student Consortium Disambiguating “ filetoload ” file to loadfile 2 loadfile toload 4

September 28, 2004VL/HCC 2004 Graduate Student Consortium Disambiguating “ filetoload ” file to loadfile 2 loadfile toload 1.file to load 6.file.to load 7.file to.load 9.file 2 load12.file toload 10.file(2, load)2.file(to, load) 3.file(to.load) 4.file(to(load)) 13.file(toload) 11.(file, 2, load) 8.file.to.load 5.file.to(load) 14.file.toload 15.filetoload() 16.filetoload 4

September 28, 2004VL/HCC 2004 Graduate Student Consortium Disambiguating “ filetoload ” file to loadfile 2 loadfile toload 1.file to load 6.file.to load 7.file to.load 9.file 2 load12.file toload 10.file(2, load)2.file(to, load) 3.file(to.load) 4.file(to(load)) 13.file(toload) 11.(file, 2, load) 8.file.to.load 5.file.to(load) 14.file.toload 15.filetoload() 1.No immediately adjacent identifiers 16.filetoload 4

September 28, 2004VL/HCC 2004 Graduate Student Consortium Disambiguating “ filetoload ” file to loadfile 2 loadfile toload 10.file(2, load)2.file(to, load) 3.file(to.load) 4.file(to(load)) 13.file(toload) 11.(file, 2, load) 8.file.to.load 5.file.to(load) 14.file.toload 15.filetoload() 1.No immediately adjacent identifiers 2.No variables named “file” 16.filetoload 4

September 28, 2004VL/HCC 2004 Graduate Student Consortium Disambiguating “ filetoload ” file to loadfile 2 loadfile toload 10.file(2, load)2.file(to, load) 3.file(to.load) 4.file(to(load)) 13.file(toload) 15.filetoload() 1.No immediately adjacent identifiers 2.No variables named “file” 3.No methods named “file” 16.filetoload 4

September 28, 2004VL/HCC 2004 Graduate Student Consortium Disambiguating “ filetoload ” file to loadfile 2 loadfile toload 15.filetoload() 1.No immediately adjacent identifiers 2.No variables named “file” 3.No methods named “file” 4.No method named “filetoload” 16.filetoload 4

September 28, 2004VL/HCC 2004 Graduate Student Consortium Disambiguating “ filetoload ” file to loadfile 2 loadfile toload 1.No immediately adjacent identifiers 2.No variables named “file” 3.No methods named “file” 4.No method named “filetoload” 3 filetoload 4

September 28, 2004VL/HCC 2004 Graduate Student Consortium The Vision Naturally Verbalized Programs –Spoken Java Language Navigation and Editing Command Language Analyses that Resolve Ambiguities Prototype: SPED: SPeech EDitor 5

September 28, 2004VL/HCC 2004 Graduate Student Consortium Implementation Speech Recognition: IBM ViaVoice Eclipse IDE Harmonia program analysis toolkit –Generalized LR parsing with input stream ambiguities –Persistent, incremental semantics 6

September 28, 2004VL/HCC 2004 Graduate Student Consortium Evaluation Hypothesis –Programmers can learn to use SPED efficiently for many programming tasks User Study 1.Train users on Spoken Java and command language 2.Edit an existing program 3.Create some new code Metrics: –Speed, vocabulary mistakes, grammatical mistakes, system understanding errors, subjective impressions 7

September 28, 2004VL/HCC 2004 Graduate Student Consortium Contributions Enabling programming by voice –New methods for handling input ambiguities –Exploiting syntax and semantics of programming domain –Analyses for mixed command and programming languages –Interface to commercial speech recognition tools 8 Andrew Begel: