Download presentation
Presentation is loading. Please wait.
1
1 Spoken Language Support for Software Development Andrew Begel Advisor: Susan L. Graham Computer Science Division, EECS University of California, Berkeley
2
2 Interactive Software Development Human-Computer Interaction Programming Languages Software Engineering Human-Centric Software Tools
3
3 Motivation Programmers conventionally use keyboard –Long hours at keyboard leads to higher risk of RSI Can speech-based programming be an alternative? Combines an unambiguous domain (programming) with an inherently ambiguous input modality (speech) –Great for exploring ambiguity handling in a new context while (counter < limit) { }
4
4 Programming by Voice My Goal –Find out how developers use code verbally. Use this to develop a naturally verbalizable input form. –Build development environment that supports verbal authoring, navigation, modification. Extend conventional compiler analyses to support ambiguities generated by speech. –Learn how developers can use voice-based programming, and iterate design. while counter is less than limit do...
5
5 Challenges HProgramming languages were not designed to be spoken. HSpeech is inherently ambiguous. Programming tools were not designed for ambiguity. HSpeech tools are poorly suited for programming tasks. HProgrammers are not used to verbal software development.
6
6 Talk Outline Introduction and Motivation Programming by Voice Program Analyses for Ambiguous Inputs Program Navigation and Editing Conclusion
7
7 Programming by Voice for (int i = 0; i < 10; i++ ) { ▌ }
8
8 Current Tools are Awkward! for (▌ ; ; ) { } VoiceCode [Desilets 2004] for loop … after left paren … declare india of type integer … assign zero … after semi … recall one … less than ten … after semi … recall one … increment … after left brace
9
9 for (int i = 0;▌ ; ) { } 1 2 3 4 i Current Tools are Awkward! i VoiceCode [Desilets 2004] for loop … after left paren … declare india of type integer … assign zero … after semi … recall one … less than ten … after semi … recall one … increment … after left brace
10
10 for (int i = 0; i < 10;▌ ) { } Current Tools are Awkward! 1 2 3 4 i i VoiceCode [Desilets 2004] for loop … after left paren … declare india of type integer … assign zero … after semi … recall one … less than ten … after semi … recall one … increment … after left brace
11
11 for (int i = 0; i < 10; i++ ) { ▌ } 1 2 3 4 i Current Tools are Awkward! 1 2 3 4 i i VoiceCode [Desilets 2004] for loop … after left paren … declare india of type integer … assign zero … after semi … recall one … less than ten … after semi … recall one … increment … after left brace
12
12 Programming by Voice Related Work Human-CentricComputer-Centric Multiple Tasks Authoring Only Arnold ‘00 Snell ‘00 Price ‘00 ‘02 Desilets ‘01 ‘04 Gray ‘03 Begel ‘05
13
13 How do Programmers Speak Code? 10 programmers read Java code out loud (Begel ‘05) Most programmers spoke the same way
14
14 for int i equals zero i less than ten i plus plus for (int i = 0; i < 10; i++ ) { ▌ } How do Programmers Speak Code?
15
15 Awkwardness by Design (Structural) (int)foo (3 + 5) * 7 How do Programmers Speak Code?
16
16 How do Programmers Speak Code? Individual Inconsistency System.out.println vs. System out println bar sub i vs. bar of i vs. i from bar
17
17 How do Programmers Speak Code? Native English speakers vs. non-native speakers (Pronunciation) tur vs. t u r println array[i++] vs. array[i]++
18
18 A Natural Way to Code for int i equals zero i less than ten i plus plus for (int i = 0; i < 10; i++ ) { ▌ }
19
19 Too Many Ambiguities for (int i = 0; i < 10; i++ ) { ▌ } 4 int eye equals 0 aye less then ten i plus plus KW or #? Spelling of ID? KW or ID? for int i equals zero i less than ten i plus plus
20
20 Sometimes It’s Non-Obvious for (times = 8; file(2, load); times == one) { ▌ } for times equals 8 file 2 load times equals one fore *= 8; file.tooLode.times = won ▌ 4; times = ate(file).to(load).equals(1) ▌
21
21 Design Tradeoffs Command Language Easy to analyze, but prescriptive Natural Language Flexible, but ambiguous Programming by Voice
22
22 Spoken Java Semantically identical to Java Syntactically easier to say than Java –Methodology generalizable to any computer language 1.All punctuation has English equivalents Open Brace, End For Loop 2.Most punctuation is optional 3.Provide verbalization for all abbreviations 4.Relaxed phrasing for better fit with English (int)foo “cast foo to integer” foo = 6 “set foo to 6” foo[i]++ “increment the ith element of array foo”
![22 Spoken Java Semantically identical to Java Syntactically easier to say than Java –Methodology generalizable to any computer language 1.All punctuation has English equivalents Open Brace, End For Loop 2.Most punctuation is optional 3.Provide verbalization for all abbreviations 4.Relaxed phrasing for better fit with English (int)foo cast foo to integer foo = 6 set foo to 6 foo[i]++ increment the ith element of array foo](http://images.slideplayer.com/25/8006741/slides/slide_22.jpg "22 Spoken Java Semantically identical to Java Syntactically easier to say than Java –Methodology generalizable to any computer language 1.All punctuation has English equivalents Open Brace, End For Loop 2.Most punctuation is optional 3.Provide verbalization for all abbreviations 4.Relaxed phrasing for better fit with English (int)foo cast foo to integer foo = 6 set foo to 6 foo[i]++ increment the ith element of array foo")
23
23 SPEED: Speech Editor Build an editor that supports naturally verbalized programs SPEED: SPEech EDitor Based on IBM ViaVoice, Eclipse IDE, Harmonia –Spoken Java Language for Composition –Spoken Command language for Navigation, Editing, Template instantiation, Refactorings, Search –Audible and visual feedback Similar to JavaSpeak (Smith 2000)
24
24 Harmonia Analysis Framework Framework to support interactive editors –Language-based, programmer-oriented tools Incremental analyses –Lexing (Wagner ‘97), GLR Parsing (Wagner ‘97, Begel ‘04), Static Semantics (Garrison ‘87, Begel, Jamison) C, Java, Titanium, Cool, Flex, Bison –Also, languages where indentation and CRs are significant Interactive Program Transformations (Java) CodeLink (Toomim et. al. ‘04) Shorthand Editing
25
25 Talk Outline Introduction and Motivation Programming by Voice Program Analyses for Ambiguous Inputs Program Navigation and Editing Conclusion
26
26 Traditional Compiler Analyses Lexical Analysis FORI ParsingSemantic Analysis for (i = 0; i < 10; i++ ) { } Programming languages are designed to be unambiguous For Loop FOR Assign Expr I=0 i Local Var int
27
27 Ambiguity-Aware Analyses for i equals zero... Handles input stream, syntactic and semantic ambiguities Lexical Analysis FORI Ambiguous Parsing Semantic Ambiguity Resolution For Loop FOR Assign Expr I=0 i Local Var int four eye Local Var ? 4EYE FOUREYE Assign Expr =0 Ambig Stmt
28
28 Scan Input Stream Commercial Speech Recognizer Homophone Dictionary Lexical Analysis
29
29 Homophones Cause Ambiguities foriequals for i = 4 i equals foreeye == fore ayeequalsfoureyeequals foriequals Concatenated words cause them too 4 fore fou r eye aye = ==
30
30 Ambiguity-Aware Analyses for i equals zero... Lexical Analysis FORI XGLR Ambiguous Parsing Semantic Ambiguity Resolution For Loop FOR Assign Expr I=0 i Local Var int four eye Local Var ? 4EYE FOUREYE Assign Expr =0 Ambig Stmt
31
31 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID
![31 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID](http://images.slideplayer.com/25/8006741/slides/slide_31.jpg "31 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID")
32
32 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID
![32 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID](http://images.slideplayer.com/25/8006741/slides/slide_32.jpg "32 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID")
33
33 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID
![33 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID](http://images.slideplayer.com/25/8006741/slides/slide_33.jpg "33 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID")
34
34 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID
![34 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID](http://images.slideplayer.com/25/8006741/slides/slide_34.jpg "34 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID")
35
35 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID
![35 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID](http://images.slideplayer.com/25/8006741/slides/slide_35.jpg "35 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID")
36
36 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID
![36 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID](http://images.slideplayer.com/25/8006741/slides/slide_36.jpg "36 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID")
37
37 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID FOO ID IF KW Expr IF KW Expr
![37 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID FOO ID IF KW Expr IF KW Expr](http://images.slideplayer.com/25/8006741/slides/slide_37.jpg "37 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID FOO ID IF KW Expr IF KW Expr")
38
38 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID FOO ID IF KW Expr IF KW Expr Stmt
![38 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID FOO ID IF KW Expr IF KW Expr Stmt](http://images.slideplayer.com/25/8006741/slides/slide_38.jpg "38 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID FOO ID IF KW Expr IF KW Expr Stmt")
39
39 GLR Parsing [ Tomita 85 ] ELSE KW BAR ID ELSE KW FOO ID IF KW Expr IF KW Expr FOO ID IF KW Expr IF KW Expr Stmt
![39 GLR Parsing [ Tomita 85 ] ELSE KW BAR ID ELSE KW FOO ID IF KW Expr IF KW Expr FOO ID IF KW Expr IF KW Expr Stmt](http://images.slideplayer.com/25/8006741/slides/slide_39.jpg "39 GLR Parsing [ Tomita 85 ] ELSE KW BAR ID ELSE KW FOO ID IF KW Expr IF KW Expr FOO ID IF KW Expr IF KW Expr Stmt")
40
40 GLR Parsing [ Tomita 85 ] ELSE KW BAR ID ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr FOO ID IF KW Expr IF KW Expr Stmt
![40 GLR Parsing [ Tomita 85 ] ELSE KW BAR ID ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr FOO ID IF KW Expr IF KW Expr Stmt](http://images.slideplayer.com/25/8006741/slides/slide_40.jpg "40 GLR Parsing [ Tomita 85 ] ELSE KW BAR ID ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr FOO ID IF KW Expr IF KW Expr Stmt")
41
41 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID Stmt ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr Stmt
![41 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID Stmt ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr Stmt](http://images.slideplayer.com/25/8006741/slides/slide_41.jpg "41 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID Stmt ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr Stmt")
42
42 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID Stmt ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr Stmt
![42 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID Stmt ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr Stmt](http://images.slideplayer.com/25/8006741/slides/slide_42.jpg "42 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID Stmt ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr Stmt")
43
43 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID Stmt ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr Stmt
![43 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID Stmt ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr Stmt](http://images.slideplayer.com/25/8006741/slides/slide_43.jpg "43 FOO ID GLR Parsing [ Tomita 85 ] ELSE KW IF KW Expr IF KW Expr BAR ID Stmt ELSE KW BAR ID FOO ID IF KW Expr IF KW Expr Stmt")
44
44 GLR Ambiguity Support 1.Fork when there is more than one possible interpretation (tree)
45
45 XGLR Ambiguity Support 1.Fork when there is more than one possible interpretation (tree) 2.Fork when the lexical input is ambiguous
46
46 XGLR Parsing [Begel 04] <XIFFIFTYFIVE
![46 XGLR Parsing [Begel 04] <XIFFIFTYFIVE](http://images.slideplayer.com/25/8006741/slides/slide_46.jpg "46 XGLR Parsing [Begel 04] <XIFFIFTYFIVE")
47
47 XGLR Parsing [ Begel 04 ] <XIFFIFTYFIVE KW
![47 XGLR Parsing [ Begel 04 ] <XIFFIFTYFIVE KW](http://images.slideplayer.com/25/8006741/slides/slide_47.jpg "47 XGLR Parsing [ Begel 04 ] <XIFFIFTYFIVE KW")
48
48 XGLR Parsing [ Begel 04 ] <XFIFTYFIVE IF KW
![48 XGLR Parsing [ Begel 04 ] <XFIFTYFIVE IF KW](http://images.slideplayer.com/25/8006741/slides/slide_48.jpg "48 XGLR Parsing [ Begel 04 ] <XFIFTYFIVE IF KW")
49
49 XGLR Parsing [ Begel 04 ] <XFIFTYFIVEIF KW FIFTY5 50FIVE 505 55 ID # # # # # <X <X <X <X
![49 XGLR Parsing [ Begel 04 ] <XFIFTYFIVEIF KW FIFTY5 50FIVE ID # # # # # <X <X <X <X](http://images.slideplayer.com/25/8006741/slides/slide_49.jpg "49 XGLR Parsing [ Begel 04 ] <XFIFTYFIVEIF KW FIFTY5 50FIVE ID # # # # # <X <X <X <X")
50
50 XGLR Parsing [ Begel 04 ] FIFTYFIVE IF KW FIFTY5 50FIVE 5 ID # 50 # 55 # # ID # IF KW IF KW IF KW IF KW <X <X <X <X <X Op
![50 XGLR Parsing [ Begel 04 ] FIFTYFIVE IF KW FIFTY5 50FIVE 5 ID # 50 # 55 # # ID # IF KW IF KW IF KW IF KW <X <X <X <X <X Op](http://images.slideplayer.com/25/8006741/slides/slide_50.jpg "50 XGLR Parsing [ Begel 04 ] FIFTYFIVE IF KW FIFTY5 50FIVE 5 ID # 50 # 55 # # ID # IF KW IF KW IF KW IF KW <X <X <X <X <X Op")
51
51 XGLR Parsing [ Begel 04 ] FIFTYFIVEIF KW FIFTY5 50FIVE 505 55 ID # # # # # IF KW IF KW IF KW IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW FIFTY ID IF KW.. <X <X <X <X X< Op
![51 XGLR Parsing [ Begel 04 ] FIFTYFIVEIF KW FIFTY5 50FIVE ID # # # # # IF KW IF KW IF KW IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW FIFTY ID IF KW..](http://images.slideplayer.com/25/8006741/slides/slide_51.jpg "<X <X <X <X X< Op.")
52
52 XGLR Parsing [ Begel 04 ] FIFTYFIVEIF KW FIFTY5 50FIVE 505 55 ID # # # # # IF KW IF KW IF KW IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW FIFTY ID IF KW.. <X <X <X <X X< Op FIVE 5 5 # ID # <X <X <X <X
![52 XGLR Parsing [ Begel 04 ] FIFTYFIVEIF KW FIFTY5 50FIVE ID # # # # # IF KW IF KW IF KW IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW FIFTY ID IF KW..](http://images.slideplayer.com/25/8006741/slides/slide_52.jpg "<X <X <X <X X< Op FIVE 5 5 # ID # <X <X <X <X.")
53
53 XGLR Parsing [ Begel 04 ] FIFTYFIVEIF KW FIFTY5 50FIVE 505 55 ID # # # # # IF KW IF KW IF KW IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW FIFTY ID IF KW.. <X <X <X <X X< Op FIVE 5 5 # ID # <X <X <X <X
![53 XGLR Parsing [ Begel 04 ] FIFTYFIVEIF KW FIFTY5 50FIVE ID # # # # # IF KW IF KW IF KW IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW FIFTY ID IF KW..](http://images.slideplayer.com/25/8006741/slides/slide_53.jpg "<X <X <X <X X< Op FIVE 5 5 # ID # <X <X <X <X.")
54
54 XGLR Parsing [ Begel 04 ] FIFTYFIVEIF KW FIFTY5 50FIVE 505 55 ID # # # # # IF KW IF KW IF KW IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW FIFTY ID IF KW.. <X <X <X <X X< Op FIVE 5 5 # ID # <X <X <X <X
![54 XGLR Parsing [ Begel 04 ] FIFTYFIVEIF KW FIFTY5 50FIVE ID # # # # # IF KW IF KW IF KW IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW FIFTY ID IF KW..](http://images.slideplayer.com/25/8006741/slides/slide_54.jpg "<X <X <X <X X< Op FIVE 5 5 # ID # <X <X <X <X.")
55
55 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID. X < Op FIVE 5 ID # <X <X <X
![55 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID.](http://images.slideplayer.com/25/8006741/slides/slide_55.jpg "X < Op FIVE 5 ID # <X <X <X.")
56
56 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID. X < Op FIVE 5 ID # <X <X <X FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID. FIVE ID FIFTYIF KW ID. FIVE ID.. ( ( <X <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X
![56 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID.](http://images.slideplayer.com/25/8006741/slides/slide_56.jpg "X < Op FIVE 5 ID # <X <X <X FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID. FIVE ID FIFTYIF KW ID. FIVE ID.. ( ( <X <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X.")
57
57 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID. X < Op FIVE 5 ID # <X <X <X FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID. FIVE ID FIFTYIF KW ID. FIVE ID.. ( ( <X <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID
![57 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID.](http://images.slideplayer.com/25/8006741/slides/slide_57.jpg "X < Op FIVE 5 ID # <X <X <X FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID. FIVE ID FIFTYIF KW ID. FIVE ID.. ( ( <X <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID.")
58
58 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID. X < Op FIVE 5 ID # <X <X <X FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID. FIVE ID FIFTYIF KW ID. FIVE ID.. ( ( <X <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID
![58 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID.](http://images.slideplayer.com/25/8006741/slides/slide_58.jpg "X < Op FIVE 5 ID # <X <X <X FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID ( FIVE ID FIFTYIF KW ID. FIVE ID FIFTYIF KW ID. FIVE ID.. ( ( <X <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID.")
59
59 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID. X < Op FIVE 5 ID # <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID
![59 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID.](http://images.slideplayer.com/25/8006741/slides/slide_59.jpg "X < Op FIVE 5 ID # <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID.")
60
60 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID. X < Op FIVE 5 ID # <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID
![60 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID.](http://images.slideplayer.com/25/8006741/slides/slide_60.jpg "X < Op FIVE 5 ID # <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID.")
61
61 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID. X < Op FIVE 5 ID # <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID
![61 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID.](http://images.slideplayer.com/25/8006741/slides/slide_61.jpg "X < Op FIVE 5 ID # <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID.")
62
62 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW ID. X < Op FIVE ID <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID <X <X Op ID
![62 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW ID.](http://images.slideplayer.com/25/8006741/slides/slide_62.jpg "X < Op FIVE ID <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID <X <X Op ID.")
63
63 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID. X < Op FIVE 5 ID # <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID Expr FuncCall Expr FuncCall
![63 XGLR Parsing [ Begel 04 ] 55 # IF KW FIFTYIF KW FIFTY ID IF KW ( ( FIFTYIF KW ID.](http://images.slideplayer.com/25/8006741/slides/slide_63.jpg "X < Op FIVE 5 ID # <X <X <X FIFTYIF KW FIFTY ID IF KW ( ( FIVE 5 # ID ) ) <X <X Op ID Expr FuncCall Expr FuncCall.")
64
64 XGLR Summary Generalization of traditional GLR algorithm –Forks on structural and lexical ambiguity –Preserves subtree sharing when parses have different yields –Preserves efficiency when parses get out of sync Determine parse position w.r.t. ambiguous input Blender: Combined lexer and parser generator for XGLR
65
65 GLR Parsing Genealogy Tomita 1985 Farshi 1991 Rekers 1992 Wagner 1997 Visser 1997 van den Brand 2002 Begel 2004 Incremental Scannerless Johnstone et. al. 2002 Input Stream Ambiguities
66
66 Ambiguity-Aware Analyses for i equals zero... Lexical Analysis FORI XGLR Ambiguous Parsing Semantic Ambiguity Resolution For Loop FOR Assign Expr I=0 i Local Var int four eye Local Var ? 4EYE FOUREYE Assign Expr =0 Ambig Stmt
67
67 Disambiguation Example class Loader { public void load() { String filetoload = null; InputStream stream = getStream();... ▌ } file to load equals stream dot read string filetoload = stream.readString();
68
68 Many Interpretations file(2, load) file(to, load) file(to.lode) file(to(lode)) file(toload) (file, 2, load) file.to.load file.to(load) file.toload filetoload() filetoload
69
69 Incremental Semantics What does this name mean? What names are visible at this program point? –Or, What can I say here? Visibility Graph [Garrison 1987] –Incrementally updated data structure for scopes, names and bindings –Designed Visibility Graph algorithms for name propagation and incremental update –Used for type checking, too Doesn’t do this?
70
70 Program Context Can Help class Loader { public void load() { String filetoload = null; InputStream stream = getStream();... ▌ } class Loader scope [ load, Method, () void ] method load scope [ filetoload, LocalVar, String ] [ stream, LocalVar, InputStream ]
71
71 Program Context Can Help class Loader { public void load() { String filetoload = null; InputStream stream = getStream();... ▌ } class Loader scope [ load, Method, () void ] method load scope [ filetoload, LocalVar, String ] [ stream, LocalVar, InputStream ] [ load, Method, () void ]
72
72 Semantic Disambiguation file(2, load) file(to, load) file(to.lode) file(to(lode)) file(toload) (file, 2, load) file.to.load file.to(load) file.toload filetoload() filetoload class Loader scope [ load, Method, () void ] method load scope [ filetoload, LocalVar, String ] [ stream, LocalVar, InputStream ] [ load, Method, () void ]
![72 Semantic Disambiguation file(2, load) file(to, load) file(to.lode) file(to(lode)) file(toload) (file, 2, load) file.to.load file.to(load) file.toload filetoload() filetoload class Loader scope [ load, Method, () void ] method load scope [ filetoload, LocalVar, String ] [ stream, LocalVar, InputStream ] [ load, Method, () void ]](http://images.slideplayer.com/25/8006741/slides/slide_72.jpg "72 Semantic Disambiguation file(2, load) file(to, load) file(to.lode) file(to(lode)) file(toload) (file, 2, load) file.to.load file.to(load) file.toload filetoload() filetoload class Loader scope [ load, Method, () void ] method load scope [ filetoload, LocalVar, String ] [ stream, LocalVar, InputStream ] [ load, Method, () void ]")
73
73 Semantic Disambiguation file(2, load) file(to, load) file(to.lode) file(to(lode)) file(toload) (file, 2, load) file.to.load file.to(load) file.toload filetoload() filetoload class Loader scope [ load, Method, () void ] method load scope Is “ file ” a visible variable name? [ filetoload, LocalVar, String ] [ stream, LocalVar, InputStream ] [ load, Method, () void ]
![73 Semantic Disambiguation file(2, load) file(to, load) file(to.lode) file(to(lode)) file(toload) (file, 2, load) file.to.load file.to(load) file.toload filetoload() filetoload class Loader scope [ load, Method, () void ] method load scope Is file a visible variable name.](http://images.slideplayer.com/25/8006741/slides/slide_73.jpg "[ filetoload, LocalVar, String ] [ stream, LocalVar, InputStream ] [ load, Method, () void ].")
74
74 Semantic Disambiguation file(2, load) file(to, load) file(to.lode) file(to(lode)) file(toload) filetoload() filetoload class Loader scope [ load, Method, () void ] method load scope Is “ file ” a visible method name? [ filetoload, LocalVar, String ] [ stream, LocalVar, InputStream ] [ load, Method, () void ]
![74 Semantic Disambiguation file(2, load) file(to, load) file(to.lode) file(to(lode)) file(toload) filetoload() filetoload class Loader scope [ load, Method, () void ] method load scope Is file a visible method name.](http://images.slideplayer.com/25/8006741/slides/slide_74.jpg "[ filetoload, LocalVar, String ] [ stream, LocalVar, InputStream ] [ load, Method, () void ].")
75
75 Semantic Disambiguation filetoload() filetoload class Loader scope [ load, Method, () void ] method load scope Is “ filetoload ” a visible method name? [ filetoload, LocalVar, String ] [ stream, LocalVar, InputStream ] [ load, Method, () void ]
![75 Semantic Disambiguation filetoload() filetoload class Loader scope [ load, Method, () void ] method load scope Is filetoload a visible method name.](http://images.slideplayer.com/25/8006741/slides/slide_75.jpg "[ filetoload, LocalVar, String ] [ stream, LocalVar, InputStream ] [ load, Method, () void ].")
76
76 Manual Disambiguation Some ambiguities cannot (and should not) be automatically resolved: print(“line”) vs. println() if (pred1) then if (pred2) then foo() else bar() If ambiguities remain, ask the user how to resolve them. (e.g. [Mankoff 00]) if foo() if bar() if foo() bar()
77
77 Talk Outline Introduction and Motivation Programming by Voice Program Analyses for Ambiguous Inputs Program Navigation and Editing Conclusion
78
78 Study - Navigation by Speech 1.Eight navigation tasks with commercial VR tools 2.Search through text (w/o Find dialog), e.g. Find the sentence where Romeo cries out about his fate after killing Tybalt. Metrics 1.Time to scroll to right page 2.Number of commands 3.Number of recognition errors 4.Number of system mistakes Results 1.Cognitive load is too high - too many commands, misestimation errors 2.Voice recognition induces too much delay/errors for accurate control
79
79 SPEED Navigation Shorthand Editing class Loader { public void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... }
80
80 SPEED Navigation Shorthand Editing class Loader { public void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... }
81
81 SPEED Navigation Shorthand Editing down class Loader { public void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... }
82
82 SPEED Navigation Shorthand Editing protected class Loader { protected void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... }
83
83 SPEED Navigation Shorthand Editing private class Loader { private void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... }
84
84 SPEED Navigation Shorthand Editing down class Loader { private void load() { InputStream stream = getStream(); String filetoload = null; filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... }
85
85 SPEED Navigation Shorthand Editing push down class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... }
86
86 SPEED Navigation Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... }
![86 SPEED Navigation Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = file.txt ; ▌ } public void save() {...](http://images.slideplayer.com/25/8006741/slides/slide_86.jpg "} class MenuLoader extends Loader {... }.")
87
87 SPEED Navigation Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... } 1 2
![87 SPEED Navigation Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = file.txt ; ▌ } public void save() {...](http://images.slideplayer.com/25/8006741/slides/slide_87.jpg "} class MenuLoader extends Loader {... } 1 2.")
88
88 SPEED Navigation Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] 1 class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... } 1 2
![88 SPEED Navigation Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] 1 class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = file.txt ; ▌ } public void save() {...](http://images.slideplayer.com/25/8006741/slides/slide_88.jpg "} class MenuLoader extends Loader {... } 1 2.")
89
89 SPEED Navigation Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] 1 class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... } 1 3 2 4
![89 SPEED Navigation Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] 1 class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = file.txt ; ▌ } public void save() {...](http://images.slideplayer.com/25/8006741/slides/slide_89.jpg "} class MenuLoader extends Loader {... }")
90
90 SPEED Navigation Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] 1 1 3 select class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... }
![90 SPEED Navigation Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] select class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = file.txt ; ▌ } public void save() {...](http://images.slideplayer.com/25/8006741/slides/slide_90.jpg "} class MenuLoader extends Loader {... }.")
91
91 SPEED Editing Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] edit class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... } filetoload equals quote file.txt quote
![91 SPEED Editing Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] edit class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = file.txt ; ▌ } public void save() {...](http://images.slideplayer.com/25/8006741/slides/slide_91.jpg "} class MenuLoader extends Loader {... } filetoload equals quote file.txt quote.")
92
92 SPEED Editing Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] edit class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = “file.txt”; ▌ } public void save() {... } class MenuLoader extends Loader {... } file to load equals stream dot read string
![92 SPEED Editing Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] edit class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = file.txt ; ▌ } public void save() {...](http://images.slideplayer.com/25/8006741/slides/slide_92.jpg "} class MenuLoader extends Loader {... } file to load equals stream dot read string.")
93
93 SPEED Editing Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] put it back class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = stream.readString(); ▌ } public void save() {... } class MenuLoader extends Loader {... } file to load equals stream dot read string
![93 SPEED Editing Shorthand Editing Context-Sensitive Mouse Grid [Begel] – Related Work: Tree Hierarchy Navigation [Smith 2004] put it back class Loader { private void load() { String filetoload = null; InputStream stream = getStream(); filetoload = stream.readString(); ▌ } public void save() {...](http://images.slideplayer.com/25/8006741/slides/slide_93.jpg "} class MenuLoader extends Loader {... } file to load equals stream dot read string.")
94
94 Study - SPEED Usability Goal: Understand how SPEED can be used by expert programmers 1.Train expert Java programmers on SPEED 2.Author new code –Build a Linked List data structure with associated algorithms 3.Modify existing code – Change abstraction representation and update algorithms
95
95 Metrics User Metrics –Speed –Vocabulary mistakes, Forgotten vocabulary –Grammatical mistakes, Grammatical substitutions SPEED Metrics –Word tokenization errors –Disambiguation errors –Irresolvable ambiguities
96
96 Talk Outline Introduction and Motivation Programming by Voice Program Analyses for Ambiguous Inputs Program Navigation and Editing Conclusion
97
97 Contributions 1.A study of programmers to understand and design a naturally verbalizable input for programming 2.An interactive editor designed for spoken interaction 3.The use of syntax and semantics of programming for disambiguation – Enhanced lexical, syntactic, semantic analyses for support of verbal ambiguities 4.Evaluation of design and tools by studying programmers using voice for software development
98
98 Future of Programming by Voice 1.Improved automation of semantic disambiguation – Use ideas from NLP, Machine Learning (team styles) 2.Early pruning of ambiguities using analysis feedback 3.Higher-level linguistic programming tools – Transformations, Paraphrasing – Phonetic search, Audible feedback 4.Support more software engineering tasks by voice – Debuggers, IDEs, Comments, Code reviews 5.Design spoken variants of other formal languages – General (C, C#) Scripting (PL, OS), Design (HCI), Command (Robotics), Domain-specific languages (SQL)
99
99 Future Research Directions Data mining the program interaction history –Task inference, reminders, coding guides Improving non-code programmer communication –Design history, Code orientation, Software spelunking Programming in the “Large” –Multiple hi-res displays, whiteboards Visualizing effects of compilation and optimization –Scalable task-oriented program visualization Scalable Debugging –Concurrent apps, long-lived apps
100
100 Any Questions? Andrew Begel: abegel@cs.berkeley.edu
Similar presentations
