1. Elkhound: A Fast, Practical GLR Parser Generator Scott McPeak smcpeak@cs.berkeley.edu 9/16/02 OSQ Lunch

2. 2 So what’s wrong with Bison? LALR(1) is the problem –Restrictive subset of context-free grammars –Grammar hacking breaks conceptual structure –Can’t resolve conflicts automatically if actions are present –LR is not closed under composition (union) –Fixing LR conflicts is hard: time, expertise

3. 3 Ambiguous Grammars Use of ambiguity can simplify grammar –e.g. “E ! E + E”, plus a rule for associativity Ambiguity can delay hard choices –Type/variable name ambiguity in C: “(a) & (b)” –C++: constructors, function-style casts, etc. –Other hard languages: Javascript, Perl –Natural languages?

4. 4 Generalized LR (GLR) Developed in 80’s; natural language parsing Conceptually simple Uses any context-free grammar Ambiguous grammars ! parse forest Efficient: same as LR in best case Worst case: O(2 n ) Earley (1970) best is  (n 2 ), worst is O(n 3 )

5. 5 Review: LR Parsing “L”: left-to-right parsing of input “R”: build rightmost derivation (in reverse) Build parse tables ahead of time On each token, either –shift it, pushing it onto the parse stack, or –reduce symbols at top of stack, via some production

6. Example: Arithmetic S ! E $ E ! i E ! E + E E ! E * E Grammar S ! ² E $ E ! ² i E ! ² E + E E ! ² E * E S ! E ² $ E ! E ² + E E ! E ² * E S ! E $ ² E ! i ² E ! E * ² E E ! ² i E ! ² E + E E ! ² E * E E ! E + ² E E ! ² i E ! ² E + E E ! ² E * E E ! E * E ² E ! E ² + E E ! E ² * E E ! E + E ² E ! E ² + E E ! E ² * E 013 2 4 6 5 7 E i $ * + EE+ + * * i i 2 6

7. Example LR Parse 0 3 1 2 4 6 5 7 E ! iS ! E $ E ! E * E E ! E + E 2 iii+*$ 0 1 0 2 2 3 0152 E ! i 7 01 E ! E + E 3 S ! E $ 0 6 157 E ! E * E 5 015746 E ! i 0 4 15742 Conflict (shift) 7

8. 8 GLR: Graph-structured stack Idea: pursue all possible parses at once –Allow stack to be forked into multiple “parsers” Alternate between shifts and reduces If two parsers enter same state, merge them 013 5 Stack #2 contains 3, 1, 0 Stack #1 contains 5, 1, 0

9. 9 GLR: Graph-structured stack Idea: pursue all possible parses at once –Allow stack to be forked into multiple “parsers” Alternate between shifts and reduces If two parsers enter same state, merge them 013 5 Stack #2 contains 6, 3, 1, 0 Stack #1 contains 6, 5, 1, 0 6

10. Example GLR Parse 0 3 1 2 4 6 5 7 E ! iS ! E $ E ! E * E E ! E + E 2 iii+*$ 0 2 5 232 10 E ! i 1 76 4 1 E ! E + E 7 E ! E * E E ! E + E merge 1 E ! E * E S ! E $ yielded to caller

11. 11 Aside: Nondeterminism GLR extends LR by making the stack nondeterministic Other examples: DFANFAfinite control LLLRfinite control LRGLRpushdown stack

12. 12 Optimization: Hybrid LR/GLR Full GLR is slower than LR due to the cost of interpreting the GSS But grammars are likely to be mostly deterministic (mostly linear stack) Question: How to recognize when deterministic action is possible?

13. Deterministic Depth Answer: In each stack node, remember how deep the stack’s determinism goes, e.g. Use LR if there’s only one active parser, and action is a shift, or action is reduce by , len(  ) < det_depth 123401 34 fast slow 13 Numbers in the nodes are the deterministic depths

14. 14 Programmatic Interface to GLR Other GLR parsers yield parse trees –Use a lot of memory –Not ideal for later processing stages –Commit to a given tree representation Challenges with a reduction action model –How to undo actions? –How to manage merging? –How to manage subtree sharing?

15. 15 Elkhound’s Interface Elkhound lets the user supply: –reduction action: one for each production, yields a semantic value (like Bison) –merge() : given two competing interpretations, return one value –dup() : prepare a value for being shared –del() : cancel (delete) a semantic value Claim: can build any interface on these

16. 16 Example Elkhound Specification // start symbol nonterm[PTreeNode*] StartSymbol -> tree:E EOF [ return tree; ] nonterm[PTreeNode*] E { merge(t1, t2) [ t1->addAlternative(t2); return t1; ] del(t) [] // rely on garbage collector dup(t) [ return t; ] -> a:E "+" b:E [ return new PTreeNode("E -> E + E", a, b); ] -> "b" [ return new PTreeNode("E -> b"); ] } Grammar: E ! E + E | b

17. 17 Nondeterministic Performance Grammar: E ! E + E | b Input: b(+b) n

18. 18 Deterministic Performance Grammar: E ! E + F | F F ! a | ( E ) Input: a(+a) n

19. 19 Experience Parsing C/C++ Can we just use the Standard’s grammar? –Yes: put it in and it works! –No: it’s not a parsing grammar Fails to make many important distinctions Massive number of unnecessary ambiguities I’ve modified the grammar for use with C –Ambiguity is useful for parsing __attribute__ What about C++? –Need a real C++ type-checker

20. 20 Conclusion Elkhound is as fast as Bison but far more capable due to the GLR algorithm Two contributions presented: –Hybrid LR/GLR optimization –General programmatic interface to GLR It’s available for download now! www.cs.berkeley.edu/~smcpeak/elkhound

21. 21 (blank slide)

22. Optimization Techniques 22