1. CS 154 Formal Languages and Computability March 22 Class Meeting Department of Computer Science San Jose State University Spring 2016 Instructor: Ron Mak www.cs.sjsu.edu/~mak

2. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Context-Free Decidable Properties  There is an algorithm to decide whether or not a given string w can be derived from a context- free grammar G. Membership Is w in L(G) ?  There is an algorithm to decide whether or not L(G) is empty for a context-free grammar G.  There is an algorithm to decide whether or not L(G) is infinite for a context-free grammar G. 2 YES

3. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak More Context-Free Decision Problems  There is an algorithm to decide whether or not a given context-free grammar G is ambiguous.  There is an algorithm to decide whether or not two given context-free languages L 1 and L 2 share a common word.  There is an algorithm to decide whether or not two given context-free grammars G 1 and G 2 generate the same language. 3 NO

4. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Programming Language Parsers  A parser uses a context-free grammar for a programming language.  The parser analyzes source program statements to determine if they can be generated by the grammar. Are the statements syntactically correct? A parser is a major component of a compiler.  Parsers use either a top-down or a bottom-up approach. 4

5. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Top-Down Parsers  A top-down parser starts with the higher-level productions and works its way down to the lower-level productions.  Example: When Simple Calculator parses an expression, it starts at the highest level:  Top-down parsers are relatively easy to understand, write, and debug. But they are not very efficient. Lots of recursive calls. 5     

6. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Bottom-Up Parsers  Bottom-up parsers can be very efficient. But they can also be hard to understand and debug.  Given a context-free grammar G, we can create a PDA that implements a bottom-up parser for the language L(G). The parser should be deterministic.  We will only consider deterministic context-free languages defined by LR(1) grammars. Parsed left to right with only one symbol lookahead. Rightmost derivations. 6

7. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Shift-Reduce Parsing  A shift-reduce bottom-up parser is a PDA.  It has two repeated operations: Shift: Push the next input symbol onto the stack. Reduce: If the symbols on top of the stack match the right side of a production, replace those symbols on the stack by the variable on the left side of the production.  Terminate and accept when only the start variable is left on the stack. 7

8. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Shift-Reduce Parsing, cont’d  Questions: Should you reduce now, or read and shift more symbols from the input? Use which production to reduce? 8

9. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Shift-Reduce Parsing Example 9 OperationStack (top at right) InputRule (n+n)*n shift (n+n)*n shift (n+n)*n Reduce (F(F +n)*n FnFn Reduce (T(T +n)*n TFTF Reduce (E(E +n)*n ETET shift (E+(E+n)*n shift ( E +n)*n Reduce (E+F(E+F )*n FnFn Reduce (E+T(E+T )*n TFTF Reduce (E(E )*n EE+TEE+T shift (E)(E)*n Reduce F *n F(E)F(E) Reduce T *n TFTF shift T*T*n T *n Reduce T*FT*F FnFn T TTxFTTxF E ETET Not E  T Not T  F Is (n+n)*n syntactically correct? This grammar uses recursion instead of iteration.

10. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Shift-Reduce Parsing Example 10 OperationStack (top at right) InputRule (n+n)*n shift (n+n)*n shift (n+n)*n Reduce (F(F +n)*n FnFn Reduce (T(T +n)*n TFTF Reduce (E(E +n)*n ETET shift (E+(E+n)*n shift ( E +n)*n Reduce (E+F(E+F )*n FnFn Reduce (E+T(E+T )*n TFTF Reduce (E(E )*n EE+TEE+T shift (E)(E)*n Reduce F *n F(E)F(E) Reduce T *n TFTF shift T*T*n T *n Reduce T*FT*F FnFn T TTxFTTxF E ETET Is (n+n)*n syntactically correct? Not E  T Not T  F This grammar uses recursion instead of iteration.

11. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Table-Driven Shift-Reduce Parsing  With an LR(1) grammar, one symbol lookahead of the input is enough for the parser to decide whether to shift or to reduce.  The decisions can be encoded in a table: Is the next operation a shift or a reduce? What is the next state of the parser? 11

12. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Table-Driven Shift-Reduce Parsing, cont’d  The PDA that underlies the parser has a stack where the entries alternate between symbols (both terminals and variables) and states. Example (top at right):  Given the current state of the parser and the next input symbol, the table specifies one of four possible actions: shift, reduce, accept, or error. 12 … 4 B 7 a

13. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Table-Driven Shift-Reduce Parsing, cont’d  Shift 1. Push the current state onto the stack (e.g., 7). 2. Push the next input symbol onto the stack (e.g., a ). 3. Change to the state specified in the table. 4. Do next operation specified in the table based on the new state and the next input symbol. Example: “s3” in the table means to shift and change to state 3. 13 … 4 B 7 a

14. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Table-Driven Shift-Reduce Parsing, cont’d  Reduce by the specified production Example: “R1” in the table means to use production 1. Assume it’s C  B a. 1. Pop off the right-hand-side symbols B and a and the intervening state 7 and push C. 2. Change to the new state in the table based on the top two items on the stack, 4 and C. 3. Do next operation in the table based on the new state and the next input symbol. 14 … 4 B 7 a … 4 C

15. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Table-Driven Shift-Reduce Parsing, cont’d  Accept Stop and accept the input string (i.e., it’s syntactically correct)  Error A blank table entry indicates an error in the input (i.e., it’s syntactically incorrect) 15

16. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Shift-Reduce Parsing Example #2 16 1.  real 2. , 3.  4.  A | B | C | D State real,A|B|C|DA|B|C|D λ 0s12 1s345 2 accept 3R4 4R3 5s6R1 6s37 7R2

17. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Shift-Reduce Parsing Example #2, cont’d 17 1.  real 2. , 3.  4.  A | B | C | D State rea l,A|B|C|DA|B|C|D λ 0s12 1s345 2accept 3R4 4R3 5s6R1 6s37 7R2 StateStack (top at right)InputOperation 0 real A, B s1 1 0 realA, B s3 3 0 real 1 A, B R4 4 0 real 1, B R3 5 0 real 1, B s6 6 0 real 1 5,B s3 3 0 real 1 5, 6 B R4 7 0 real 1 5, 6 R2 5 0 real 1 R1 20 accept Is real A, B syntactically correct?

18. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Shift-Reduce Parsing Example #2, cont’d 18 1.  real 2. , 3.  4.  A | B | C | D State rea l,A|B|C|DA|B|C|D λ 0s12 1s345 2accept 3R4 4R3 5s6R1 6s37 7R2 StateStack (top at right)InputOperation 0 real A, B s1 1 0 realA, B s3 3 0 real 1 A, B R4 4 0 real 1, B R3 5 0 real 1, B s6 6 0 real 1 5,B s3 3 0 real 1 5, 6 B R4 7 0 real 1 5, 6 R2 5 0 real 1 R1 20 accept Is real A, B syntactically correct?

19. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak How to Construct an LR(1) Parsing Table  Constructing a table for an LR(1) parser is tantamount to building a deterministic PDA for the parser.  The construction algorithm first determines the state change and shift operations, and then it determines the reduction operations.  Compiler construction utilities like Unix’s yacc (yet another compiler-compiler) or Linux’s bison will build these tables based on grammar files that you provide. 19

20. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Review for the Midterm  Regular languages Kleene’s theorem closure properties pumping lemma 20

21. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Review for the Midterm, cont’d  Context-free grammars and languages leftmost and rightmost derivations derivation trees ambiguity transforming and simplifying grammars Chomsky and Greibach normal forms closure properties pumping lemma 21

22. Computer Science Dept. Spring 2016: March 22 CS 154: Formal Languages and Computability © R. Mak Review for the Midterm, cont’d  Nondeterministic pushdown automata (NPDA) flowchart programming transition function relationship to context-free languages parsers for context-free languages  JavaCC BNF expression grammars calculators 22