Fall 2017-2018 Compiler Principles Context-free Grammars Refresher Roman Manevich Ben-Gurion University of the Negev

Example grammar S  S ; S S  id := E S  print (L) E  id E  num E  E + E L  E L  L, E shorthand for Statement shorthand for Expression shorthand for List (of expressions)

CFG terminology S  S ; S S  id := E S  print (L) E  id E  num E  E + E L  E L  L, E Symbols: Terminals (tokens): ; := ( ) id num print Non-terminals: S E L Start non-terminal: S Convention: the non-terminal appearing in the first derivation rule Grammar productions (rules) N  α

More definitions Sentential form: a sequence of symbols, terminals (tokens) and non-terminals Sentence: a sequence of terminals (tokens) Derivation step: given a sentential form αNβ and rule N  µ a step is the transition αNβ  αµβ Derivation sequence: a sequence of derivation steps 1 …  k such that i  i+1 is the result of applying one production and k is a sentence

Language of a CFG A word ω is in L(G) (valid program) if there exists a corresponding derivation sequence Start the start symbol Repeatedly replace one of the non-terminals by a right-hand side of a production Stop when the sentence contains only terminals ω is in L(G) if S * ω Leftmost derivation Rightmost derivation

Leftmost derivation S  S ; S S  id := E S  print (L) E  id E  num a := 56 ; b := 7 + 3 S S  S ; S S  id := E S  print (L) E  id E  num E  E + E L  E L  L, E 1 => S ; S 2 => id := E ; S 3 => id := num ; S 4 5 => id := num ; id := E 6 => id := num ; id := E + E 7 => id := num ; id := num + E 8 => id := num ; id := num + num id := num ; id := num + num

Rightmost derivation S  S ; S S  id := E S  print (L) E  id a := 56 ; b := 7 + 3 S S  S ; S S  id := E S  print (L) E  id E  num E  E + E L  E L  L, E 1 => S ; S 2 => S ; id := E 3 => S ; id := E + E 4 5 => S ; id := E + num 6 => S ; id := num + num 7 => id := E ; id := num + num 8 => id := num ; id := num + num id := num ; id := num + num

Canonical derivations Leftmost/rightmost derivations may not be unique but they allow describing a derivation by the sequence of production rules taken (since non-terminal is already known) Leftmost derivation example: 1, 2, 5, 2, 6, 5, 5 Rightmost derivation example: 1, 2, 6, 5, 5, 2, 5

Parse trees Tree nodes are symbols, children ordered left-to-right Each internal node is non-terminal and its children correspond to one of its productions N  µ1 … µk Root is start non-terminal Leaves are tokens Yield of parse tree: left-to-right walk over leaves N µ1 … µk

Parse tree exercise S  S ; S S  id := E S  print (L) E  id E  num L  L, E Draw parse tree for expression id := num ; id := num + num

Parse tree exercise S  S ; S S  id := E S  print (L) E  id E  num Order-independent representation S S  S ; S S  id := E S  print (L) E  id E  num E  E + E L  E L  L, E S S E E E E id := num ; id := num + num Equivalently add parentheses labeled by non-terminal names (S(Sa := (E56)E)S ; (Sb := (E(E7)E + (E3)E)E)S)S

Capabilities and limitations of CFGs CFGs naturally express Hierarchical structure A program is a list of classes, A Class is a list of definition… Alternatives A definition is either a field definition or a method definition Beginning-end type of constraints Balanced parentheses S  (S)S | ε Cannot express Correlations between unbounded strings (identifiers) For example: variables are declared before use: ω S ω Handled by semantic analysis (attribute grammars) p. 173

Badly-formed grammars By Oren neu dag (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

Badly-formed grammars A non-terminal N is reachable if S * αNβ A non-terminal N is generating if N * ω A grammar G is badly-formed if it either contains unreachable non-terminals or non-generating non-terminals G1 = S  x N  y G2 = S  x | N N  a N b N Exercise: algorithm to test whether a grammar is badly-formed Theorem: for every grammar G there exists an equivalent well-formed grammar G’ ( that is, L(G)=L(G’) ) Proof: exercise From now on, we will only handle well-formed grammars

Ambiguity in Context-free grammars

Sometimes there are two parse trees Arithmetic expressions: E  id E  num E  E + E E  E * E E  ( E ) 1 + 2 + 3 1 + (2 + 3) (1 + 2) + 3 E E E E E E E E E E num(1) + num(2) + num(3) num(1) + num(2) + num(3) Leftmost derivation E E + E num + E num + E + E num + num + E num + num + num Rightmost derivation E E + E E + num E + E + num E + num + num num + num + num

Is ambiguity a problem for compilers? Arithmetic expressions: E  id E  num E  E + E E  E * E E  ( E ) 1 + 2 + 3 1 + (2 + 3) (1 + 2) + 3 Depends on semantics E E E E E E E E E E num(1) + num(2) + num(3) = 6 num(1) + num(2) + num(3) = 6 Leftmost derivation E E + E num + E num + E + E num + num + E num + num + num Rightmost derivation E E + E E + num E + E + num E + num + num num + num + num

Problematic ambiguity example Arithmetic expressions: E  id E  num E  E + E E  E * E E  ( E ) 1 + 2 * 3 1 + (2 * 3) (1 + 2) * 3 This is what we usually want: * has precedence over + E E E E E E E E E E num(1) + num(2) * num(3) = 7 num(1) + num(2) * num(3) = 9 Leftmost derivation E E + E num + E num + E * E num + num * E num + num * num Rightmost derivation E E * E E * num E + E * num E + num * num num + num * num

Ambiguous grammars A grammar is ambiguous if there exists a word for which there are Two different leftmost derivations Two different rightmost derivations Two different parse trees Property of grammars, not languages

Facts about ambiguous grammars Some languages are inherently ambiguous – no unambiguous grammars exist [Parikh 1961] Checking whether an arbitrary grammar is ambiguous is undecidable [Hopcroft, Motwani, Ullman, 2001]