1. CS510
Compiler
Lecture 4

2. Syntax Analysis

3. What is Syntax Analysis?
After lexical analysis (scanning), we have a series of tokens.
In syntax analysis (or parsing), we want to interpret what those tokens mean.
Goal: Recover the structure described by that series of tokens.
Goal: Report errors if those tokens do not properly encode a structure.

4. Intro to Parsing Input: sequence of tokens from lexer
Output: parse tree of the program

5. Intro to Parsing

6. Main Topics Context-Free Grammars Derivations
Concrete and Abstract Syntax Trees
Ambiguity
Next Week: Parsing algorithms.
Top-Down Parsing
Bottom-Up Parsing

7. The Limits of Regular Languages
When scanning, we used regular expressions to define each token.
Unfortunately, regular expressions are (usually) too weak to define programming languages.
Cannot define a regular expression matching all expressions with properly balanced parentheses.
Cannot define a regular expression matching all functions with properly nested block structure.

8. We need a more powerful formalism. We need
A language for describing valid strings of tokens
A method for distinguishing valid from invalid strings of tokens
We need ……….. Context-Free Grammars

9. Context-Free Grammars
A context-free grammar (or CFG) is a formalism for defining languages.
Can define the context-free languages, a strict superset of the regular languages.
CFGs are best explained by example...

10. Context-Free Grammars
Definition of a Context-free Grammar:
An alphabet or set of basic symbols (like regular expressions, only now the symbols are whole tokens, not chars), including . (Terminals)
A set of names for structures (like statement, expression, definition). (Non-terminals)
A set of grammar rules expressing the structure of each name. (Productions)
A start symbol (the name of the most general structure – compilation-unit in C).

11. Context-Free Grammars (Continued)
Context-Free Grammars are designed to represent recursive structures. As a consequence:
The structure of a matched string is no longer given by just a sequence of symbols (lexeme), but by a tree (parse tree)
Recognition Process is much more complex: Algorithms can use stacks in many different ways.
The notation used in writing grammar rules was developed by John Backus and modified by Peter Naur
Thus, grammar rules are usually said be in Backus- Naur Form, or BNF.

12. Context-Free Grammars
A CFG consists of
A set of terminals
A set of non-terminals
A start symbol
A set of productions

13. Context-Free Grammars
Suppose we want to describe all legal arithmetic expressions
Formally, a context-free grammar is a collection of four objects:
set of nonterminal symbols (or variables),
A set of terminal symbols,
A set of production rules saying how each nonterminal can be converted by a string of terminals and nonterminal, and
A start symbol that begins the derivation.

14. Context-Free Grammars
Not Notational Shorthand :
The syntax for regular expressions does not carry over to CFGs.
Cannot use *, or parentheses.

15. Some CFG Notation
Capital letters at the beginning of the alphabet will represent nonterminals.
i.e. A, B, C, D
Lowercase letters at the end of the alphabet will represent terminals.
i.e. t, u, v, w
Lowercase Greek letters will represent arbitrary strings of terminals and nonterminals.
i.e. α, γ, ω

16. Examples We might write an arbitrary production as A → ω
We might write a string of a nonterminal followed by a terminal as At
We might write an arbitrary production containing a nonterminal followed by a terminal as
B → αAtω

17. Context-Free Grammars
Which of the strings are in the language of the given CFG?

18. Solution SaXa …….X->bY ……. Y-> cYc first not allowed
SaXa …….X->bY second not allowed
SaXa …….X->bY ……. Y-> ε
SaXa …….X->bY ……. Y-> cYc forth not allowed
The solution is : c

19. The language defined by a grammar
Grammar rules determine the legal strings of token symbols by means of derivations.
A derivation is a sequence of replacements of structure names by choices on the right-hand sides of grammar rules.
It starts with a single structure name and ends with a string of token symbols
At each step in a derivation, a single replacement is made using one choice from a grammar rule.
The set of all strings of token symbols obtained by derivations from the exp symbol is the language defined by the grammar of expressions. We can write this as:
L(G) = { s: exp * s}

20. Derivations

21. Derivations A derivation is a sequence of productions
A derivation can be drawn as a tree
Start symbol is the tree’s root
For a production X  Y1…Yn add children Y1…Yn to node X

22. Derivations

23. Derivations E
E+E
E* E+E
id *E + E
id *id + E
id *id + id

24. Derivations

25. Derivations

26. Derivations

27. Derivations

28. Derivations

29. Derivations
Which of the following is a valid parse tree for the given grammar?

30. Solution

31. Derivations
A leftmost derivation is a derivation in which each step expands the leftmost nonterminal.
A rightmost derivation is a derivation in which each step expands the rightmost nonterminal.

32. Derivations Express matching of a string of token symbols,
“(34-3)*42”, by a derivation:
(1) exp  exp op exp [exp  exp op exp]
(2)  exp op number [exp  number]
(3)  exp * number [op  * ]
(4)  ( exp ) * number [exp  ( exp )]
(5)  ( exp op exp ) * number [exp  exp op exp]
(6)  (exp op number) * number [exp  number ]
(7)  (exp - number) * number [op  - ]
(8)  (number - number)*number [exp  number ]
The above derivation is a rightmost derivation:
(The rightmost non-terminal is placed at each step.)

33. Derivations (Continued)
Another derivation, leftmost derivation, for the same string of token symbols:
(1) exp  exp op exp [exp  exp op exp]
(2)  (exp) op exp [exp  ( exp )]
(3)  (exp op exp) op exp [exp  exp op exp]
(4)  (number op exp) op exp [exp  number]
(5)  (number - exp) op exp [op  -]
(6)  (number - number) op exp [exp  number]
(7)  (number - number) * exp [op  *]
(8)  (number - number) * number [exp  number]
The representation of the structure of a string of tokens that abstracts the essential features of a derivation while factoring out differences in ordering is called a parse tree.

34. Derivations

35. Derivations: Left and Right

36. Derivations A derivation encodes two pieces of information:
What productions were applied produce the resulting string from the start symbol?
In what order were they applied?
Multiple derivations might use the same productions, but apply them in a different order.

37. Parse Tree A parse tree is a tree encoding the steps in a derivation.
Internal nodes represent nonterminal symbols used in the production.
In order walk of the leaves contains the generated string.
Encodes what productions are used, not the order in which those productions are applied.

38. Parse Trees Abstract the structure of a derivation to a parse tree
The following is the parse tree for (34-3)*42 , where the nodes are numbered according to the leftmost & rightmost derivations
exp op * 1 4 3
number 2 - 5 8 7 6 ( )
exp op * 1 2 7
number 8 - 3 4 5 6 ( )
June 6, 2018
Prof. Abdelaziz Khamis

39. Parse Trees (Continued)
A leftmost derivation corresponds to a (top-down) pre-order traversal of the parse tree.
A rightmost derivation corresponds to a (bottom-up) post-order traversal, but in reverse.
Top-down parsers construct leftmost derivations.
(LL = Left-to-right traversal of input, constructing a Leftmost derivation)
Bottom-up parsers construct rightmost derivations in reverse order.
(LR = Left-to-right traversal of input, constructing a Rightmost derivation)

40. Abstract Syntax Trees
Parse trees contain much more information than is necessary for a compiler to produce executable code.
Abstract syntax trees express the essential structure of the parse trees.
The expression can be represented more simply by the following abstract syntax tree, or syntax tree for short.
June 6, 2018
Prof. Abdelaziz Khamis

41. Ambiguity

42. Ambiguity

43. Ambiguity
A CFG is said to be ambiguous if there is at least one string with two or more parse trees.
Note that ambiguity is a property of grammars, not languages.
There is no algorithm for converting an arbitrary ambiguous grammar into an unambiguous one.
Some languages are inherently ambiguous, meaning that no unambiguous grammar exists for them.
There is no algorithm for detecting whether an arbitrary grammar is ambiguous.

44. CFG, Derivation, Parse Tree
Another example
E  E * E | E + E | ( E ) | id
Build a parse tree for: id * id + id * id
Can have different ways
Ambiguity.
If, for some input string that can be derived from the grammar,
there exists more than one parse tree to parse it,
then the grammar is ambiguous E + * id E * id + E * id +

45. Ambiguity and Derivations
Leftmost:
E  E * E  id * E  id * E + E  id * id + E 
id * id + E * E  id * id + id * E  id * id + id * id
Rightmost
E  E * E  E * E + E  E * E + E * E  E * E + E * id 
E * E + id * id  E * id + id * id  id * id + id * id
Leftmost:
E  E + E  E * E + E  id * E + E  id * id + E 
id * id + E * E  id * id + id * E  id * id + id * id
Rightmost
E  E + E  E + E * E  E + E * id  E + id * id 
E * E + id * id  E * id + id * id  id * id + id * id
Example grammar
E  E * E | E + E | ( E ) | id
Derive: id * id + id * id E * id + E + * id
Multiple derivations do not imply ambiguity,
only multiple parse trees do.
If the grammar is ambiguous then there exists multiple parse trees for the grammar, and for each parse tree, there is a unique leftmost derivation and a unique rightmost derivation.

46. Ambiguity Ambiguity implies multiple parse trees Eliminate ambiguity E
Can make parsing more difficult
Can impact the semantics of the language
Different parse trees can have different semantic meanings, yield different execution results
Sometimes, rewrite the grammar can eliminate ambiguity
But it may add additional semantics in the language
Eliminate ambiguity
E  E + T | E * T | (E) | T
T  id
Derive: id * id + id * id using leftmost derivation
E  E * T  E + T * T  E * T + T * T  E * T + T * T  T * T + T * T  …  id * id + id * id E * T id +
Is this what we want?

47. Ambiguity Rewrite grammar to eliminate ambiguity
Many ways to rewrite the grammar
The new grammar should accept the same language
For each input string, there may be multiple parse trees
Each has a different semantic meaning
Which one do we want?
Rewrite grammar should be based on the desired semantics
There is no general algorithm to rewrite ambiguous grammars

48. Rewrite Ambiguous Grammar
Try to use a single recursive nonterminal in each rule
When the left symbol appears more than once on the right side
Use additional symbols to substitute them and allow only one
Force to only allow one expansion
Example grammar
E  E + E | E – E | E * E | E / E | (E) | id
It is ambiguous
Change to
E  T + E | T – E | T * E | T / E | (E) | T
T  id
Parse: id * id – id
E  T * E  T * T – E  T * T – T  …  id * id – id E T * E id T – E id T id

49. Rewrite Ambiguous Grammar
Build desired precedence in the grammar
Example
E  E + E | E * E | (E) | id
Ambiguous
Desired precedence: * executes before +
Change to
E  E + T | T
T  T * F | F
F  (E) | id
Parse id + id * id E E + T T T * F F F id id id

50. Ambiguity – Another Example
if statement
stmt  if-stmt | while-stmt | …
if-stmt  if expr then stmt else stmt | if expr then stmt
Parse: if (a) then if (b) then x = c else x = d
if-stmt
if-stmt if
expr
then
stmt
else if
expr
then
stmt
stmt
(a)
(a)
if-stmt
x=d
if-stmt if
expr
then
stmt if
expr
then
stmt
else
stmt
(b)
x=c
(b)
x=c
x=d

51. Ambiguity – Another Example
if statement
stmt  if-stmt | while-stmt | …
if-stmt  if expr then stmt else stmt | if expr then stmt
Recursion in the rule
Not immediate recursion
Expands to another nonterminal then to itself
Indirect recursion in two places
Desired semantics
Match the else with the closest if
How to rewrite the if-stmt grammar to eliminate ambiguity?
By defining different if statements: unmatched and matched
Matched: with both then and else parts

52. Ambiguity – Another Example
Solution
if-stmt  unmatched-stmt | matched-stmt
(1) matched-stmt if expr then matched-stmt else matched-stmt
(2) unmatched-stmt if expr then both
(3) unmatched-stmt if expr then matched-stmt else unmatched-stmt
For (1)(3), only matched-stmt can appear between then and else
For (1), matched-stmt cannot leave any dangling if; otherwise, the else of the parent’s enclosing then-else should belong to the dangling if  both then and else parts should be matched
(3)’else can have matched, but will introduce ambiguity with (1)
After all matches, for each dangling if, use (2)
After then, can have either matched or unmatched
Consider all statements
matched-stmt {stmt} – {unmatched-stmt}
unmatched-stmt {stmt} – {matched-stmt}