1. Syntax Analysis (Chapter 4) 1 Course Overview PART I: overview material 1Introduction 2Language processors (tombstone diagrams, bootstrapping) 3Architecture of a compiler PART II: inside a compiler 4Syntax analysis 5Contextual analysis 6Runtime organization 7Code generation PART III: conclusion 8Interpretation 9Review

2. Syntax Analysis (Chapter 4) 2 Nullable, First sets (starter sets), and Follow sets A non-terminal is nullable if it derives the empty string First(N) or starters(N) is the set of all terminals that can begin a sentence derived from N Follow(N) is the set of terminals that can follow N in some sentential form Next we will see algorithms to compute each of these.

3. Syntax Analysis (Chapter 4) 3 Algorithm for computing Nullable For each terminal t Nullable(t) = false For each non-terminal N Nullable(N) = is there a production N ::=  ? Repeat For each production N ::= x 1 x 2 x 3 … x n If Nullable(x i ) for all of x i then set Nullable(N) to true Until nothing new becomes Nullable

4. Syntax Analysis (Chapter 4) 4 Generalizing the definition of Nullable Define Nullable(x 1 x 2 x 3 … x n ) as: if n==0 then true else if !Nullable(x 1 ) then false else Nullable(x 2 x 3 … x n )

5. Syntax Analysis (Chapter 4) 5 Algorithm for computing First sets For each terminal t First(t) = { t } For each non-terminal N First(N) = { } Repeat For each production N ::= x 1 x 2 x 3 … x n First(N) = First(N)  First(x 1 ) For each i from 2 through n If Nullable(x 1 … x i-1 ), then First(N) = First(N)  First(x i ) Until no First set changes

6. Syntax Analysis (Chapter 4) 6 Generalizing the definition of First sets Define First(x 1 x 2 x 3 … x n ) as: if !Nullable(x 1 ) then First(x 1 ) else First(x 1 )  First(x 2 x 3 … x n ) Note: some textbooks add  (empty string) to First(N) whenever N is nullable, so that First(N) is never { } (empty set)

7. Syntax Analysis (Chapter 4) 7 Algorithm for computing Follow sets Follow(S) = {$}// the end-of-file symbol For each non-terminal N other than S Follow(N) = { } Repeat For each production N ::= x 1 x 2 x 3 … x n For each i from 1 through n-1 if x i is a non-terminal then Follow(x i ) = Follow(x i )  First(x i+1 … x n ) For each i from n downto 1 if x i is a non-terminal and Nullable(x i+1 … x n ) then Follow(x i ) = Follow(x i )  Follow(N) Until no Follow set changes

8. Syntax Analysis (Chapter 4) 8 Example of computing Nullable, First, Follow S ::= TUVW | WVUT T ::= aT | e U ::= Ub | f V ::= cV |  W ::= Wd |  Nullable?FirstFollow Sfalse{a, e, d, c, f}{$} Tfalse{a, e}{f, $} Ufalse{f}{c, d, $, a, e, b} Vtrue {c} or {c,  } {d, $, f} Wtrue {d} or {d,  } {$, c, f, d}

9. Syntax Analysis (Chapter 4) 9 Parsing We will now look at parsing. Topics: –Some terminology –Different types of parsing strategies bottom up top down –Recursive descent parsing What is it How to implement a parser given an EBNF specification

10. Syntax Analysis (Chapter 4) 10 Parsing: Some Terminology Recognition To answer the question “does the input conform to the syntax of the language” Parsing Recognition + also determine structure of program (for example by creating an AST data structure) Unambiguous grammar: A grammar is unambiguous if there is only at most one way to parse any input. (i.e. for syntactically correct program there is precisely one parse tree)

11. Syntax Analysis (Chapter 4) 11 Different kinds of Parsing Algorithms Two big groups of algorithms can be distinguished: –bottom up strategies –top down strategies Example: parsing of “Micro-English” Sentence ::= Subject Verb Object. Subject ::= I | A Noun | The Noun Object::= me | a Noun | the Noun Noun::= cat | bat | rat Verb::= like | is | see | sees Sentence ::= Subject Verb Object. Subject ::= I | A Noun | The Noun Object::= me | a Noun | the Noun Noun::= cat | bat | rat Verb::= like | is | see | sees The cat sees the rat. The rat sees me. I like a cat. The rat like me. I see the rat. I sees a rat.

12. Syntax Analysis (Chapter 4) 12 Bottom up parsing Thecatseesarat.Thecat Noun Subject sees Verb arat Noun Object. Sentence The parse tree “grows” from the bottom (leafs) up to the top (root).

13. Syntax Analysis (Chapter 4) 13 Top-down parsing Thecatseesarat.Thecatseesrat. The parse tree is constructed starting at the top (root). Sentence SubjectVerbObject. Sentence Noun Subject The Noun cat Verb seesa Noun Object Noun rat.

14. Syntax Analysis (Chapter 4) 14 Quick review Syntactic analysis –Lexical analysis Group letters into words (or group characters into tokens) Use regular expressions and deterministic FSM’s –Grammar transformations Left-factoring Left-recursion removal Substitution –Parsing = structural analysis of program Group words into sentences, paragraphs, and documents (or tokens into expressions, commands, and programs) Top-Down and Bottom-Up

15. Syntax Analysis (Chapter 4) 15 Recursive Descent Parsing Recursive descent parsing is a straightforward top-down parsing algorithm. We will now look at how to develop a recursive descent parser from an EBNF specification. Idea: the parse tree structure corresponds to the recursive calling structure of parsing functions that call each other.

16. Syntax Analysis (Chapter 4) 16 Recursive Descent Parsing Sentence ::= Subject Verb Object. Subject ::= I | A Noun | The Noun Object::= me | a Noun | the Noun Noun::= cat | bat | rat Verb::= like | is | see | sees Sentence ::= Subject Verb Object. Subject ::= I | A Noun | The Noun Object::= me | a Noun | the Noun Noun::= cat | bat | rat Verb::= like | is | see | sees Define a procedure parseN for each non-terminal N private void parseSentence( ) ; private void parseSubject( ); private void parseObject( ); private void parseNoun( ); private void parseVerb( ); private void parseSentence( ) ; private void parseSubject( ); private void parseObject( ); private void parseNoun( ); private void parseVerb( );

17. Syntax Analysis (Chapter 4) 17 Recursive Descent Parsing public class MicroEnglishParser { private TerminalSymbol currentTerminal; //Auxiliary methods will go here... //Parsing methods will go here... } public class MicroEnglishParser { private TerminalSymbol currentTerminal; //Auxiliary methods will go here... //Parsing methods will go here... }

18. Syntax Analysis (Chapter 4) 18 Recursive Descent Parsing: Auxiliary Methods public class MicroEnglishParser { private TerminalSymbol currentTerminal; private void accept (TerminalSymbol expected) { if (currentTerminal matches expected) currentTerminal = next input terminal ; else report a syntax error }... } public class MicroEnglishParser { private TerminalSymbol currentTerminal; private void accept (TerminalSymbol expected) { if (currentTerminal matches expected) currentTerminal = next input terminal ; else report a syntax error }... }

19. Syntax Analysis (Chapter 4) 19 Recursive Descent Parsing: Parsing Methods private void parseSentence( ) { parseSubject( ); parseVerb( ); parseObject( ); accept(‘.’); } private void parseSentence( ) { parseSubject( ); parseVerb( ); parseObject( ); accept(‘.’); } Sentence ::= Subject Verb Object.

20. Syntax Analysis (Chapter 4) 20 Recursive Descent Parsing: Parsing Methods private void parseSubject( ) { if (currentTerminal matches ‘ I ’) accept(‘ I ’); else if (currentTerminal matches ‘ A ’) { accept(‘ A ’); parseNoun( ); } else if (currentTerminal matches ‘ The ’) { accept(‘ The ’); parseNoun( ); } else report a syntax error } private void parseSubject( ) { if (currentTerminal matches ‘ I ’) accept(‘ I ’); else if (currentTerminal matches ‘ A ’) { accept(‘ A ’); parseNoun( ); } else if (currentTerminal matches ‘ The ’) { accept(‘ The ’); parseNoun( ); } else report a syntax error } Subject ::= I | A Noun | The Noun

21. Syntax Analysis (Chapter 4) 21 Recursive Descent Parsing: Parsing Methods private void parseNoun( ) { if (currentTerminal matches ‘ cat ’) accept(‘ cat ’); else if (currentTerminal matches ‘ bat ’) accept(‘ bat ’); else if (currentTerminal matches ‘ rat ’) accept(‘ rat ’); else report a syntax error } private void parseNoun( ) { if (currentTerminal matches ‘ cat ’) accept(‘ cat ’); else if (currentTerminal matches ‘ bat ’) accept(‘ bat ’); else if (currentTerminal matches ‘ rat ’) accept(‘ rat ’); else report a syntax error } Noun::= cat | bat | rat

22. Syntax Analysis (Chapter 4) 22 Recursive Descent Parsing: Parsing Methods private void parseObject( ) { ? } private void parseVerb( ) { ? } private void parseObject( ) { ? } private void parseVerb( ) { ? } Object::= me | a Noun | the Noun Verb::= like | is | see | sees Object::= me | a Noun | the Noun Verb::= like | is | see | sees Test yourself: Can you complete parseObject( ) and parseVerb( ) ?

23. Syntax Analysis (Chapter 4) 23 Systematic Development of Rec. Descent Parser (1)Express grammar in EBNF (2)Grammar Transformations: Left factorization and Left recursion elimination (3)Create a parser class with –private variable currentToken –methods to call the scanner: accept and acceptIt (4) Implement a public method for main function to call: –public parse method that fetches the first token from the scanner calls parse S (where S is start symbol of the grammar) verifies that scanner next produces the end–of–file token (5)Implement private parsing methods: –add private parse N method for each non terminal N